Some more info: Generally the long running task is a python operator which runs either a presto sql query or Databricks job via Prestohook or DatabricksOperator respectively. I Would appreciate any solution around this. Since tasks in my dags can run anywhere between few minutes to few hours so I don't want to put a large value for gracefullTerminationPeriod. 2 days ago &0183 &32 Kind is one of the easiest ways of starting out with Kubernetes development, especially if youre just beginning your work with containers. And my main issue is there any way I can setup config so that worker pod only terminates when task running on it finishes execution. So I am not sure why I see total of 8 min for worker pod termination. I can see that worker pod should shutdown after 5 mins or irrespective task running or not. # to understand with KubernetesPodOperator(), as Pods may continue running To deploy the quickstarter the component name must be airflow-worker. # - tasks that are still running during SIGKILL will be orphaned, this is important This boilerplate provides an Airflow Cluster using Kubernetes Executor hosted in. # how many seconds to wait after SIGTERM before SIGKILL of the celery worker # how many seconds to wait for tasks to finish before SIGTERM of the celery worker wait AT MOST `workers.terminationPeriod` for kill to finish # enough available workers during graceful termination waiting periods # - consider defining a `workers.podDisruptionBudget` to prevent there not being # if celery worker Pods are gracefully terminated Clean up Kubernetes pods (created by KubernetesExecutor/KubernetesPodOperator) in evicted/failed/succeeded/pending states. I have setup below two properties in helm chart for worker pod termiantion. If any task is running on a worker pod, it terminates in about 8min, and after one more minute, I find the task failing on UI.If no tasks is running on a worker pod, it terminates within 40sec.When worker pods start scaling down, two things happen:
However I am facing an issue when these worker pods are scaling down. If CPU/Mem > 70% then airflow spins up new worker pod. I have setup some scaling config for worker setup. I am running an airflow cluster on EKS on AWS. Understand the purpose of the three most popular Executors: Local, Celery, and Kubernetes Assumed knowledge To get the most out of this guide, you should have an understanding of: Basic Airflow concepts.
0 kommentar(er)
