9/19/2023, 5:19:04 AM

Deploying Airflow in production involves several steps and considerations:

Infrastructure: Ensure you're running Airflow on an infrastructure that meets your organization's requirements. Local setups like using the Astro CLI are great for testing and DAG development, but not for running DAGs in production.

Scalability: Consider the scalability of your setup. Depending on your needs, you might need to scale out your Airflow deployment. You can find more information on this in the <|Scaling out Airflow> guide.

Configuration: Configure your Airflow environment according to best practices. This includes setting up your database, adding DAG tags and owner links, setting up notifications, and more. Check the /start section for more detailed guides on these tasks.

Images and Charts: Use production-ready resources like the OSS Production Docker Images and the OSS Official Helm Chart.

Managed Airflow: Consider using a managed service like Astro, which provides a lot of pre-configured settings and allows you to scale as needed.

Remember that Airflow's resource needs can vary significantly based on factors like the complexity of your DAGs, the number of tasks you're running, and more. You should be prepared to continuously monitor your system and adjust your settings as needed to ensure smooth operation.

Not the answer you were looking for or need more help? Contact us!

9/19/2023, 5:18:52 AM

how should I deploy airflow in production?