9/19/2023, 5:16:07 AM

There are several ways to run dbt in Airflow:

Using Cosmos: This is the recommended approach. With Cosmos, you can integrate dbt jobs into your Airflow orchestration environment as a standalone DAG or as a task group within a DAG.

Using the BashOperator: You can use the BashOperator to execute specific dbt commands. It's recommended to run dbt-core and the dbt adapter for your database in a virtual environment because there often are dependency conflicts between dbt and other packages.

Using a manifest file: Using a dbt-generated manifest.json file gives you more visibility into the steps dbt is running in each task. This file is generated in the target directory of your dbt project and contains its full representation.

Remember that each method has its own pros and cons, and the choice depends on your specific use case and requirements.

Not the answer you were looking for or need more help? Contact us!

9/19/2023, 5:15:53 AM

What are the different options of running dbt in Airflow?