How to get conf value from airflow dag?

I want to get conf value from dag area.

"{{ dag_run.conf['company'] }}" is recognized as a string.

How can I get this value?

Values ​​are passed fine when calling other dags.

t_trigger = TriggerDagRunOperator(
    task_id="t-trigger",
    trigger_dag_id="example_dag",
    conf={
        "company": "{{ dag_run.conf['company'] }}",
    },
)

However, in the dag area the value is recognized as a string.

t_task_a = PythonOperator(
    task_id="t-task-a",
    python_callable=task-a,
)
employees = Variable.get(
    "{{ dag_run.conf['company'] }}", # problem
    default_var=['company'],
    deserialize_json=True
)
for employee in employees:
    t_employee_operator = PythonOperator(
        task_id=f"t-test-operator",
        python_callable=employee_operator,
        op_kwargs={"employee": employee}
    )
    t_task_a >> t_employee_operator

1 answer

  • answered 2022-05-08 07:20 Elad Kalif

    As you already noticed Airflow does not render templates outside of Operators scope - this is expected. You can read this answer for more information about it.

    Furthermore you are using Variable.get() as top level code which is very abusive. I explained the reasons in detail in this answer.

    I would suggest to redesign your DAG structure to fit Airflow DAG writing practices.

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum