Questions about Variable Settings

[allenhaozi]

Apache Airflow version

2.2.5

What happened

I use a Variable to dynamically set the number of a task, and the code is as follows

    task_num_str = Variable.get("simulate_task_num", "10")
    simulate_task_num = int(task_num_str)

    for index in range(simulate_task_num):
        tmp = gen_k8s_pod_operator("simulate", index)
        t1 >> t2 >> tmp >> t4 >> t5

But I found that if I changed the value when the last instance was running
You will find that it affects instances that are already running
Is this logic by Design?

What you think should happen instead

No response

How to reproduce

No response

Operating System

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Pleae avoid opening issue, when you have a question. You've been advisede (by template instructions you probably ignored) that if you have questions, you should open discussion rather than an issue (and especially the "bug" type).

[potiuk]

Yes. It's by design.

[potiuk]

https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning is the proposal on making DAGs versionable but it's not yet even approved. For now DAGs should be changing slowly and in non breaking fashion

https://airflow.apache.org/docs/apache-airflow/stable/index.html

Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.

[potiuk]

Besides you are not following the best practices https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html?highlight=best+practices#top-level-python-code - you should avoid using variables as top-level Python code as you are increasing database pressure immensely with that. Airlfow Variables should be use mostly inside of the tasks not in the top-level DAG code.

[allenhaozi]

got it , thank you @potiuk