Add bundle_name and latest_bundle_version to dags list command

[jedcunningham]

Body

We need to add bundle_name and latest_bundle_version to the dags list command

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ambika-garg]

Hey @jedcunningham, I can take this up. You can assign it to me.

[ambika-garg]

Hey @jedcunningham, I have a quick question about creating DAG Bundles. I noticed several classes under dag_processing/bundles. How can we use them to create bundles locally?

[jedcunningham]

You set them in config. The default and example there should get you going, but feel free to reach back out if you get stuck!

[ambika-garg]

I have created a draft PR (#45779) that adds the bundle name as an argument to the dags list command. Could you please review it to ensure it's on the right track? Once confirmed, I will proceed to add the bundle_version argument as well.

[jedcunningham]

Couple things:

• This issue was more intended to show the bundle name and version in the output. But adding support for the flag is what Remove subdir arg support from cli commands #45647 was meant for, which you also grabbed :)
• As for the display part of this issue, we have 2 options. First would be to wait for AIP-81 Transition of DAG Command #45665 and Add bundle_name and latest_bundle_version to DAG api endpoint #45644 to be done, and make sure these are added to the output. The other, which I like better (if you want to move fast) is to switch this cli command to use DagBag(read_dags_from_db=True) and get those columns added. The approach of using dag_schema from api_connexion for the list of columns also will need to go in AIP-81 Transition of DAG Command #45665, so feel free to "just make it work".

[ambika-garg]

Thankyou @jedcunningham for reviewing it, I'll go with the approach enabling DagBag(read_dags_from_db=True) and using the added columns in the output of cli command.

[ambika-garg]

Hi @jedcunningham,

I have a few questions:

1. I was able to display the bundle name and version in the output as shown in the image. Let me know if this aligns with your expectations. You can also review the changes in the draft PR.

2. I tested this on LocalBundle, but since it doesn’t support versions, I wanted to test it on GitBundle. However, I encountered an error. I updated the config.yml with the following changes but ran into issues. Could you guide me on whether any specific configurations are required?

3. As per your suggestion, I attempted to use Dagbag(read_dags_from_db=True). While exploring this approach, I tried using the collect_dags_from_db function from dagbag file, It was unable to load any file from the db.

Looking forward to your guidance!

[ambika-garg]

Once we are good with list_dags command, we can extend the output functionality to the other CLI commands as well.

[jedcunningham]

1. I was able to display the bundle name and version in the output as shown in the image. Let me know if this aligns with your expectations. You can also review the changes in the [draft PR.](https://github.com/apache/airflow/pull/45779)

Yep, exactly!

2. I tested this on LocalBundle, but since it doesn’t support versions, I wanted to test it on GitBundle. However, I encountered an error. I updated the config.yml with the following changes but ran into issues. Could you guide me on whether any specific configurations are required?

Without the error message, it's tough to say. There has been a bit of churn in GitHook and GitDagBundle recently. If you need a working config, let me know and I'll craft one to share.

3. As per your suggestion, I attempted to use Dagbag(read_dags_from_db=True). While exploring this approach, I tried using the collect_dags_from_db function from [dagbag](https://github.com/apache/airflow/blob/main/airflow/models/dagbag.py) file, It was unable to load any file from the db.

Hmm, in theory, you can just do:

>>> from airflow.models.dagbag import DagBag
>>> bag = DagBag(read_dags_from_db=True)
>>> bag.collect_dags_from_db()
[2025-01-26T19:00:50.726-0700] {dagbag.py:612} INFO - Filling up the DagBag from database
...
>>> bag.dag_ids
['hello', 'mapping_always_short', 'ke_image_override']

Did you get any errors? Or it was just empty?

[ambika-garg]

Hmm, in theory, you can just do:

>>> from airflow.models.dagbag import DagBag
>>> bag = DagBag(read_dags_from_db=True)
>>> bag.collect_dags_from_db()
[2025-01-26T19:00:50.726-0700] {dagbag.py:612} INFO - Filling up the DagBag from database
...
>>> bag.dag_ids
['hello', 'mapping_always_short', 'ke_image_override']

Did you get any errors? Or it was just empty?

Hey @jedcunningham , I get the following error upon trying it.

[ambika-garg]

Could you also please share a working configuration for the GithubDagBundle?

[jedcunningham]

Sounds like your db isn't migrated.

Here is an example. First you need to set up a connection:

AIRFLOW_CONN_AIRFLOW_GIT_CONN='{"host": "https://github.com/apache/airflow.git"}'

Then, add this to your config:

[dag_processor]
dag_bundle_config_list = [
        {
            "name": "airflow_example_dags",
            "classpath": "airflow.dag_processing.bundles.git.GitDagBundle",
            "kwargs": {"tracking_ref": "main", "git_conn_id": "airflow_git_conn", subdir="airflow/example_dags"}
        }
    ]

However, something is broken right now and the processor is getting stuck on servicing the sockets. I'll have to dig into this tomorrow. In the meantime, you can safely go into the db and just put a string in the dag.bundle_version column. That will unblock you for now.

(edit: added .git to the host - we need that!)