Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCSToGCSOperator source_objects failing to parse xcom #42391

Open
1 of 2 tasks
blaklaybul opened this issue Sep 20, 2024 · 1 comment
Open
1 of 2 tasks

GCSToGCSOperator source_objects failing to parse xcom #42391

blaklaybul opened this issue Sep 20, 2024 · 1 comment
Labels
area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:google Google (including GCP) related issues

Comments

@blaklaybul
Copy link

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

10.15.0

Apache Airflow version

2.7.3

Operating System

debian

Deployment

Astronomer

Deployment details

Using astro runtime 9.15.0

What happened

The GCSToGCSOperator is throwing an error when trying to pass task output into the source_objects field.

 File "/usr/local/lib/python3.11/site-packages/airflow/providers/google/cloud/transfers/gcs_to_gcs.py", line 211, in __init__
    if source_objects and any(WILDCARD in obj for obj in source_objects):
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/models/xcom_arg.py", line 268, in __iter__
    raise TypeError("'XComArg' object is not iterable")

What you think should happen instead

This operator should be able to handle xcom, and we have no issue using the GCSToBigQueryOperator in the same way

How to reproduce

@task
def return_files():
    files = ['a','b','c','d']
    return files

files = return_files()

 extract_files = GCSToGCSOperator(
        task_id="fetch_data",
        source_bucket="source_bucket_name",
        source_objects=files,
        destination_bucket="dest_bucket_name",
        destination_object="foo",
        dag=dag,
    )

files >> extract_files

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@blaklaybul blaklaybul added area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Sep 20, 2024
Copy link

boring-cyborg bot commented Sep 20, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@dosubot dosubot bot added the provider:google Google (including GCP) related issues label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:google Google (including GCP) related issues
Projects
None yet
Development

No branches or pull requests

1 participant