Airflow Random Failure with SQS

[iamharshit]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.7.3

What happened?

We have an Airflow job that sends a large number of messages (>100k) to SQS. The job mostly works perfectly, but once in a while, it fails for no apparent reason. When retriggered, it succeeds automatically, even though there is no change in the underlying code.

Error message:
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.

Is there a solution to resolve such random failures?

What you think should happen instead?

Possible some bug internally in airflow.

How to reproduce

Internal to my firm - confidential

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

You need to look at the thing that is failing - i.e. your bash command than randomly fails. Why it fails only you can answer also because it's "company confidential" what you run.

You can add retries to tasks (see the documentation) to mitigate such random failures, but you should rather find out why they are failing in the first place (and again - you are the only person who can investigate it I am afraid).

Converted to a discussion - please avoid to open issues and if you want to discuss your issues and get help with solving them rather than reporting "airflow issue", use discussions instead.