Replies: 2 comments
-
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Converted to a discussion - it's not an airflow issue. The easiest will likely be to override SFTPOperator with your own CustomSFTPOperator and override it's execute method where you will set the self.sftp_hook just before calling super.execute(). Smth like: def execute(context):
self.sftp_hook = ....
super().execute(context=context) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Apache Airflow version
2.9.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
This is an issue where DAG scheduler errors are showing up in our local development airflow containers due to missing connections.
To give some background: We had to work around and override the
retrieve_file
method for SFTPHook due to performance issues with large CSV file loads.This is what the override looks like (setting max_concurrent_prefetch_requests to 64):
This then we use it in our DAG like so:
However, the DAG scheduler in a developer environment which doesn't have the connection ID defined in the connections will show an error:
I believe this is due to SSHHook calling
get_connection()
in__init__
:airflow/airflow/providers/ssh/hooks/ssh.py
Line 154 in 3583329
What you think should happen instead?
Is there a way SSHHook can handle fetching a connection differently so that it catches the missing conn_id on DAG runtime instead of scheduler check interval?
How to reproduce
Operating System
Windows 11
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else?
This problem is consistent between many developers local machines
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions