AirflowNotFoundException(f"The conn_id {conn_id} isn't defined")

[ghost]

Hi! I have my connections defined in Admin > Connections -- Since 2 days I receive the error: "AirflowNotFoundException(f"The conn_id {conn_id} isn't defined")", even though nothing changed in my connections nor in my dags. It is all the same as tw0 days ago, only that I now get this error.

I am running Airflow 2.2.1 on linux.

[potiuk]

Something must have changed in your deployment. Look at whoever manage your deployment, For sure nothing that could happen without any change.

[Tristan-Zaleski]

You had enough time to respond curtly within the hour but not enough to provide more insight beyond read connection docs? I can respect anyone offering their services free of charge, as most any user such as myself is capable of doing, as you say, but their is a way of doing it respectfully as well. This response, like the others, shows you are more willing to critique than offer help. Just something to think about when choosing to respond or NOT on a general question you say you don't have clear understanding on or time for yourself. What is the point of a Q&A if every answer is just read the docs, as it could be for any problem? Just say politely to provide more detail/logs or move on.

To address the issue @micaelemaria may be experiencing, something to consider is the webserver's usage of the connection. In my case, plugin's code was using the connection to generate a view, but the Pod hosting the webserver required a full scaling down and back up in order to apply changes made manually within the connection view to be detected. Basically, if your webserver is not regularly refreshing its codebase, whether that's for DAG code or plugin as was my case (settable in airflow.cfg under webserver as reload_on_plugin_change = True) it will not recognize your connection. So a general fix might be to bring down and back up your instance of the webserver after you have created the connection and that it is regularly refreshing the instance.

Since your dags are are using the connection, best suggestions I can give without code is ensure you are importing the correct connection/hook modules, your reference to the connection is typed out correctly (for postgres 'postgres_conn_id="myConn_ID" ), and finally that the connection you have created is being shown in your backend server under the connection table. You can view your connections with CLI command:

airflow connections export -

This will print out your connections to the terminal that are currently in your backend.
Hope this provides actual help and insight to the initial question/issue. Here's to doing my 'part'. ;D Feel free to reach out or continue to update if issue persists.

[ghost]

Thank you Tristan for your help. In the end, as I didn't get any answer from airflow, I just used a different solution. But thank you again.

[potiuk]

@Tristan-Zaleski - This is precisely why we have docs actually. But nothing prevent you as a user to help other users here as you did. Demanding answers from airllow maintainers when you can (and provided) some yourself is just demanding too much. Eventually you did exeactly what is best in the discussions.

If there is net enough "specific" information that can indicate what special. condition the user experiences, sending the user to docs for general question "how connections work" is PRECISELY the right answer. This is what our documentaiton is for, so that maintainers do not have to again and again explain the same "how it works".

If there are any specific deatails - maintainers can (and often do if you look at many 10s of questions we answerd in the last month or so) use our knowledge and experience to help users who have specific conditions or problems or possibly experience bugs in Airflow. But in order to do so we need to have enough information to act on. Otherwise we will just repeat what has been already captured in docs. And there are countless times where we learned from such a discussion and either updated the docs ourselves or asked the user to provide a PR with documentation update.

And eventually @Tristan-Zaleski - you did somethign cool - you actually provided some elaboration based on your experience. This is the best way you can pay back as well for the free software you have. Discussions here are mainly for the users helping each other in exactly the way you did, so seems that my approach was working eventually.

I (maintainer) sent the user to general docs which we maintain, you (another) user shared youre experience that could have helped.

All is cool and apparently the problem is solved. Though @micaelamaria it woudl be also great if you share the different solution you found - this way also you might be able to help others who will have similar problens and look for solutions. Could you please explain what was your problem and how you solved it ?

Then we (or you by PR) could possible capture it in the docs so that others can find it there if it's missing.

[potiuk]

I am really genuinly ineterested in knowing that could happen. My knowledge of Airflow does not bring me any closer to knowing what happens when connection stopped working without any changes out of a sudden?

@micaelamaria - can you please explain what happened that you experienced it?

[ghost]

it was a while ago. I stopped using it.

[selimchergui]

Hello, I did have the same problem. In my case it was due to fernetKey is different between pods specially when they are re-deployed (I am using Airflow 2.2.4 deployed on EKS using Helm charts). As a result decrypting connections data from metadata base was not possible.
To resolve this problem I set the fernetKey parameter (or fernetKeySecretName if you use a sealed secret) in Airflow chart (in values.yaml).

[AdamStormhardtGH]

Heads up this issue will be thrown if you accidentally create a PythonOperator with a python_callable which includes brackets.

ie.

task_number1 = PythonOperator(
        task_id="task_name_for_task_number1",
        python_callable=my_function(), # this bit
        provide_context=True,
        dag=dag,
    )

using python_callable=my_function(), will execute on import, which will mean Airflow will attempt to resolve values which have not yet been imported correctly, or initialised.

I had this connection error thrown due to this, since the function included a Bq Hook which had a connection id I needed to reference from the dag.

correct would be:

task_number1 = PythonOperator(
        task_id="task_name_for_task_number1",
        python_callable=my_function, # this bit with the removal of ()
        provide_context=True,
        dag=dag,
    )

Hopefully this helps someone who's seeing this weird error raise. tbh airflow should probably have a check within the PythonOperator to expect a function, not an executed function.

[potiuk]

tbh airflow should probably have a check within the PythonOperator to expect a function, not an executed function.

It's already defined as expected type - and you can have liniting/MyPy to verify it:

    def __init__(
        self,
        *,
        python_callable: Callable,

It's pretty rare to do a runtime check of expected type in Python. I am not sure if this is a good idea to make a specific check here, if there are ways to check it via static checks at linting time.