Indicate warnings or other "non-failing" indicators in Airflow UI and callbacks

[matthewblock]

Sometimes a task shouldn't fail but could contain a warning that the user should know about and fix; for example, a deprecation warning. These warnings don't "bubble up" anywhere in the Airflow UI.

Making warnings more visible could have the impact of making upgrades (like the upcoming 3.0 upgrade) much easier for developers.

Possible but not optimal current solutions:

• You can log a warning-level task log, but aggregating all warning-level task logs in the Airflow UI is (as far as I know) not possible.
• You could also customize the log handler to handle WARNING-level messages across all DAGs, but sending these messages to Slack or otherwise might be expensive within this process
• Aggregate using third party tool

A solution could look like all otherwise successful task instances in a DAG run labeled by a yellow (or other color) dot instead of a green dot. It could also help to aggregate warning-level or error-level messages in the UI.

Implementation wise - Maybe there could be a separate task instance state (warning) or different UI indicator on the current success state that could be leveraged to visually indicate this situation. If it were a separate warning state, an on_warning_callback could also be added to send more information to the user's existing Slack alert workflow.

[matthewblock]

@merobi-hub @RNHTTR we briefly discussed this during the Airflow DAG writing best practices workshop. Thanks!

[matthewblock]

Some more thoughts:

I'm realizing that for a TaskInstance, both success and failed states can have warnings, where a warning is at least one call of warning.warn() during task execution. Maybe a TaskInstance should have an additional attribute has_warnings if any warnings were issued during its execution.

We can collect any warnings during task execution using the Python built in warnings.catch_warnings context manager, at least when using PythonOperator and similar. We could then set ti.has_warnings and handle these warnings in an on_warning_callback, or even aggregated somewhere else in the UI.

This would be useful to help developers stay on top of Airflow-communicated deprecation warnings and avoid unpleasant surprises when upgrading.

Is there a better alternative to see all deprecation warnings across all DAGs/tasks that currently exists?

[matthewblock]

@kaxil