Add way to view results & outputs of Airflow processes

[MadDenker]

It would be nice to have a "tab" in airflow dag view that allows the user to query/view/analyze the results of the process. Now I have to load that somewhere else. Would just be nice to have it in the Airflow app. For instance, a way to execute a quick SQL query of your data from an ETL process to see that it completed as desired. Or a dashboard of some sort that plots the data that was loaded. Could be implemented by (1) leveraging Jupyter (see image below), or (2) some simple query + graphing code (like a SQL program), or (3) the ability to view an html page that the user can generate in their Airflow process.

[potiuk]

Can you please take a look at the recent discussion and AIP proposal about OpenTelemetry?

Proposal here: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow
Discussion here: https://lists.apache.org/thread/8w5hg72ckzwjkbqd38dzkxo4qmpdv2yt

I believe implementing open-telemetry reporting (with tracing) will allow to use any tool of your choice supporting Open Telemetry to query it in the way you want.

It makes completely no sense for Airflow to implement the graphing and DB solution for that if there is a ready-to-use protocol solution that allows to do it in the way that you already do it for your other apps.

Amy comments/review/support/encouragement there is most welcome. (or constructive criticisim if what is proposed there is not good for you)

[MadDenker]

Thank you for the quick response. The proposal you reference seems to be referencing Airflow generated data, like task durations, landing times, logs, etc. I am looking for a tool that can visualize the data that Airflow is moving, for instance the data that is being loaded in an ETL process. So if I have an airflow process that updates my database every day, I need a way to visualize the loaded data to confirm that indeed a new day's data is in my database. Apologies for any confusion. I welcome more discussion.

[potiuk]

Sure. But Airflow "tab" is not the right solution. Airflow shoul simply provide data and some visualisation/analysis tools that can use the data. Airflow is great in scheduling (including managing retries and re-runs), If you need graphs/charts etc. - use dedicated tools. Same for verification and expectations - actually there are some good tools that perform anomaly detection - https://www.getre.io/ for example (disclaimer - I am advising re_data :)).

[MadDenker]

Looks like they call them 'pills'. I agree that using dedicated tools is the quickest and easiest way to implement. Hence the suggestion of Jupyter or html. I think html is the easiest and quickest to implement, because you can simply add an iframe and link to a user supplied html file. The user can prepare it in their process. Very little code.

[potiuk]

I mean NO CODE. Not Very Little Code.

Airflow is not a tool any more it's not one-stop-gap-do-everything.

It's a framework. It provides APIs so that other tools can integrate with it. We are long past the days that we add stuff to Airflow, we think more what to remove from it and expose for other tools to consume.

We do not want to make decisions whether it is Jupyter or Html or some other solution. It's user choice. And airflow as a framework is in the position to build a rich ecosystem of tools that want to read Airflow's data and do something useful.

That being said. If you think such a tool based on Jupyter/Graphana/Superset or other tools - feel absolutely free to build an release such a tool on your own. Then you can make a PR to https://airflow.apache.org/ecosystem/ to make it available for others.

But this is extremely unlikely such a visualisation will become part of Airflow UI.