JSON formatter functionality

[nikhilkarve]

Hello everyone..I am new to airflow so I am not sure if this exists but from my own research I was not able to find it so please free to let me know if I am missing something

I want the airflow logs to be in JSON. I can see a config in the elasticsearch section which is to format logs in JSON but I am not using ES so setting that doesn't work for me. I have seen some solutions using customer logging mechanism to achieve the same, but there is no solid implementation of it or the guide to do so. I want the logs to be displayed in JSON in the UI and also in the files.

Any help in appreciated.

I tried implementing a custom formatter, but for some reason airflow is not picking that up and displaying the logs in the UI as it is. I want the logs displayed in the UI in JSON too.

[potiuk]

Just to set your expectations - You are mainly on your own, I think such use -case was never foreseen and It is not likely to work as the UI has a number of assumptions about streaming the logs and where it reads them - you will have to dig deeper and understand (rather complex) airflow logging system if you want to modify it, but I think what you are trying to do is a bad idea - things will continue breaking with newer versions of airflow, so you are probably better off with doing it differently rather than modifying Airflow UI.

Your starting point is here https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html but you will need to dig deep in airflow logging code

[potiuk]

Airflow webserver reads the logs from workers or remote source (depends on logging handler) - so where and how it reads the logs from depends on many factors (and also it might be different for running tasks to allow streaming during task logging or after task is finshing to sometimes pull the logs from remote. Plus a lot of work is done to combine logs from different components (like scheduler or triggerer) to display it in the UI - it's all pretty custom code you would need to dig in and find your way around.

Definitely changing formatter is not going to cut it, but the only way for you to find out what's needed is to dig deeper. I am afraid.

[nikhilkarve]

Ah, I see. Thank you!:)
Would it be possible for you to help me understand what the person in the answer has achieved? https://stackoverflow.com/a/55502536
From what I understand, he changed the handlers to the customer JSON handlers. I may have misunderstood what he did there, so just trying to correct my understanding. As it is mentioned in that StackOv question, it says that they wanted logs to stdout in JSON, and the person who answered also says that, they got the DAG logs and "output" in JSON?

Edit : Not in the UI, but is it possible to get the airflow logs in JSON format - using airflow logging system but adding a custom handler at least somewhere? (example: log files(?))

[potiuk]

The answer is from 2020 - it was a different airflow than now.

[potiuk]

Edit : Not in the UI, but is it possible to get the airflow logs in JSON format - using airflow logging system but adding a custom handler at least somewhere? (example: log files(?))

See the link I posted first. Foilow advanced logging.

[nikhilkarve]

I am trying advanced logging. I can see the message saying "Successfully imported user-defined logging config from log_config.LOGGING_CONFIG".

LOGGING_CONFIG is the modified dictionary which contains custom JSON handlers. Problem is I am not sure where it is writing those as I don't see any change anywhere.

I will try a couple more things and see if I can find anything more about the problem and then will post it..thanks for the help!