Change stdout and stderr log access modes

[THISisZOLI]

Starting Airflow components using the --stdout and --stderr arguments redirects the output streams to a log file. Currently the access type for these log files is w+, as you can see below (might not include all affected components).

Scheduler:

airflow/airflow/cli/commands/scheduler_command.py

Line 68 in de6938e

with open(stdout, 'w+') as stdout_handle, open(stderr, 'w+') as stderr_handle:

Flower:

airflow/airflow/cli/commands/celery_command.py

Line 70 in de6938e

with open(stdout, "w+") as stdout, open(stderr, "w+") as stderr:

Celery:

airflow/airflow/cli/commands/celery_command.py

Line 180 in de6938e

with open(stdout, 'w+') as stdout_handle, open(stderr, 'w+') as stderr_handle:

Webserver:

airflow/airflow/cli/commands/webserver_command.py

Line 452 in de6938e

with open(stdout, 'w+') as stdout, open(stderr, 'w+') as stderr:

Triggerer:

airflow/airflow/cli/commands/triggerer_command.py

Line 42 in de6938e

with open(stdout, 'w+') as stdout_handle, open(stderr, 'w+') as stderr_handle:

My problem with this access mode is that it can create sparse files. If the log file is moved or deleted, a new one will be created and the stdout/stderr will be written into it, but not from the start of the file. Instead if the original log file was x characters long, the new log file will contain x null bytes, and the new lines will be appended after that.

This behaviour prevents us from rotating the log files properly. The only solution for it currently without changing the code is restarting the components after each log rotation, which is not really feasible.

I propose changing the access mode for these output files from w+ to a. As I've seen it from my tests it still creates the log file if it doesn't exist, but it does not print null bytes at the beginning of new files if the old one is deleted.

The problem with this solution is that old output files are not automatically deleted (or, to put it better, overridden) at component start, which might result in very big output files in the long run. To preserve this behaviour, we could do something like this after opening the file:

with open(stdout, "a") as stdout, open(stderr, "a") as stderr:
    stdout.seek(0)
    stdout.truncate()
    stderr.seek(0)
    stderr.truncate()

[potiuk]

Why don't you provide a PR fixing it ? I think the solution with "append" mode and truncation sounds lke a good idea.

[enoperm]

Are you sure about the truncation part? This issue arises on rotation, which I believe is in itself a common solution to log files getting big in the long run, and unlike truncating on open, it shouldn't lose any already written log records. I think simply switching to append mode in itself would be a nicer solution.

[enoperm]

That said, I have played around with a simplified version of the code, and indeed, the current version truncates, changing it may cause unforeseen problems in existing deployments without log rotation.

#!/usr/bin/env python3

import daemon
import time

with open('stdout.log', 'w+') as stdout_handle:
    ctx = daemon.DaemonContext(
        stdout=stdout_handle,
    )
    with ctx:
        i = 0
        while True:
            time.sleep(1)
            print(f"record #{i}")
            i += 1

I think I'll prepare a PR with truncation included, but I really think truncating logs should be a job for logrotate.

[enoperm]

See: #25363

[potiuk]

Closeed #25363 as we already had #25253 opened

[enoperm]

Oh, didn't see that one.