Skip to content

Non-atomic config publication can leave no or truncated haproxy.cfg on write failure #1509

Description

@hsdfat

Description of the problem

template.writeToDisk publishes config files non-atomically, so a failed write leaves the live config missing or truncated instead of leaving the previous good one in place. Two paths:

  1. --max-old-config-files > 0 — the live file is renamed away (L111) before the new content is written (L127). If that write fails (ENOSPC, EACCES, read-only remount), there is no file at output at all. The old content survives only under its timestamped name and is never restored.

  2. --max-old-config-files = 0 (the default) — os.WriteFile opens with O_TRUNC, so the current config is truncated before the new content lands. A partial write leaves an invalid config.

Validation doesn't help: instance.check() runs haproxy -c -f at instance.go#L696, but its only caller (L373) runs after writeConfig() (L356), so the file is already live by then.

While the file is missing or invalid, reloads fail and a pod restart cannot start haproxy at all.

Expected behavior

A failed write leaves the previously published config intact. Publication is atomic — either the old or the new content is visible, never nothing and never a partial file.

Steps to reproduce the problem

Requires the write to fail, so it isn't reachable from traffic or API input:

  1. Run with --max-old-config-files=2 and let a config be published.
  2. Make the next write fail — fill the filesystem backing HAProxyCfgDir, or remount it read-only.
  3. Trigger an ingress change so the config is rewritten.
  4. haproxy.cfg is gone, only haproxy.cfg.<timestamp> remains, and reloads now fail.

With the default --max-old-config-files=0, step 4 leaves a truncated haproxy.cfg instead.

Scope

Robustness, not a security issue: it needs an I/O failure, and is not remotely triggerable. The controller retries on the next reconcile, so it recovers once the I/O condition clears.

Environment information

HAProxy Ingress version: master @ 14ea46f (v0.17.0-alpha.2), long-standing code path, so released versions are affected too.

Possible fix

Write to a temporary file in the same directory, Sync(), then os.Rename over the target — rename is atomic, and the Sync() surfaces a delayed ENOSPC before publication rather than after. Rotation can keep its current naming and retention by hardlinking (os.Link) the previous file to the timestamped name instead of renaming the live file away, so the live path is never vacated.

I'm happy to send a PR if you agree with the approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions