Skip to content

Nix daemon caches wrong narHash after manual flake input rev edit #15198

@michalrus

Description

@michalrus

Describe the bug

On a self-hosted GitHub Actions runner using the Nix daemon, if a flake input revision is updated by editing rev in both flake.nix and flake.lock without updating narHash or running nix flake update <input>, the old narHash is assumed to be correct also for the new rev by the daemon. After subsequently running nix flake update <input> (which updates the narHash), Nix fails with error: mismatch in field 'narHash' of input – the daemon refuses to accept the corrected hash for that rev.

Steps To Reproduce

  1. On a self-hosted GitHub Actions runner with a Nix daemon, run a nix build on the original (rev, hash) pair – the cache is initialized correctly now.
  2. Update an input's rev in flake.nix and flake.lock (manually), but do not run nix flake update <input>.
  3. Run any Nix command that evaluates the flake (e.g. nix build) on this new commit – the cache is poisoned now.
  4. Later, run nix flake update <input> so flake.lock updates the narHash for that new rev, and commit and push.
  5. Have CI run the same Nix command again on the same runner.

Expected behavior

Once flake.lock is updated with the correct narHash, the evaluation/build should succeed using the new narHash.

Actual behavior

The last command fails with:

error: mismatch in field 'narHash' of input 
      '{"__final":true,"lastModified":1770733010,"narHash":"sha256-4cKMeE/8GptnxWnB6LliJy3gCvjVI34TXTwQHEsHcUA=","owner":"…","repo":"…","rev":"5297592395d1dbd46e88247e459896838c854340","type":"github"}', 
  got '{"__final":true,"lastModified":1770733010,"narHash":"sha256-YR+cemfm4AMEJcUEr/tm0cis3rWKQ0dPFbjwhEVrV94=","owner":"…","repo":"…","rev":"5297592395d1dbd46e88247e459896838c854340","type":"github"}'

It appears the daemon has cached the old narHash for the new rev, and treats the updated narHash as invalid even though it matches the upstream content.

Metadata

nix-env (Nix) 2.25.4

Additional context

  • System: self-hosted GitHub Actions runner on Linux

Why this is a problem

This permanently poisons the cache for that rev on the runner. Our mitigation is to reboot the runner (which clears the bad state). This is easy to trigger by mistake when someone edits flake.nix but forgets to run nix flake update <input>.

Arguably a +2–2 diff shouldn’t require a reboot or other manual intervention.

The only way a developer can work around it without SRE intervention is to use a different commit of <input>, which is not always possible.

Checklist


Add 👍 to issues you find important.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions