Skip to content

Keep preparing sandboxes through deployment changes - #977

Merged
SaladDay merged 5 commits into
mainfrom
keep-preparing-sandboxes-through-deployment-changes
Nov 19, 2026
Merged

SaladDay merged 5 commits into
mainfrom
keep-preparing-sandboxes-through-deployment-changes

Conversation

@SaladDay

@SaladDay SaladDay commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • A deployment change no longer disturbs the node's generation bookkeeping: a failed collection of an old generation no longer leaves a collecting mark that could later turn a Deployment frame into an ownership error and end the node, and the node helper treats collecting a generation it never prepared (no local files) as a no-op instead of failing on every retention heartbeat.
  • Tests pin the expected behavior from the 11-09 incident: a deployment change during a Session's preparation leaves that allocation on its generation and specification while Sessions created afterwards use the new generation; a Session whose compute never starts fails its unrun first input with an error instead of reading idle.
  • The 11-09 trigger itself could not be determined: its Core logs are gone, the node did not restart in that window, and five live repros on main (resource and network changes during preparation, a node restart during preparation) did not interrupt any preparation. Every path where a first input cannot run ends the Session failed (Fail pending input when an archive expires its Environment #951 removed the idle fork).

Known and deliberately not fixed in this PR (review findings, medium):

  • Clearing collecting after a failed removal lets a generation Core already released be prepared and reported again, while the node protocol says a dropped generation is never prepared or used again.
  • A generation whose preparation was killed after its lease file was created but before its lease identity was written still fails collection on every exchange until the file is removed by hand.

Test plan

  • node package go test -race: a failed collection leaves the generation usable
  • deploy/node unittest: collecting a never-prepared generation is a no-op; a bare lease without identity still fails loudly
  • integration: deployment change during preparation keeps the allocation's generation; unstarted compute fails the Session's unrun first input
  • go vet, make check-docs
  • Independent review: one blind reviewer (no blocker/high; two medium listed above, two low)
  • CI

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@SaladDay
SaladDay merged commit e6788d1 into main Nov 19, 2026
19 checks passed
@SaladDay
SaladDay deleted the keep-preparing-sandboxes-through-deployment-changes branch November 19, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant