feat(api,runner): surface sandbox file-descriptor exhaustion as degradedReason - #5002
Open
mu-hashmi wants to merge 13 commits into
Open
feat(api,runner): surface sandbox file-descriptor exhaustion as degradedReason#5002mu-hashmi wants to merge 13 commits into
mu-hashmi wants to merge 13 commits into
Conversation
…endpoint Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
…ce degradedReason Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
|
View your CI Pipeline Execution ↗ for commit d6e6572
💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗ ☁️ Nx Cloud last updated this comment at |
There was a problem hiding this comment.
Pull request overview
This PR introduces a new, additive “degraded” signal on sandboxes to surface host-side file-descriptor exhaustion while the sandbox remains STARTED. It adds API persistence + runner-driven reporting/clearing logic, and regenerates SDK/API clients to expose the new field and runner endpoint.
Changes:
- API: add nullable
sandbox.degradedReason, DB migration, invariant clearing on non-STARTED, and a runner-authenticatedPUT /sandbox/:sandboxId/degraded-reason. - Runner: detect fd-exhaustion signatures from toolbox proxy responses, push the degraded reason once, and probe periodically to clear (with stale expiry); add explicit guard tests ensuring fd-exhaustion never triggers automated recovery.
- SDK/clients/docs: expose
degradedReasonacross generated clients + TS SDK model/docs and add round-trip tests.
Reviewed changes
Copilot reviewed 53 out of 56 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/sdk-typescript/src/Sandbox.ts | Adds degradedReason to TS SDK Sandbox model. |
| libs/sdk-typescript/src/tests/Sandbox.test.ts | Tests DTO round-trip for degradedReason. |
| libs/api-client/src/models/update-sandbox-degraded-reason-dto.ts | Adds TS API-client DTO for update endpoint. |
| libs/api-client/src/models/sandbox.ts | Adds degradedReason to TS API-client Sandbox model. |
| libs/api-client/src/models/index.ts | Re-exports new TS DTO model. |
| libs/api-client/src/api/sandbox-api.ts | Adds TS API-client method for degraded-reason endpoint. |
| libs/api-client/src/.openapi-generator/FILES | Tracks new generated TS model file. |
| libs/api-client-ruby/lib/daytona_api_client/models/update_sandbox_degraded_reason_dto.rb | Adds Ruby DTO model for update endpoint. |
| libs/api-client-ruby/lib/daytona_api_client/models/sandbox.rb | Adds degraded_reason to Ruby Sandbox model. |
| libs/api-client-ruby/lib/daytona_api_client/api/sandbox_api.rb | Adds Ruby client call for degraded-reason endpoint. |
| libs/api-client-ruby/lib/daytona_api_client.rb | Requires new Ruby DTO model. |
| libs/api-client-ruby/.openapi-generator/FILES | Tracks new generated Ruby model file. |
| libs/api-client-python/daytona_api_client/models/update_sandbox_degraded_reason_dto.py | Adds Python DTO model for update endpoint. |
| libs/api-client-python/daytona_api_client/models/sandbox.py | Adds degraded_reason to Python Sandbox model. |
| libs/api-client-python/daytona_api_client/models/init.py | Exports new Python DTO model. |
| libs/api-client-python/daytona_api_client/api/sandbox_api.py | Adds Python client call for degraded-reason endpoint. |
| libs/api-client-python/daytona_api_client/init.py | Exposes new Python DTO model at package root. |
| libs/api-client-python/.openapi-generator/FILES | Tracks new generated Python model file. |
| libs/api-client-python-async/daytona_api_client_async/models/update_sandbox_degraded_reason_dto.py | Adds async Python DTO model for update endpoint. |
| libs/api-client-python-async/daytona_api_client_async/models/sandbox.py | Adds degraded_reason to async Python Sandbox model. |
| libs/api-client-python-async/daytona_api_client_async/models/init.py | Exports new async Python DTO model. |
| libs/api-client-python-async/daytona_api_client_async/api/sandbox_api.py | Adds async Python client call for degraded-reason endpoint. |
| libs/api-client-python-async/daytona_api_client_async/init.py | Exposes new async Python DTO at package root. |
| libs/api-client-python-async/.openapi-generator/FILES | Tracks new generated async Python model file. |
| libs/api-client-java/src/test/java/io/daytona/api/client/model/UpdateSandboxDegradedReasonDtoTest.java | Adds generated Java model test stub. |
| libs/api-client-java/src/test/java/io/daytona/api/client/model/SandboxTest.java | Adds generated Java Sandbox test stub for field. |
| libs/api-client-java/src/test/java/io/daytona/api/client/api/SandboxApiTest.java | Adds generated Java API test stub for endpoint. |
| libs/api-client-java/src/main/java/io/daytona/api/client/model/UpdateSandboxDegradedReasonDto.java | Adds generated Java DTO model. |
| libs/api-client-java/src/main/java/io/daytona/api/client/model/Sandbox.java | Adds degradedReason to generated Java Sandbox. |
| libs/api-client-java/src/main/java/io/daytona/api/client/JSON.java | Registers new Java DTO type adapter factory. |
| libs/api-client-java/src/main/java/io/daytona/api/client/api/SandboxApi.java | Adds generated Java API call for degraded-reason endpoint. |
| libs/api-client-java/.openapi-generator/FILES | Tracks new generated Java files. |
| libs/api-client-go/model_update_sandbox_degraded_reason_dto.go | Adds Go DTO model for update endpoint. |
| libs/api-client-go/model_sandbox.go | Adds DegradedReason field to Go Sandbox model. |
| libs/api-client-go/api/openapi.yaml | Adds endpoint + schema updates to OpenAPI spec. |
| libs/api-client-go/api_sandbox.go | Adds Go client method for degraded-reason endpoint. |
| libs/api-client-go/.openapi-generator/FILES | Tracks new generated Go model file. |
| apps/runner/pkg/services/sandbox_degraded.go | Implements runner degraded tracking, push, probe, clear. |
| apps/runner/pkg/services/sandbox_degraded_test.go | Tests pushInFlight behavior. |
| apps/runner/pkg/runner/runner.go | Wires SandboxDegradedService into runner singleton. |
| apps/runner/pkg/docker/daemon_exec_probe.go | Adds daemon exec probe helper for recovery/clear checks. |
| apps/runner/pkg/common/recovery.go | Documents fd-exhaustion exclusion from recoverable patterns. |
| apps/runner/pkg/common/recovery_test.go | Asserts fd-exhaustion never maps to recoverable actions. |
| apps/runner/pkg/common/degraded.go | Adds fd-exhaustion signature matching + classifier. |
| apps/runner/pkg/common/degraded_test.go | Unit tests for matcher/classifier and truncation. |
| apps/runner/pkg/api/controllers/proxy.go | Adds response sniffing hook for fd-exhaustion detection. |
| apps/runner/cmd/runner/main.go | Initializes and starts degraded tracking service. |
| apps/docs/src/content/docs/en/typescript-sdk/sandbox.mdx | Documents degradedReason in TS SDK docs. |
| apps/api/src/sandbox/services/sandbox.service.ts | Adds service method to update degraded reason. |
| apps/api/src/sandbox/entities/sandbox.entity.ts | Adds DB column + invariant clearing on state changes. |
| apps/api/src/sandbox/entities/sandbox.entity.spec.ts | Tests entity invariant behavior for degradedReason. |
| apps/api/src/sandbox/dto/update-sandbox-degraded-reason.dto.ts | Adds DTO for degraded-reason update payload. |
| apps/api/src/sandbox/dto/sandbox.dto.ts | Adds degradedReason to API Sandbox DTO mapping. |
| apps/api/src/sandbox/controllers/sandbox.controller.ts | Adds runner endpoint PUT :sandboxId/degraded-reason. |
| apps/api/src/sandbox/controllers/sandbox.controller.auth.spec.ts | Adds auth-coverage test entry for new endpoint. |
| apps/api/src/migrations/pre-deploy/1781135992705-migration.ts | Adds pre-deploy migration for degradedReason column. |
Files not reviewed (3)
- libs/api-client-go/api_sandbox.go: Language not supported
- libs/api-client-go/model_sandbox.go: Language not supported
- libs/api-client-go/model_update_sandbox_degraded_reason_dto.go: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+234
to
+258
| s.mu.Lock() | ||
| entry, ok := s.entries[sandboxId] | ||
| if !ok { | ||
| s.mu.Unlock() | ||
| return | ||
| } | ||
| entry.reason = reason | ||
| entry.lastConfirmed = time.Now() | ||
| needsPush := !entry.reported && !entry.pushing | ||
| s.mu.Unlock() | ||
|
|
||
| if !needsPush { | ||
| return | ||
| } | ||
|
|
||
| if err := s.pushReason(sandboxId, reason); err != nil { | ||
| s.log.Warn("Failed to push degraded reason, will retry on next probe tick", "sandboxId", sandboxId, "error", err) | ||
| return | ||
| } | ||
|
|
||
| s.mu.Lock() | ||
| if entry, ok := s.entries[sandboxId]; ok { | ||
| entry.reported = true | ||
| } | ||
| s.mu.Unlock() |
Collaborator
Author
Comment on lines
+271
to
+279
| s.mu.Lock() | ||
| entry, ok := s.entries[sandboxId] | ||
| stale := ok && time.Since(entry.lastConfirmed) > degradedStaleAfter | ||
| s.mu.Unlock() | ||
|
|
||
| if !stale { | ||
| return | ||
| } | ||
|
|
Collaborator
Author
There was a problem hiding this comment.
Fixed in aef5b23 — expiry clear is skipped while a push is in flight and retried next tick, same guard as the healthy-clear path.
Comment on lines
+197
to
+202
| c, err := s.docker.ContainerInspect(ctx, sandboxId) | ||
| if err != nil || c == nil || c.State == nil || !c.State.Running { | ||
| s.drop(sandboxId) | ||
| s.log.DebugContext(ctx, "Dropped degraded tracking for non-running sandbox", "sandboxId", sandboxId) | ||
| return | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
Fixed in aef5b23 — inspect errors are classified: not-found drops tracking, transient errors keep the entry probing (stale-expiry remains the backstop).
Comment on lines
+94
to
+107
| body, err := io.ReadAll(resp.Body) | ||
| if err != nil { | ||
| // Partial read: hand back what was consumed plus the original body so | ||
| // the proxy surfaces the same read error to the client; skip | ||
| // classification. | ||
| resp.Body = &compositeReadCloser{ | ||
| Reader: io.MultiReader(bytes.NewReader(body), resp.Body), | ||
| Closer: resp.Body, | ||
| } | ||
| return | ||
| } | ||
| resp.Body.Close() | ||
| resp.Body = io.NopCloser(bytes.NewReader(body)) | ||
|
|
Collaborator
Author
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
7 issues found across 56 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/runner/pkg/docker/daemon_exec_probe.go">
<violation number="1" location="apps/runner/pkg/docker/daemon_exec_probe.go:38">
P2: This probe redundantly calls `ContainerInspect`, doubling Docker API work in the degraded polling loop.</violation>
</file>
<file name="apps/runner/cmd/runner/main.go">
<violation number="1" location="apps/runner/cmd/runner/main.go:210">
P2: Starting the degraded tracker without retrying the initial API seed can strand pre-existing degradedReason flags after runner restart.</violation>
</file>
<file name="apps/api/src/sandbox/entities/sandbox.entity.ts">
<violation number="1" location="apps/api/src/sandbox/entities/sandbox.entity.ts:454">
P2: This invariant misses pending transitions away from STARTED, so degradedReason stays stale during stop/archive/destroy flows.</violation>
</file>
<file name="apps/api/src/sandbox/services/sandbox.service.ts">
<violation number="1" location="apps/api/src/sandbox/services/sandbox.service.ts:3115">
P1: Silently ignoring non-STARTED degraded-reason updates can permanently suppress the degraded flag after a startup race.</violation>
</file>
<file name="apps/runner/pkg/common/degraded.go">
<violation number="1" location="apps/runner/pkg/common/degraded.go:38">
P2: Bare `EMFILE`/`ENFILE` substring checks can falsely mark healthy sandboxes degraded when toolbox errors echo user-controlled paths.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| Logger: logger, | ||
| Docker: dockerClient, | ||
| }) | ||
| sandboxDegradedService.Start(ctx) |
There was a problem hiding this comment.
P2: Starting the degraded tracker without retrying the initial API seed can strand pre-existing degradedReason flags after runner restart.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/runner/cmd/runner/main.go, line 210:
<comment>Starting the degraded tracker without retrying the initial API seed can strand pre-existing degradedReason flags after runner restart.</comment>
<file context>
@@ -202,6 +202,13 @@ func run() int {
+ Logger: logger,
+ Docker: dockerClient,
+ })
+ sandboxDegradedService.Start(ctx)
+
// Initialize SSH Gateway if enabled
</file context>
Collaborator
Author
Comment on lines
+454
to
+456
| if (this.state !== SandboxState.STARTED && this.degradedReason != null) { | ||
| changes.degradedReason = null | ||
| } |
There was a problem hiding this comment.
P2: This invariant misses pending transitions away from STARTED, so degradedReason stays stale during stop/archive/destroy flows.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/api/src/sandbox/entities/sandbox.entity.ts, line 454:
<comment>This invariant misses pending transitions away from STARTED, so degradedReason stays stale during stop/archive/destroy flows.</comment>
<file context>
@@ -448,6 +451,10 @@ export class Sandbox {
changes.recoverable = true
}
+ if (this.state !== SandboxState.STARTED && this.degradedReason != null) {
+ changes.degradedReason = null
+ }
</file context>
Suggested change
| if (this.state !== SandboxState.STARTED && this.degradedReason != null) { | |
| changes.degradedReason = null | |
| } | |
| if ( | |
| (this.state !== SandboxState.STARTED || this.desiredState !== SandboxDesiredState.STARTED) && | |
| this.degradedReason != null | |
| ) { | |
| changes.degradedReason = null | |
| } |
Collaborator
Author
There was a problem hiding this comment.
Reviewed, not changed: the invariant runs on every entity update write, and stop/archive/destroy flows write real state transitions through it. While state==STARTED with only desiredState pending, the sandbox is still running and the flag is still accurate; it clears on the actual transition.
There was a problem hiding this comment.
Got it — thanks for the clarification.
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/runner/cmd/runner/main.go">
<violation number="1" location="apps/runner/cmd/runner/main.go:210">
P2: Starting the degraded tracker without retrying the initial API seed can strand pre-existing degradedReason flags after runner restart.</violation>
</file>
<file name="apps/api/src/sandbox/entities/sandbox.entity.ts">
<violation number="1" location="apps/api/src/sandbox/entities/sandbox.entity.ts:454">
P2: This invariant misses pending transitions away from STARTED, so degradedReason stays stale during stop/archive/destroy flows.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…tching Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
…dbox Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 55 out of 58 changed files in this pull request and generated no new comments.
Files not reviewed (3)
- libs/api-client-go/api_sandbox.go: Language not supported
- libs/api-client-go/model_sandbox.go: Language not supported
- libs/api-client-go/model_update_sandbox_degraded_reason_dto.go: Language not supported
There was a problem hiding this comment.
2 issues found across 8 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/runner/cmd/runner/main.go">
<violation number="1" location="apps/runner/cmd/runner/main.go:210">
P2: Starting the degraded tracker without retrying the initial API seed can strand pre-existing degradedReason flags after runner restart.</violation>
</file>
<file name="apps/api/src/sandbox/entities/sandbox.entity.ts">
<violation number="1" location="apps/api/src/sandbox/entities/sandbox.entity.ts:454">
P2: This invariant misses pending transitions away from STARTED, so degradedReason stays stale during stop/archive/destroy flows.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
…ntil success Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 55 out of 58 changed files in this pull request and generated 2 comments.
Files not reviewed (3)
- libs/api-client-go/api_sandbox.go: Language not supported
- libs/api-client-go/model_sandbox.go: Language not supported
- libs/api-client-go/model_update_sandbox_degraded_reason_dto.go: Language not supported
Comment on lines
+97
to
+105
| body, err := io.ReadAll(io.LimitReader(resp.Body, bodyCap+1)) | ||
| if err != nil { | ||
| // Partial read: hand back what was consumed plus the original body so | ||
| // the proxy surfaces the same read error to the client; skip | ||
| // classification. | ||
| resp.Body = &compositeReadCloser{ | ||
| Reader: io.MultiReader(bytes.NewReader(body), resp.Body), | ||
| Closer: resp.Body, | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
Fixed in d6e6572 — the consumed prefix is replayed followed by the stored read error itself, so the client observes the same failure instead of a silent truncation. Regression test reads the replayed body and asserts the original error.
…nsensitively Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
This was referenced Jun 11, 2026
vedranjukic
force-pushed
the
main
branch
2 times, most recently
from
June 23, 2026 14:12
2c15a7f to
b40f732
Compare
rajpratham1
approved these changes
Jul 9, 2026
rajpratham1
left a comment
rajpratham1
left a comment
There was a problem hiding this comment.
Nice feature addition. This introduces a clear way to surface sandbox file-descriptor exhaustion without conflating it with recoverable failures.
I like that the implementation spans the full stack consistently: the API, runner, persistence layer, SDKs, and generated clients all stay in sync. The detection logic is intentionally conservative, avoids false positives, and explicitly keeps fd-exhaustion separate from the automatic recovery pipeline. The proxy buffering safeguards, retry behavior, invariant enforcement, and extensive regression tests also make the feature much more robust.
The scope is large, but the changes appear cohesive, well-tested, and aligned around a single capability. I don't see any blocking issues. Nice work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: sandboxes whose host-side filesystem path exhausts file handles now expose a nullable
degradedReason(set by the runner, cleared on recovery, readable via API/clients/TS SDK) - surfacing only, with a dedicated test guaranteeing it can never trigger automated recovery.Description
When the host-side fd budget under a sandbox is exhausted, everything inside breaks at once (dynamic loader failures, exec errors) while the sandbox still reports
started; users debug their own code while the platform is the cause. This PR names that condition end to end:API
sandbox.degradedReason(nullable text, pre-deploy migration) + an entity invariant clearing it on any transition away fromSTARTED(the invariant runs on every entity update write).PUT /sandboxes/:sandboxId/degraded-reasonfor runners, mirroring theupdateSandboxStateguard stack. Setting a non-empty reason on a non-STARTED sandbox returns 409 so the runner knows the push did not land and retries (a silent ignore could permanently suppress the flag after a startup race); clears stay idempotent. DTO requires non-empty when non-null.Runner
EMFILE:/ENFILE:immediately followed by the phrase. Bare EMFILE/ENFILE tokens do NOT classify (filename-shaped text like/tmp/EMFILE.txt: no such filemust never brick-flag a sandbox). Content-type gate is case-insensitive.modifyResponsehook with hard size caps (8KB for >=400, 64KB for 200/process/execute),io.LimitReaderwith consumed-byte restoration, and - on a failed body read - replay of the consumed prefix followed by the original read error, so the client observes the same failure. The hook can never fail the proxied request.SandboxDegradedService: pushes the reason on first sighting (thepushingclaim is taken synchronously before the goroutine spawns), 30s probe loop (daemon exec oftrue) clears on recovery, 30m stale expiry force-clears when probes are indeterminate, restart re-seeds from the API with retry-until-success (capped backoff, context-aware). Probe-clear and stale-expiry both skip while a push is in flight; inspect errors distinguish not-found (drop) from transient (keep probing).recoverableErrorPatterns;recovery_test.goasserts every signature maps toUnknownRecoveryType/!IsRecoverable. This can never trigger restart/recreate.SDK: TypeScript
SandboxmirrorsdegradedReason(dashboard reads the regenerated client directly). Python/Go/Ruby wrapper parity is a deliberate follow-up.Design tradeoffs: detection is traffic-or-probe driven (onset can lag until something touches the sandbox); 30m stale expiry favors self-healing over a stuck flag. Rollout: runner pushes against an older API 404 and are treated as not-pushed (retried), so deploy order is safe in both directions.
Review guide
Core (~20 hand-written files) vs generated (35+ client files across 6 libs; guarded by the CI regen-drift gate - skim):
apps/runner/pkg/services/sandbox_degraded.go- THE risk center: per-sandbox state machine, push/clear/expiry serialization.apps/api/src/sandbox/services/sandbox.service.ts+entities/sandbox.entity.ts- 409 contract + clearing invariant.apps/runner/pkg/common/degraded.go+recovery.go- classifier + the non-recoverability guarantee.apps/runner/pkg/api/controllers/proxy.go- sniffing caps and error replay.Commit map
Related PRs
Independent of the snapshot family. Siblings: #5001 (fd telemetry), #5003 (daemon shell resolution).
Validation
nx test apigreen twice (52 suites).go test ./pkg/services/ ./pkg/common/ ./pkg/api/controllers/in a linux container, incl. dedicated race tests (claim observable immediately; clear-vs-push interleavings) and-raceruns; classifier table tests incl. four filename false-positive shapes and uppercase content type; read-error replay regression test.yarn generate:api-clientidempotent on the committed regen;yarn lint:fix+yarn format(incl.format:pyunder the poetry venv) zero drift;go work syncclean.Closes #4996