Unify filesystem storage accounting

[kmatasfp]

Implements GOL-416.

Summary

• replace logical filesystem byte deltas, storage permits, and custom hard-link accounting with one AgentFilesystems / AgentFilesystem lifecycle
• use managed XFS project quotas as the authority for per-agent allocated-byte usage, filesystem-object usage, and limits, while using fresh statvfs observations for physical node capacity
• route WASI P2 and P3 direct operations and asynchronous streams through shared lifecycle admission and failure classification
• move filesystem creation, initial-file materialization, replay settlement, and cleanup under the AgentFilesystem lifecycle
• meter authoritative allocated-byte observations over the same ConcurrentAgentPermit window as linear memory, with compute, memory, and filesystem metering configured independently

Behavior

AgentFilesystem now owns each resident agent's ephemeral filesystem across creation, initial-file materialization, oplog replay, execution, and unload. Startup removes and verifies stale local resources before creating an unmanaged directory or managed XFS project. Replay-owned filesystem effects settle before startup publishes the runtime. Unload seals new filesystem work, tears down runtime resources, deletes the filesystem, and verifies cleanup before publishing Unloaded.

Managed XFS validates project-quota support, project identity and inheritance, reflink support, permissions, and exclusive ownership of the managed root. Project observations supply authoritative allocated bytes and filesystem-object counts. The executor installs the public allocated-byte limit together with an internally derived object limit. Read-only and read-write initial files both count toward quota and billing. Initial-file revisions are staged, validated, and installed atomically so a failed update does not publish partial files. Reducing either limit below current usage suspends and unloads the agent until reconstruction fits the current policy.

Unmanaged filesystems remain available when project quotas are not configured, but reject finite filesystem limits because they cannot provide authoritative enforcement. Unsupported allocation observation disables only the filesystem metering dimension; compute and memory metering continue independently.

Physical capacity remains separate from project usage. The executor obtains available bytes and filesystem objects from fresh statvfs observations. On physical exhaustion it may unload only eligible loaded-idle agents, and recognizes reclaimed capacity only after deletion and another observation. Agent quota exhaustion never triggers pressure eviction.

P2 and P3 preserve errno, completed prefixes, and effect evidence until classification. Retries are bounded and apply only to a proven unwritten suffix or an operation whose postcondition proves no effect. Unknown effects and terminal infrastructure failures invalidate the disposable local runtime while leaving durable work available for reconstruction. XFS reports ENOSPC for both project-quota and physical exhaustion, so classification uses fresh quota, capacity, and backend-health observations instead of errno alone.

Enabled storage billing integrates ordered absolute allocated_bytes observations prospectively. Storage and memory start and stop together at one monotonic timestamp while the worker owns its concurrent-agent permit. Active execution, host calls, host I/O waits, and replay are billable while retaining the permit. Loaded-idle, unloaded, and durably sleeping workers accrue nothing. Unload and filesystem deletion happen outside the billing window.

Compute, memory, and filesystem usage metering are independent switches and all default to disabled for OSS and self-hosted deployments. The worker executor and standalone golem server honor GOLEM__RESOURCE_USAGE_METERING__COMPUTE, GOLEM__RESOURCE_USAGE_METERING__MEMORY, and GOLEM__RESOURCE_USAGE_METERING__FILESYSTEM. These switches affect measurement and export, not quota enforcement, memory admission, or physical-pressure recovery.

The superseded logical accounting path has been removed across oplog types, protobuf, WIT, CLI schemas, SDK bindings, documentation, and the built-in OTLP exporter. ConcurrentAgentPermit remains because it defines the execution and billing window; it is unrelated to the removed filesystem byte permits.

[netlify]

✅ Deploy Preview for golemcloud canceled.

Name	Link
🔨 Latest commit	471806f
🔍 Latest deploy log	https://app.netlify.com/projects/golemcloud/deploys/6a970876f3c2aa000884ebea

[vigoo]

You need to rebuild the built-in plugins (cargo make build-plugins) because of the WIT change

[vigoo]

Both filesystem APIs (p2 and p3) expose hard-link creation without charging another copy (link_at), which is correct because both names refer to one inode, but unlink releases the file's entire size after any successful unlink without tracking reference counts.

[vigoo]

One more review finding to check:

Read-write provisioning writes directly to the destination:

• Cache hits use tokio::fs::copy
• Cache misses File::create the final path and stream into it

Either operation can fail after leaving a partial regular file. On a revision-update retry, update_filesystem accepts any existing regular file as NoChanges, without validating its expected size or content.

The resulting sequence is:

1. A revision update reserves N bytes for a new read-write file.
2. The download writes K < N bytes and fails.
3. The reservation is rolled back because no successful growth was reported.
4. The partial K-byte destination remains.
5. Retry sees a regular file and reports NoChanges.
6. The update can succeed with those K bytes absent from quota, capacity, billing, and recovery accounting.

Required fix: write/copy into a sibling temporary file, delete it on failure, and atomically rename it only after complete success. Do not accept an arbitrary pre-existing regular destination without validating that it represents a completed provisioning operation.

[kmatasfp]

All of the above you mentioned @vigoo plus some other things I ran into myself reviewing this are the reason I changed the direction and abandoned the golem internal quota/disk usage tracking/management and now building out a proper FS support I mentioned during the standup. GOL-416 issue description contains details of the new approach

[vigoo]

Good to go once the conflict is resolved. Make sure you use forked wit-bindgen for regenreating the moonbit bindings (there is a proper regen script for it somewhere)