Add filesystem notification (FSNotify) support #294

realrajaryan · 2025-09-17T22:33:20Z

Addresses apple/container#141, where containers don't receive filesystem events on mounted volumes, preventing incremental rebuilds and other file-watching features. This PR implements the guest-side components for FSNotify. Host-side implementation in the container repo will complete the pipeline.

Summary:

Add gRPC protocol definitions for filesystem event notifications
Implement guest-side event handler that generates Linux inotify events
Add CLI testing tool (cctl fsnotify) and integration test infrastructure

Sources/Containerization/SandboxContext/SandboxContext.proto

Sources/Integration/VMTests.swift

vminitd/Sources/vminitd/Server+GRPC.swift

dcantah · 2025-09-19T19:07:28Z

@realrajaryan Can you take a peek at the CI fail, it's for the new test added

Sources/Integration/VMTests.swift

jglogan · 2025-10-02T14:25:04Z

Sources/Containerization/SandboxContext/SandboxContext.proto

+  LINK = 2;
+  UNLINK = 3;
+  MODIFY = 4;
+  UNDEFINED = 99;


What does an UNDEFINED event represent? When would such an event be propagated through the gRPC API?

jglogan · 2025-10-02T14:27:07Z

Sources/Containerization/SandboxContext/SandboxContext.proto

+message NotifyFileSystemEventRequest {
+  string path = 1;
+  FileSystemEventType event_type = 2;
+  string container_id = 3;


It's not clear why, but it seems we use snake case for just about everything in this file, except a container ID is containerID 🤷

jglogan · 2025-10-02T14:52:25Z

Sources/Containerization/Vminitd.swift

    }
+
+    /// Send a filesystem event notification to the guest.
+    public func notifyFileSystemEvent(


Since we're using streaming, the API here should be some sort of collection of Com_Apple_Containerization_Sandbox_V3_NotifyFileSystemEventRequest (or a typealias of it).

jglogan · 2025-10-02T14:59:39Z

Sources/Integration/Suite.swift


        let fs: Containerization.Mount = try await {
-            let fsPath = Self.testDir.appending(component: "rootfs.ext4")
+            let imageHash = Self.hashImageReference(reference)


Just use image.digest in place of manually hashing the image reference.

Good idea, I'd suggested doing this verbally but I forgot we have the image object already at this point.

What's good about just using the digest is in the (no idea why we'd do this but) odd scenario where it's the same image, just a different reference/tag, we still wouldn't unpack for no reason if we already had.

jglogan · 2025-10-02T15:01:19Z

Sources/Integration/VMTests.swift

+        }
+        print("MODIFY event succeeded")
+
+        // Wait for events to be processed


Remove all the debug prints before merging.

jglogan · 2025-10-02T15:18:35Z

Sources/cctl/FSNotifyCommand.swift

+import NIOCore
+import NIOPosix
+
+extension Application {


Does this command work? What commands should I use to test it locally?

jglogan · 2025-10-02T15:22:42Z

Sources/Integration/VMTests.swift

+        print("All FSNotify events tested successfully")
+    }
+
+    private func createFSNotifyTestDirectory() throws -> URL {


Do we really need this, or could we instead use createMountDirectory since we're only working with one file? If we do need to create a subdirectory we could do it as part of the test.

jglogan · 2025-10-02T15:24:27Z

Sources/Integration/VMTests.swift

+        print("=== End Container Output ===")
+
+        // Verify that inotify detected the modify event
+        guard inotifyOutput.contains("change") && inotifyOutput.contains("existing.txt") else {


It seems that this should below the delete event, and rather than checking for the presence of any change and any existing.txt, we should be ensuring that the actual output exactly matches the expected output.

dcantah · 2025-10-02T21:22:57Z

vminitd/Sources/vminitd/ManagedContainer.swift

+        private func readResponseFromChild(socket: Int32) throws -> (UInt32, UInt8)? {
+            // Read event_id:4
+            var eventID: UInt32 = 0
+            guard read(socket, &eventID, 4) == 4 else { return nil }


I don't see a mechanism to get notified of when we have events to read so it seems we're just doing blocking read(2)'s on a swift concurrency thread. For low core count guests this will likely cause us to hang 😅. You're not supposed to block any of the swift concurrency threads, as they're always supposed to be making forward progress or hitting some suspension point. We could use nio for these socket reads and that'd at least get us into shape

jglogan

Let's get the concurrency working so that it's not @unchecked. See Danny's suggestion about using NIO for the socket communication.

jglogan · 2025-10-02T21:29:53Z

vminitd/Sources/vminitd/ManagedContainer.swift

+    private var namespaceWorker: NamespaceWorker?
+
+    /// Worker child process that runs in container's namespace for filesystem operations
+    private final class NamespaceWorker: @unchecked Sendable {


A few things here:

Is there a reason for nesting this class? It's adding about 350 lines to ManagedContainer, which makes it harder for someone unfamiliar with the code to reason about either. I can move it outside of ManagedContainer and still compile it, so let's move it into its own file.

I could see this being called NamespaceWorker if it facilitated any kind of work taking place between the guest agent and namespace, which could be a useful abstraction later on. For now let's change the name to describe specifically the work being done.

We need to manage the concurrency between guest agent, worker parent, and worker child such that @unchecked isn't needed for Sendable. In particular, we can't do blocking socket read() calls in Task context.

jglogan · 2025-10-02T21:30:46Z

vminitd/Sources/vminitd/ManagedContainer.swift

+                    throw ContainerizationError(.internalError, message: "Child process failed to start")
+                }
+
+                if signal == 0xFF {


Extract constants for these and then you can get rid of the comments.

Also, let's not call it a "signal" as that has a specific meaning that is not this.

jglogan · 2025-10-02T21:31:30Z

vminitd/Sources/vminitd/ManagedContainer.swift

+                }
+
+                // Start response reader task
+                self.responseReaderTask = Task { [weak self] in


This is no different from:

self.responseReaderTask = Task { await self.readChildResponses() }

… failing)

…reading restrictions

dcantah · 2025-10-09T21:22:16Z

vminitd/Sources/vminitd/FilesystemEventWorker.swift

+import Glibc
+#endif
+
+final class FilesystemEventWorker: @unchecked Sendable {


We should try and make this actually sendable. Just shove all the mutable state in a Mutex (we can reuse the one we have for pendingEvents, but just make a struct with all of this data on it, and call the field state or something similar.

dcantah · 2025-10-09T21:27:10Z

vminitd/Sources/vminitd/FilesystemEventWorker.swift

+        let parentSocket = sockets[0]
+        let childSocket = sockets[1]
+
+        let pid = fork()


I'm skeptical this will work alright. Here's a decent article on some of the pitfalls https://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/

We'd ideally want to exec a new process that we pass the other half of the socket pair to.

(the new process does not physically need to be a new binary, we can just edit vminitd to have a fs-notify subcommand and fork+exec ourselves)

realrajaryan force-pushed the users/realrajaryan/fsnotify branch from f89a058 to 128421e Compare September 17, 2025 22:35

realrajaryan requested a review from jglogan September 17, 2025 22:36

realrajaryan force-pushed the users/realrajaryan/fsnotify branch from 128421e to 816c698 Compare September 17, 2025 23:14

jglogan reviewed Sep 17, 2025

View reviewed changes

Sources/Containerization/SandboxContext/SandboxContext.proto Outdated Show resolved Hide resolved

Sources/Integration/VMTests.swift Outdated Show resolved Hide resolved

jglogan reviewed Sep 17, 2025

View reviewed changes

vminitd/Sources/vminitd/Server+GRPC.swift Outdated Show resolved Hide resolved

jglogan reviewed Sep 17, 2025

View reviewed changes

vminitd/Sources/vminitd/Server+GRPC.swift Outdated Show resolved Hide resolved

cfergeau mentioned this pull request Sep 18, 2025

cannot receive inotify on shared directories when using AppleHV containers/podman#22343

Open

jglogan reviewed Sep 22, 2025

View reviewed changes

Sources/Integration/VMTests.swift Outdated Show resolved Hide resolved

jglogan requested changes Sep 24, 2025

View reviewed changes

Sources/Integration/VMTests.swift Outdated Show resolved Hide resolved

Sources/Integration/VMTests.swift Outdated Show resolved Hide resolved

jglogan reviewed Sep 25, 2025

View reviewed changes

Sources/Integration/VMTests.swift Outdated Show resolved Hide resolved

realrajaryan force-pushed the users/realrajaryan/fsnotify branch 4 times, most recently from b33caa9 to 07a7db3 Compare October 2, 2025 05:49

jglogan reviewed Oct 2, 2025

View reviewed changes

dcantah reviewed Oct 2, 2025

View reviewed changes

jglogan requested changes Oct 2, 2025

View reviewed changes

realrajaryan added 6 commits October 8, 2025 17:05

Add filesystem notification (FSNotify) support

0bbc056

fix comments

8bf23a7

fix integration test issues

e51e85d

implement thread-per-container namespace isolation for FSNotify (test…

f8f80f2

… failing)

switch to fork-based FSNotify architecture to resolve setns() multith…

e48124c

…reading restrictions

fix comments

eb651c5

realrajaryan force-pushed the users/realrajaryan/fsnotify branch from 11e1720 to eb651c5 Compare October 9, 2025 00:07

dcantah reviewed Oct 9, 2025

View reviewed changes

Add filesystem notification (FSNotify) support #294

Are you sure you want to change the base?

Add filesystem notification (FSNotify) support #294

Conversation

realrajaryan commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcantah commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcantah Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcantah Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jglogan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

realrajaryan commented Sep 17, 2025 •

edited

Loading

dcantah commented Sep 19, 2025 •

edited

Loading

dcantah Oct 2, 2025 •

edited

Loading

dcantah Oct 2, 2025 •

edited

Loading