Implement graceful shutdown for Garnet server ( #1382 )

[yuseok-kim-edushare]

📌 Summary

This PR introduces a robust, graceful shutdown mechanism for Garnet, specifically addressing the data loss issues identified in #1382.

The implementation ensures a deterministic, four-phase shutdown sequence: Quiesce → Ingress Throttling → Connection Draining → Final Data Persistence (AOF/Checkpoint). This ensures the server shuts down safely without losing in-flight data, especially when running as a Windows Service or under container orchestrators.

---

🛠 Key Changes

1️⃣ Graceful Shutdown Implementation (Lifecycle Management)

• GarnetServer.ShutdownAsync: Centralized shutdown interface that orchestrates: quiescing all sessions, stopping listeners, waiting for active connections within a configurable timeout, and committing final state (AOF or Checkpoint — one call only, per reviewer guidance).
• noSave flag: ShutdownAsync(noSave: bool) allows callers to skip data finalization entirely — used during forced/OS-initiated shutdowns where the cancellation token is already signalled.
• Bounded data-finalization timeout: FinalizeDataAsync runs under an independent 15-second CancellationTokenSource so the final AOF commit/checkpoint always completes within the host's shutdown budget, regardless of the external token state.
• Worker & Entry Point Integration: Updated the Windows Service worker (Garnet.worker) and the CLI entry point (Program.cs) to trigger the graceful shutdown sequence upon receiving termination signals (SIGINT, SIGTERM).

2️⃣ Quiesce Mechanism (Pre-drain Ingress Gate)

• BeginQuiesce / IsQuiescing: Added to IGarnetServer and implemented in GarnetServerBase using an atomic flag. GarnetServer.ShutdownAsync calls BeginQuiesce on all servers and the SubscribeBroker before stopping listeners.
• Session-level rejection: RespServerSession checks IsQuiescing on its server; if quiescing, it completes the in-flight command, replies with a LOADING error on the next message, and closes the connection.
• Pub/Sub gate: SubscribeBroker drops Publish/PublishNow calls while quiescing, preventing new fan-out during the drain window.

3️⃣ Configurable Shutdown Timeout

• --shutdown-timeout <seconds> CLI option: New command-line argument parsed by both Program.cs and Garnet.worker/Program.cs.
• GarnetServerOptions.ShutdownTimeoutSeconds: Exposes the timeout (default: 5 s, minimum recommended: 5 s to match Windows SCM pre-kill wait) so it is available to the host.
• Host budget auto-calculation: Garnet.worker/Program.cs pre-parses --shutdown-timeout and sets HostOptions.ShutdownTimeout = connectionDrainTimeout + 20 s so the .NET host (and Windows SCM via WindowsServiceLifetime) waits long enough for both connection draining and data finalization to complete.
• Worker receives typed timeout: Worker(string[] args, TimeSpan shutdownTimeout) uses the parsed value for ShutdownAsync instead of a hardcoded 5 s.

4️⃣ Enhanced Connection Handling (Infrastructure)

• Listener Control: StopListening on IGarnetServer stops accepting new connections at the socket level while maintaining existing ones.
• Removed isListening flag: The volatile bool isListening guard has been removed from GarnetServerTcp — tests confirmed that catching ObjectDisposedException / SocketError.OperationAborted on the closed listen socket is sufficient to terminate the accept loop cleanly, with no additional flag needed.
• Socket management alignment: GarnetServerTcp was updated to align with concurrent upstream refactoring of socket lifecycle management.

5️⃣ Comprehensive Test Coverage

• ShutdownDataConsistencyTests — new test class covering data-persistence scenarios across the full AOF/Checkpoint matrix:

  • CheckpointThenAofCommit_DataConsistencyTest
  • AofCommitThenCheckpoint_DataConsistencyTest
  • AofCommitOnly_DataConsistencyTest
  • CheckpointOnly_DataConsistencyTest
  • NoFinalization_DataConsistencyTest
  • CheckpointThenMoreWritesThenAofCommit_DataConsistencyTest
  • AofCommitThenMoreWritesThenCheckpoint_DataConsistencyTest

• GarnetServerTcpTests — updated with graceful-shutdown behavioral tests:

  • StopListeningPreventsNewConnections
  • StopListeningIdempotent
  • StopListeningDuringActiveConnectionAttempts
  • ShutdownAsyncCompletesGracefully
  • ShutdownAsyncRespectsTimeout
  • ShutdownAsyncRespectsCancellation
  • ShutdownAsyncWithAofCommit

---

💡 Why This Approach?

• Data Integrity: A single final AOF commit or checkpoint (not both) after all connections are drained guarantees the store state is fully persisted. The bounded 15 s finalization window ensures this always completes within the host's shutdown budget.
• Deterministic Ingress Shutdown: Quiescing sessions before closing the listen socket eliminates the race where in-flight commands arrive after the drain window starts, enabling a stricter "no new writes after shutdown signal" invariant.
• Force-shutdown Safety: Passing noSave: true when the OS cancellation token is already triggered avoids a lengthy persistence step during hard kills, letting the process exit promptly.
• Configurable Timeout: Making the connection-drain timeout a CLI option means operators can tune it to their workload without recompiling, while the automatic host-budget calculation prevents the Windows SCM from force-killing before finalization finishes.
• Minimal Invasive Change: Modifications to Program.cs replace Thread.Sleep with a CancellationTokenSource-based wait, maintaining codebase consistency while enabling clean signal handling. The isListening flag removal simplifies GarnetServerTcp with no correctness regression, as proven by the new socket tests.
• Architectural Alignment: The quiesce and shutdown designs extend existing Garnet abstractions (IGarnetServer, GarnetServerBase, SubscribeBroker) for a consistent shutdown experience across standalone and Windows Service hosting.

---

✅ Related Issues

• Closes: #1382

• Resolves: #1390

• Reflects Discussion: r2535724513

• Refactored: #1448

  Performance & Reliability Optimizations

  • Allocation Reduction: Active connection counting uses C# 7.0+ pattern matching (is) over LINQ (OfType<T>), reducing GC overhead on the critical shutdown path.
  • Signal Synchronization: ManualResetEventSlim / CancellationTokenSource-based waiting in the main entry point replaces Thread.Sleep for lightweight, efficient shutdown signal handling.
  • Log Level Tuning: Verbose shutdown-path logging downgraded from Information to Debug to reduce I/O overhead during the shutdown hot path.

---

The key additions since the original description are:

• Quiesce mechanism (new pre-drain ingress gate on sessions and pub/sub)
• noSave flag on ShutdownAsync for forced-shutdown fast-exit. Furthermore Requesting support for SHUTDOWN and CLIENT PAUSE, CLIENT UNPAUSE #1004 's Shutdonw Command request can be use this logic
• Configurable --shutdown-timeout
• Removal of isListening flag from GarnetServerTcp (proven unnecessary)
• ShutdownDataConsistencyTests replacing the older GracefulShutdownTests class with a full AOF/Checkpoint matrix
• Bounded 15 s finalization window in FinalizeDataAsync

[yuseok-kim-edushare]

Looks like my PR has some conflicts after the latest merged #1648 , #1646 , #1645 PRs
Since I’ll be on PTO tomorrow, I’ll take a look and fix it then.

[yuseok-kim-edushare]

​I’ve updated my code to align with the recent socket handling changes from @hamdaankhalid’s PRs ( #1646, #1648 ).

​I believe this PR is now in a mergeable state, as I have addressed the concerns you raised previously.

However, I am still a bit worried that my approach might not fully align with your broader roadmap. I would appreciate your feedback on these fixes and the recent alignment.

​@badrishc, if you have some time, could you share some insights on what we might have missed? As a company, we are eager to learn from your experience with large-scale services.

[yuseok-kim-edushare]

#1714 teaches me, about how to use async more correctly,
I will reflect that PR's enhancement into my PR in soon

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yuseok-kim-edushare]

umm I add a option that control shudtown wait times, So I need to fix option related test codes;;