Hold per-DB checkpoint locks until all general-BGSAVE per-DB checkpoints complete

[badrishc]

Summary

Fixes a residual race in #1767 that caused MultiDatabaseTests.MultiDatabaseSaveInProgressTest to flake in CI Release builds (e.g. https://github.com/microsoft/garnet/actions/runs/25757540604/job/75650328662).

Root cause

#1767 made the general BGSAVE synchronously pause all per-DB checkpoint locks (TryPauseCheckpoints(id)) before returning Background saving started, so a subsequent BGSAVE <dbId> would observe the in-progress checkpoint and fail. But RunPausedCheckpointAsync released each per-DB lock in finally as soon as that single DB's checkpoint completed, not when the entire general save finished.

In the failing test:

// Issue general background save
res = db1.Execute("BGSAVE");
ClassicAssert.AreEqual("Background saving started", res.ToString());

// Issue background save to DB 0 while general save is in progress - illegal
Assert.Throws<RedisServerException>(() => db1.Execute("BGSAVE", "0"),
    Encoding.ASCII.GetString(CmdStrings.RESP_ERR_CHECKPOINT_ALREADY_IN_PROGRESS));

If DB 0's checkpoint completed before BGSAVE 0 arrived over the wire, BGSAVE 0 succeeded and the assertion failed. Locally the test takes 6-7s and the race never loses; in CI Release it ran in 1s and reliably failed.

Fix

In libs/server/Databases/MultiDatabaseManager.cs:

1. RunPausedCheckpointAsync: removed the ResumeCheckpoints(dbId) from its finally block — caller now owns the resume.
2. RunPausedCheckpointsAndReleaseLocksAsync (used by both general and per-DB BGSAVE): resumes all pre-paused DBs in its outer finally, after Task.WhenAll. Pre-fills checkpointTasks[] with Task.CompletedTask and double-awaits in the catch block so a synchronous task-creation throw cannot leave a per-DB checkpoint running while its lock is being resumed. The handedOffCount partial-resume logic is removed — no longer needed since the helper no longer self-resumes.
3. TaskCheckpointBasedOnAofSizeLimitAsync (the only other caller of RunPausedCheckpointAsync, used by AOF-size-driven checkpoints): hoists pausedDbId to outer scope and calls ResumeCheckpoints(pausedDbId) in its outer finally.

Net effect

• General BGSAVE: per-DB locks held until ALL per-DB checkpoints complete, so any per-DB BGSAVE issued mid-flight reliably fails with checkpoint already in progress. ✓
• Per-DB BGSAVE alone (single-DB path with pausedCount=1): unchanged — that single lock is still released exactly when that single checkpoint completes.
• AOF-size-driven checkpoint: unchanged — still releases the per-DB lock when its checkpoint completes (just resumed in caller's finally instead of in the helper).
• Other legal scenarios preserved:
  • per-DB then per-DB on different DB → both succeed
  • per-DB then general → general succeeds (skips already-paused DBs)
  • general then general → second one fails (guarded by multiDbCheckpointingLock)

Verification

• 15/15 runs of MultiDatabaseSaveInProgressTest pass in Release config locally.
• Full MultiDatabaseTests suite (31/31) passes locally.
• Reviewed by GPT-5.5 code-review agent — no findings.

[Copilot]

Pull request overview

This PR fixes a race in multi-database background checkpointing where a general BGSAVE could release an individual DB’s checkpoint lock as soon as that DB finished, allowing a subsequent BGSAVE <dbId> to sometimes succeed mid-flight and flake MultiDatabaseSaveInProgressTest (notably in fast CI Release runs). The change centralizes ownership of per-DB lock resumption so locks remain held until the full general save completes.

Changes:

• Moved per-DB checkpoint lock resumption responsibility out of RunPausedCheckpointAsync and into its callers.
• Updated the general/per-DB BGSAVE helper to resume all pre-paused DB checkpoint locks only after Task.WhenAll completes.
• Adjusted the AOF-size-driven checkpoint path to resume the paused DB lock in its outer finally.

[badrishc]

Addressed in 7a46983: reworded the doc comment so the parameter range reads as plain prose instead of mixing [0..N) half-open notation with self-closing <paramref/> XML tags.

[badrishc]

Pushed e2d43a3 to address the residual flake on https://github.com/microsoft/garnet/actions/runs/25773997865.

The previous server-side fix correctly held per-DB checkpoint locks until ALL per-DB checkpoints in a general save completed. But for ~1MB of small in-memory data, the actual checkpoint takes microseconds-to-milliseconds, which is comparable to (and sometimes shorter than) the client→server roundtrip between issuing the general BGSAVE and the follow-up BGSAVE 0 via two sequential SE.Redis Execute calls. So the lock window was correct, just not observable to the test client.

This commit replaces those two sequential calls with a single pipelined LightClient send of BGSAVE\r\nBGSAVE 0\r\n. Both commands arrive at the server in the same network packet, so BGSAVE 0 is processed by the network thread immediately after the general BGSAVE's synchronous setup completes — while DB 0's per-DB checkpoint lock is definitely still held. The assertion is now deterministic regardless of how fast the actual checkpoint runs.

CountResponseType.Bytes is required because LightClient's Tokens mode only counts - as an error marker at position 0; a pipelined response +...\r\n-...\r\n would never satisfy CompletePendingRequests under the default Tokens mode.

15/15 runs pass locally in Release. Full MultiDatabaseTests suite (31/31) still passes.