[BUG] # Server can boot with a permanently frozen clock on hosts with unsynchronised TSC — keys then never expire

[szvyagin-gj]

Summary

On x86-64 Linux hosts whose TSC is not synchronised across cores, monotonicInit_x86linux()
can calibrate to a multiplier of zero. getMonotonicUs() then returns 0 for the lifetime
of the process. The event loop's timers are driven by that clock, so serverCron never runs:
the cached wall clock never advances, TIME returns the server's birth instant forever, and
active key expiry stops permanently. File events keep working, so the server continues to
accept connections and answer commands normally — it just never expires anything again.

Version: 9.1.0 (also present in 9.1.0-rc1, which the line numbers below refer to).
Platform: __x86_64__ && __linux__ && __SIZEOF_INT128__, with the processor clock enabled,
which is the default (monotonic.c:23-25).

This is a silent data-retention hazard, not only a timing nuisance: a server in this state
will hold keys with TTLs indefinitely, and eviction/expiry-dependent behaviour never fires.

How to tell if a running server is affected

$ valkey-cli info server | grep -E 'monotonic_clock|uptime_in_seconds'
monotonic_clock:X86 TSC @ inf ticks/us
uptime_in_seconds:0

The inf is the giveaway — monotonic.c:111 computes ticks_per_us by dividing by the
multiplier, so a zero multiplier renders as inf. A wedged server also returns a
microsecond-identical TIME indefinitely, including while idle.

A healthy server on the same machine reports e.g. monotonic_clock:X86 TSC @ 3193.84 ticks/us
and an advancing uptime.

Root cause

src/monotonic.c, monotonicInit_x86linux():

73:    uint64_t tsc_elapsed = tsc_end - tsc_start;
74:    double sample_ticks_per_us = (double)tsc_elapsed / (double)elapsed_us;
75:    uint64_t sample_mult = (uint64_t)((double)(1ULL << MONO_FPMULT_SHIFT) / sample_ticks_per_us);
...
79:    if (sample_mult < mono_ticks_speed) {
80:        mono_ticks_speed = sample_mult;
81:    }

1. Line 73 subtracts unsigned. If the calibrating thread migrates mid-sample to a core
   whose TSC is behind, tsc_end < tsc_start and tsc_elapsed wraps to ~2^64.
2. Lines 74-75 then truncate to zero. On the machine this was found on the kernel measured
   a 6602625888-cycle warp between cores, so the wrapped value is
   2^64 - 6.6e9 = 1.8446744e19. Over a 10 ms sample that is 1.8e15 ticks/us, and
   (uint64_t)(2^24 / 1.8e15) is 0.
3. Lines 79-81 keep it. Zero is the minimum of anything.
4. Line 101 does not catch it. The only guard is mono_ticks_speed == UINT64_MAX
   ("unable to determine clock rate"); zero passes straight through and the TSC clock is
   installed at line 113.
5. Line 46 is then identically zero:
   ((__uint128_t)__rdtsc() * 0) >> MONO_FPMULT_SHIFT == 0.

The consequence, in src/ae.c:

272:    te->when = getMonotonicUs() + milliseconds * 1000;   /* 0 + 1000 for serverCron */
315:    monotime now = getMonotonicUs();                     /* 0, always */

now never reaches when, so no time event ever fires. serverCron — and with it the cached
clock, active expiry cycle, and everything else on the cron — is dead for the process lifetime.

The minimum-of-N selection makes this worse, not better

It is tempting to read this as a missing zero check, but the selection strategy actively
prefers the corrupted sample. Taking the minimum is intended to pick the most accurate
calibration (smallest multiplier = highest measured ticks/us), and a wrapped tsc_elapsed
always produces the smallest representable value. So one bad sample out of three always
wins, regardless of how good the other two were. Increasing
TSC_CALIBRATION_ITERATIONS cannot help; it only adds chances to draw the poisoned sample.

The CPU check does not cover this

Lines 86-98 verify only that /proc/cpuinfo advertises constant_tsc. That flag is per-core
and says nothing about cross-core synchronisation, which is the property that is actually
broken here. On the affected machine the kernel had already rejected the TSC outright:

kernel: Measured 6602625888 cycles TSC warp between CPUs, turning off TSC clock.
kernel: tsc: Marking TSC unstable due to check_tsc_sync_source failed
kernel: clocksource: Switched to clocksource hpet

/sys/devices/system/clocksource/clocksource0/current_clocksource reads hpet, while all 16
cores still advertise constant_tsc — so valkey uses the TSC on a host whose kernel has
explicitly stopped trusting it.

Reproduction

This needs a host with an unsynchronised TSC; it cannot be triggered on demand on healthy
hardware, since it depends on the calibrating thread being migrated across the warp inside a
10 ms window.

On the affected machine it occurred 3 times in ~40 minutes under repeated server spawns
with heavy concurrent fork/exec churn (a parallel test suite spawning a private server per
run). Under CPU load alone, with no fork churn, it did not occur in 60 spawns — process churn
during startup appears to be what triggers the migration.

The failure mode can be confirmed without waiting for the race by forcing the multiplier to
zero (e.g. mono_ticks_speed = 0 under a debugger after init): the server continues serving
commands, TIME freezes, and uptime_in_seconds stays at 0.

Suggested fix

Two lines address the mechanism directly:

@@ monotonicInit_x86linux
-        uint64_t tsc_elapsed = tsc_end - tsc_start;
+        /* A thread migrated across cores whose TSCs disagree can read a lower
+         * end value; the unsigned difference would wrap. Discard the sample. */
+        if (tsc_end <= tsc_start) continue;
+        uint64_t tsc_elapsed = tsc_end - tsc_start;
@@
-    if (mono_ticks_speed == UINT64_MAX) {
+    if (mono_ticks_speed == UINT64_MAX || mono_ticks_speed == 0) {
         fprintf(stderr, "monotonic: x86 linux, unable to determine clock rate");
         return;
     }

Both failure paths already fall through to monotonicInit_posix(), so a rejected calibration
degrades to clock_gettime rather than to a broken clock.

Worth considering as a stronger guard: refuse the TSC path when
/sys/devices/system/clocksource/clocksource0/current_clocksource is not tsc. The kernel has
already done the cross-core validation that the constant_tsc flag does not cover, and
consulting its conclusion is cheaper and more reliable than re-deriving it.

Workaround

Build with CFLAGS="-DNO_PROCESSOR_CLOCK" (documented at monotonic.c:20-21), which selects
the POSIX clock unconditionally.

Note

The redis-server 8.8.0 binary installed on the same machine reports
monotonic_clock:POSIX clock_gettime and does not exhibit this. I have not checked whether
that reflects an upstream difference or how that particular package was built, so it is offered
only as a data point.

[dvkashapov]

Overall great catch, will post some notes on the PR with some thoughts.