SoftReadWriteLock blocks indefinitely on an abandoned cross-host .state marker #725

Description

SoftReadWriteLock can block indefinitely if a process terminates while
holding its internal <lock>.state mutex and the next contender has a
different hostname.

The writer and reader markers use heartbeat-based stale detection, but the
internal state mutex is constructed as:

SoftFileLock(self._paths.state, timeout=-1)

SoftFileLock cannot reclaim a well-formed marker belonging to another
hostname because it cannot verify that host's PID. The .state marker has no
heartbeat or lifetime, so it remains permanently held.

Additionally, a finite timeout passed to SoftReadWriteLock is not respected:
acquisition blocks inside the nested SoftFileLock(timeout=-1) before the
outer acquisition loop can check its deadline.

This is particularly relevant to Kubernetes: replacement pods have different
hostnames even when scheduled on the same Kubernetes node.

Affected versions

• Reproduced with filelock 3.30.2
• Source inspection indicates the same behavior in 3.32.5

Reproduction

import subprocess
import sys
import tempfile
from pathlib import Path

from filelock import SoftReadWriteLock


def acquire(lock_path: str) -> None:
    lock = SoftReadWriteLock(
        lock_path,
        timeout=0.2,
        is_singleton=False,
    )
    lock.acquire_write()


if len(sys.argv) == 3 and sys.argv[1] == "acquire":
    acquire(sys.argv[2])
    raise SystemExit("unexpectedly acquired")

with tempfile.TemporaryDirectory() as directory:
    lock_path = Path(directory) / "cache.lock"

    # Simulate a process from another pod/host dying while holding .state.
    Path(f"{lock_path}.state").write_text("123\nterminated-pod\n")

    try:
        subprocess.run(
            [sys.executable, __file__, "acquire", str(lock_path)],
            check=True,
            timeout=1,
        )
    except subprocess.TimeoutExpired:
        print(
            "Still blocked after 1 second despite "
            "SoftReadWriteLock(timeout=0.2)"
        )
    else:
        raise AssertionError("The abandoned state marker was reclaimed")

Output:

Still blocked after 1 second despite SoftReadWriteLock(timeout=0.2)

Making the marker arbitrarily old does not change the result.

As a control, an old valid <lock>.write marker is reclaimed according to
stale_threshold, confirming that heartbeat recovery applies to the public
writer marker but not to .state.

Expected behavior

At minimum, the public acquisition timeout should apply while waiting for the
internal state mutex.

For cross-host crash recovery, an abandoned .state marker should eventually
be reclaimable under a protocol consistent with the heartbeat/lease semantics
of SoftReadWriteLock.

Actual behavior

The contender waits indefinitely. The .state mtime is not refreshed and its
age does not permit recovery.

Production context

We encountered this twice in two weeks on Kubernetes with an AWS EFS RWX PVC.
The workload uses a CronJob with concurrencyPolicy: Forbid.

The mount should be compatible, the relevant mount options are
nfs4 rw,relatime,vers=4.1,hard,proto=tcp,timeo=600,retrans=2,local_lock=none 0 0

Both hangs occurred immediately after application work completed and before
the code entered a write-locked code path. Removing the lock artifacts
allowed the workload to proceed.

Possible fixes

Two related changes appear necessary:

1. Apply the remaining public acquisition deadline when acquiring .state (this does not solve the main issue, but at least honors the timeout)
2. Give .state a cross-host crash-recovery mechanism, rather than relying
   solely on same-host PID liveness.

The second change needs to account for a process pausing/aborting during a state
transition and later resuming. SoftReadWriteLock already has lease-like
semantics for reader/writer markers, so equivalent token/expiry handling for
the state transition may be appropriate.