fix(mariadb): linearize primary write acceptance with HA authority

[weicao]

Problem and runtime evidence

Fresh focused r9 on addon source 378d7934e8d2f0b167035275b5c5ca7d77a9627b and syncer 10ee54d8ec86f5d04db0fc6f7628fcde46b2432c hit a first non-environment red during planned switchover.

The addon secondary reconcile fenced the candidate with global read_only=ON while syncer promoted that same Pod. Full addon primary acceptance took about five seconds; syncer then treated the fenced leader as unhealthy and released the new lease.

Frozen focused boundary (N=1, full suite not launched): Promote 1 -> 1, marker present 25/25, old primary unavailable 25/25, candidate writable 23/25, lease refresh 1 -> 1, expiry 0 -> 14. Evidence outer SHA256 is d6281be829a6f749e27ed316cf80d41b8a3fba277f07ddc17d4796588e94c581; root manifest SHA256 is 58d7a21ab4b32c70c70fe7bb6015a9c810106e2faa9cd8944682e46669a873b4 (65/65). Natural convergence after the failure window does not turn r9 into a PASS.

Rejected designs

The following addon exacts and the previous cross-layer pair are withdrawn and must not be packaged or launched:

• 7e23d315...: opened global read-only early and incorrectly described it as an internal-only write plane.
• fde4576b...: left the production order opening user writers before required gates.
• 137f9c47...: checked preStop only before the internal gate and failed to restore the strongest fence on post-gate DCS drift.
• efd452617...: added more fresh reads, but an independent production-function interleaving proved that completed preStop fencing and DCS drift could still be overwritten at the first open. Audit SHA256 eb65c682b9edd20db961aa6fb900a77bad7dd350c7e12fc3f1151ca334ff5fc2; repro SHA256 e5be68045e687d98a369d262b7489d0b2886ff1073eae8d4340ad2a100624c78.
• addon 786f9188... + syncer 9742c492...: serialized the first open but still left root unlock and readiness publication after the authoritative HTTP commit. It also let recurring/repair callers use a stale getrole==primary observation as direct-open authority. The old preStop path additionally continued marker/SQL mutation after its five-second lock budget expired.

More snapshots or a shorter post-response tail cannot make a read-followed-by-write sequence atomic. This revision removes those windows.

Corrected linearizable contract

Addon exact aa6ed6508c17febc312e4c2fef704433abc82ab5 works with syncer PR #304 exact 8cfcc248fccf9b0d7011ce78db8831342a0714b8:

1. set_primary_read_write owns an atomic local commit lock through its complete acceptance attempt.
2. user-root bypass checks run first; global read-only is narrowed to ON, which still fences ordinary user writers.
3. local and remote root accounts are unlocked while that global user fence remains active. The remote-root fence marker is removed under the same lock.
4. fenced promotion calls syncerctl primary-write-commit as the last visible commit operation. After HTTP success the addon only releases its local lock and emits a neutral acknowledgement log; it does not open writes, unlock accounts, or publish readiness.
5. syncer serializes the authoritative operation with HA cycles. Under one mutex it refreshes DCS, requires the local lease, performs Kubernetes resourceVersion CAS, opens and reads back MariaDB global writes, clears pending/divergence, and publishes .primary-read-write-ready plus .replication-ready (the latter last) before returning success.
6. any syncer filesystem-publication failure removes both ready markers, restores pending, and returns an error so addon rollback runs.
7. Demote publishes pending and removes both ready markers before it tightens MariaDB read-only. If that transition fails, demotion fails and the switchover handler does not release the lease.
8. a still-running leader that exceeds the health-failure threshold must also complete Demote before lease release and recovery. Demote failure must renew DCS authority; if renewal fails, it fail-stops and positively verifies the local DB is no longer running. These branches never release or recover a writable/ready old primary, and incomplete fail-stop is reported fail_closed=false.
9. every recurring, repair, service-route, stale-data, and syncer-promoted acceptance call uses fenced promotion. Only a true first default-pod0-primary bootstrap with no DCS lease may use the addon-owned standard path.
10. recurring syncer Promote prepares replication only. It never opens writes or publishes readiness.
11. preStop shares the local commit lock. Within the container's 120-second termination budget it waits at most 40 seconds; if the lock remains unavailable, it exits 1 before any marker or SQL mutation. Kubelet process termination is the truthful exceptional fail-close boundary, so this branch does not claim durable strongest-fence or root-lock state.

Rollback still distinguishes a complete strongest fence plus both root locks (rc=2) from incomplete rollback (rc=3, fail_closed=false). MariaDB NO_LOCK_NO_ADMIN remains the required complete rollback state because ON permits the addon-owned READ_ONLY ADMIN account to write.

Predecessor TDD and source verification

The tests extract and execute production functions rather than wrapper-only substitutes.

• recurring-open structural RED: the previous head had non-bootstrap callers that could reach standard direct open; the new head permits it only for true first pod-0 bootstrap.
• post-response linearization RED: the previous head committed through syncer and then changed root/readiness state in addon; the new production harness injects an immediate next demotion and proves that commit publishes complete readiness, the next HA cycle retracts it, and addon performs no later visible transition.
• preStop RED: the previous five-second lock timeout continued marker/fence mutation through || true; the new production harness holds acceptance beyond the old budget and proves timeout means zero marker/SQL mutation before process termination.
• focused acceptance/lock/static set: 96 examples, 0 failures.
• related acceptance + lock + r9 race + fence/startup set: 101 examples, 0 failures.
• all MariaDB ShellSpec files: 766 examples = 757 pass / 0 fail / 9 historical pending.
• switchover suite: 201 pass / 0 fail / 6 historical pending.
• addon bash -n, sh -n, ShellCheck severity=error, helm template, and git diff --check: PASS.
• syncer go test ./... -count=1, focused go test -race, and go vet ./...: PASS on the paired exact.

CI, review, and runtime boundary

Current addon head aa6ed6508c17febc312e4c2fef704433abc82ab5 is pushed, OPEN, non-draft, mergeable, and its official CI is terminal green. The cross-repository syncer references above document the predecessor integration boundary; this metadata refresh does not reassert their present CI state. Earlier review/checker results remain scoped to the exact paths and heads they covered.

Runtime for the corrected pair is N=0. This PR update is not a Test GO. After both official CIs, Magnus's new-head addon review, and Jack's independent production checker close, Dev must freshly enumerate every current weicao open PR touching MariaDB, build a new combined source, and publish a new exact manifest. Test independently owns package, render, runner, live identity, focused runtime evidence, cleanup, and any later full-suite decision.

Current-head successor correction

Current addon exact: aa6ed6508c17febc312e4c2fef704433abc82ab5 (tree a177476dc28e3283c5308599942f5dd5df531d76). This successor keeps the authority-linearization contract above and fixes one additional preStop SQL-literal contract gap: values now escape backslashes before doubling single quotes, matching the runtime SQL-quote contract for socket and TCP paths.

• old-source adversarial embedded/trailing backslash + quote cases: 3 failures
• current successor focused spec: 3 examples, 0 failures
• related runtime-race + semisync templates: 78 examples, 0 failures
• current exact GitHub Actions: terminal green
• runtime on this current exact remains N=0; these source tests do not replace Jack-owned product validation

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 444 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (378d793) to head (aa6ed65).

Files with missing lines	Patch %	Lines
...ut-spec/replication_full_accept_user_fence_spec.sh	0.00%	172 Missing ⚠️
...ec/replication_runtime_role_promotion_race_spec.sh	0.00%	90 Missing ⚠️
...ication_primary_write_commit_linearization_spec.sh	0.00%	85 Missing ⚠️
...ipts-ut-spec/replication_prestop_sql_quote_spec.sh	0.00%	56 Missing ⚠️
...pts-ut-spec/semisync_rejoin_fence_template_spec.sh	0.00%	39 Missing ⚠️
...adb/scripts-ut-spec/replication_switchover_spec.sh	0.00%	2 Missing ⚠️

Additional details and impacted files

@@                                 Coverage Diff                                  @@
##           helen/mariadb-open-prs-combined-20260714t164119z-v3   #3208    +/-   ##
====================================================================================
  Coverage                                                 0.00%   0.00%            
====================================================================================
  Files                                                      150     154     +4     
  Lines                                                    23752   24180   +428     
====================================================================================
- Misses                                                   23752   24180   +428     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[weicao]

Superseded by #3276. This PR's current head 8aca6a2 is included in the consolidated candidate 7978812; further review and testing should use #3276.

[weicao]

Reopened per owner direction after rolling back the consolidated PR #3276. This PR is again an independent review and validation unit.