[BUG] MySQL backup may start before kbadmin is usable on a secondary

[weicao]

Describe the bug

On a newly created MySQL semisync cluster, KubeBlocks can publish the
Component system-account condition as AllProvisioned and make the Cluster
Ready before kbadmin is usable on the secondary Pod.

The backup policy prefers a secondary Pod and uses kbadmin. A backup started
in this window can fail with MySQL access denied even though the same generated
Secret works on the primary and later works on the secondary.

This is tracked as a known issue. The exact readiness-window duration and the
underlying replication timing are still being measured.

Environment

• Addon: MySQL
• Topology: semisync
• Replicas: 2 (one primary, one secondary)
• MySQL: 8.0.46
• Addon source: cb195154944dc3e0911b0b1eb6639edea033ed1b

Evidence

One unchanged-code reproduction recorded this timeline:

• 21:30:14 UTC: kbadmin first authenticated successfully on the primary.
• 21:30:25 UTC: Cluster Ready/Available and
  SystemAccountProvision=True, reason AllProvisioned.
• 21:30:28 UTC: the backup action selected the secondary Pod.
• 21:30:46 and 21:31:08 UTC: the backup action was denied using kbadmin.
• 21:31:17 UTC: a separate read-only kbadmin probe to the secondary was
  still denied.
• About 12 minutes later, kbadmin and the backup account both authenticated
  successfully on the primary and secondary.

The addon references the intended generated Secret and key names. The
AllProvisioned condition currently means the account-provision lifecycle
action returned successfully; it does not verify login on every database
replica. For kbadmin, the current addon action executes select 1 on the
primary.

Steps to reproduce

1. Create a fresh two-Pod MySQL semisync cluster.
2. Observe kbadmin login on both Pod UIDs and roles from before the generated
   account Secret appears.
3. When the Cluster first becomes Ready, immediately create one backup.
4. Record the selected backup Pod UID and continue read-only kbadmin probes
   on both roles every two seconds.
5. Observe that the backup can select the secondary while kbadmin is not yet
   accepted there.

Expected behavior

A backup started after the Cluster becomes Ready should not fail because its
configured system account is not yet usable on the selected secondary.

Temporary workaround

Wait after initial cluster creation and retry the backup. This is a workaround,
not a fix. Operators should keep retries bounded and surface the final
authentication failure instead of retrying indefinitely.

Scope and open questions

• The issue is a readiness window, not evidence of permanent account damage.
• Changing the backup account to root is not an accepted solution.
• A fresh two-second probe run will measure the first secondary recovery time,
  up to 120 seconds.
• The final fix, if needed, must preserve least privilege and validate the
  original kbadmin backup contract.

[weicao]

Investigation update: the affected addon does not resolve the current mutable syncer:0.7.7 tag. It pins the older digest sha256:2cf5d059a029f06372c615ddf79b2fc7ff1876934811932f049e8e7977d224dd, built from syncer revision fda7324024635be310573deb446b39b0e68a434e.

That revision reports every non-DCS-leader MySQL member as secondary without checking replication progress. The current syncer:0.7.7 digest is sha256:80a9bbfafe420035b9512d8a61277fe5fb465551929663d6014f6d2c270c1626, revision 6f6af72f9a69dc8f4a4bf93524366d2d5964b0be. It includes syncer PR #309 (f3a63e992a2732ac81ad9de78e184da4c15f7f3e), which delays initial secondary role publication until the replica's operation timestamp is within five seconds of the DCS leader.

This is the leading root-cause hypothesis because backup target selection consumes the published secondary role. It is not yet the final conclusion. The planned control is the same two-Pod semisync timing test on the pinned old digest and the current 0.7.7 digest, with replica receive/apply state captured at the readiness edge.

[weicao]

Control update: one fresh run on the pinned old digest (2cf5...224dd) did not reproduce the readiness window. The first Ready/Available edge was followed by one xtrabackup targeting the secondary; it completed in 24 seconds with both action containers exiting 0 and no access-denied error. Because the triggering password error did not occur, the bounded post-error probe gate was not entered.

This observation does not erase the earlier frozen old-digest failure, and it cannot confirm or reject the leading root-cause hypothesis because the suspected behavior is timing-sensitive. The planned current-0.7.7 run remains useful as a current-image observation, but a green result will not be described as a strict single-variable reproduction closure.