ci: make crates.io publish idempotent and retry index lag (#735)

* ci: make crates.io publish idempotent and retry index lag

The publish job ran six `cargo publish` commands unguarded under `set -e`.
Two failure modes broke it:

  1. Partial release: if a later crate fails, the already-uploaded crates
     can't be re-published — a re-dispatch aborts immediately on the first
     `cargo publish` ("crate version already uploaded").
  2. Index lag: a dependent crate (e.g. icx -> ic-utils) can fail to
     resolve a just-uploaded dependency until the crates.io index catches
     up, which can take a minute or two.

Both hit the 0.48.0 release: ic-transport-types, ic-agent, ic-identity-hsm,
and ic-utils published, then `icx` failed to resolve `ic-utils = ^0.48.0`
because the index hadn't surfaced it yet.

Wrap each publish in a helper that treats "already uploaded/exists" as
success (idempotent re-runs) and retries other failures up to 6x with a
30s backoff (rides out index lag). Re-dispatching now finishes icx and
icx-cert without re-publishing the four crates already live.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: skip final-attempt sleep in publish retry loop

Don't sleep 30s after the last attempt fails — it only delayed the job's
failure. Gate the backoff on there being another attempt to come, and
include attempt count in the messages.

Addresses Copilot review feedback on #735.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: lwshang

.github/workflows/publish.yml

@@ -11,11 +11,38 @@ jobs:
       - uses: rust-lang/crates-io-auth-action@b7e9a28eded4986ec6b1fa40eeee8f8f165559ec # v1
         id: auth
       - run: |
-          cargo publish -p ic-transport-types
-          cargo publish -p ic-agent
-          cargo publish -p ic-identity-hsm
-          cargo publish -p ic-utils
-          cargo publish -p icx
-          cargo publish -p icx-cert
+          set -euo pipefail
+
+          # Publish one crate, tolerating the two things that make a naive
+          # `cargo publish` loop brittle:
+          #   1. Re-running after a partial release: a crate already uploaded at
+          #      this version is a success, not a failure (lets a re-dispatch
+          #      finish the crates that didn't make it the first time).
+          #   2. crates.io index lag: a dependent crate can fail to resolve a
+          #      just-uploaded dependency for up to a minute or two. Retry.
+          publish() {
+            local crate="$1" attempt max=6
+            for attempt in $(seq 1 "$max"); do
+              if cargo publish -p "$crate" 2>&1 | tee /tmp/publish.log; then
+                echo "== $crate published"
+                return 0
+              fi
+              if grep -qiE "already (uploaded|exists)" /tmp/publish.log; then
+                echo "== $crate already published at this version; skipping"
+                return 0
+              fi
+              if [ "$attempt" -lt "$max" ]; then
+                echo "== $crate publish attempt $attempt/$max failed (likely dependency index lag); retrying in 30s"
+                sleep 30
+              fi
+            done
+            echo "== ERROR: $crate failed to publish after $max attempts" >&2
+            return 1
+          }
+
+          # Order matters: dependencies before dependents.
+          for crate in ic-transport-types ic-agent ic-identity-hsm ic-utils icx icx-cert; do
+            publish "$crate"
+          done
         env:
           CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}