[bug] S3 checkpoint syncer ignores --checkpointSyncer.endpoint and AWS_ENDPOINT_URL_S3 — breaks DO Spaces, MinIO, R2, B2

[davidtai]

Summary

The validator and relayer always send checkpoint S3 traffic to real AWS, even when configured for an S3-compatible store. Two compounding bugs are in play. First, --checkpointSyncer.endpoint is parsed but never reaches the SDK. Second, the standard AWS escape hatch AWS_ENDPOINT_URL_S3 is also ignored, because the aws-config dependency pin predates the version that added support for it. The result is InvalidAccessKeyId from real AWS S3, and the validator panics on its first report_agent_metadata call.

Reproduction

Run gcr.io/abacus-labs-dev/hyperlane-agent:main with --checkpointSyncer.endpoint=https://nyc3.digitaloceanspaces.com and DO Spaces credentials in AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. Setting AWS_ENDPOINT_URL_S3 to the same URL does not change the behavior.

thread 'main' panicked at agents/validator/src/validator.rs:287:14:
Failed to report agent metadata: service error
Caused by:
   1: Error { code: "InvalidAccessKeyId", aws_request_id: "<AWS-format-id>" }
Location: hyperlane-base/src/types/s3_storage.rs:67:9

The AWS-format aws_request_id confirms the request reached real AWS, not the configured endpoint.

Root cause

Bug 1 — the CLI flag is dropped at the SDK boundary. S3Storage carries only bucket, region, and folder fields, and it builds the SDK config with .region(...) only. .endpoint_url(...) is never called, so the parsed CLI value is discarded.

Bug 2 — the documented env-var fallback can't save you. rust/main/Cargo.toml pins aws-config = "1.1.7", released February 2024. Support for service-specific endpoint env vars like AWS_ENDPOINT_URL_S3 landed in aws-config 1.2 via smithy-rs#3568, two months later. As a result, ConfigLoader::default() in this build silently ignores the variable.

Either fix alone would unblock every S3-compatible store, but currently neither does.

Proposed fix

1. Bump aws-config = "1.1.7" → "1.5" in rust/main/Cargo.toml. This is a one-line change that makes AWS_ENDPOINT_URL_S3 work as documented.
2. Add endpoint: Option<String> to S3Storage and wire --checkpointSyncer.endpoint through to .endpoint_url(...). This is roughly 20 lines and makes the CLI flag work as the docs imply.
3. Optionally add --checkpointSyncer.forcePathStyle for backends that require path-style addressing (Cloudflare R2, MinIO defaults).

Workaround

A re-signing reverse-proxy sidecar inside the validator/relayer pod lets the agent keep talking to an S3-compatible backend without any code change to Hyperlane. The pod's hostAliases route the AWS S3 hostnames the agent uses (both the bare service host and the bucket-prefixed virtual-host form) to 127.0.0.1, where the sidecar listens on :443. The sidecar terminates TLS with a server certificate signed by a locally-generated CA, and an init container writes both the cert and a merged CA bundle into a shared emptyDir so the agent container can mount the bundle over /etc/ssl/certs/ca-certificates.crt via a subPath mount — AWS_CA_BUNDLE is also unsupported in this build, so the system trust store is the only path that works. For each request, the sidecar strips the AWS SigV4 authorization headers, rewrites the host to its DO Spaces equivalent (for example <bucket>.s3.us-east-1.amazonaws.com becomes <bucket>.nyc3.digitaloceanspaces.com), re-signs with the same credentials, and forwards. The re-signing is straightforward with github.com/aws/aws-sdk-go-v2/aws/signer/v4's SignHTTP; the only non-obvious detail is that you have to set the X-Amz-Content-Sha256 header explicitly before signing, otherwise DO Spaces rejects the signature.

The relevant pod-spec wiring looks like this:

spec:
  hostAliases:
    - ip: "127.0.0.1"
      hostnames:
        - "s3.us-east-1.amazonaws.com"
        - "s3.amazonaws.com"
        - "<bucket>.s3.us-east-1.amazonaws.com"
        - "<bucket>.s3.amazonaws.com"
  initContainers:
    - name: gen-cert
      image: <sigproxy-image>
      args: ["--mode=init", "--cert-dir=/shared"]
      volumeMounts:
        - { name: sigproxy-shared, mountPath: /shared }
  containers:
    - name: sigproxy
      image: <sigproxy-image>
      args: ["--mode=serve", "--cert-dir=/shared",
             "--upstream=https://nyc3.digitaloceanspaces.com"]
      env:
        - { name: AWS_ACCESS_KEY_ID,     valueFrom: { secretKeyRef: { name: do-spaces, key: AWS_ACCESS_KEY_ID } } }
        - { name: AWS_SECRET_ACCESS_KEY, valueFrom: { secretKeyRef: { name: do-spaces, key: AWS_SECRET_ACCESS_KEY } } }
      volumeMounts:
        - { name: sigproxy-shared, mountPath: /shared, readOnly: true }
    - name: validator
      image: gcr.io/abacus-labs-dev/hyperlane-agent:main
      # ... usual validator args ...
      volumeMounts:
        - name: sigproxy-shared
          mountPath: /etc/ssl/certs/ca-certificates.crt
          subPath: ca-bundle.crt
          readOnly: true
  volumes:
    - name: sigproxy-shared
      emptyDir: { medium: Memory, sizeLimit: 1Mi }

Environment

• Image: hyperlane-agent:main (commit c558a9f)
• aws-config 1.1.7, aws-sdk-s3 1.65.0, aws-smithy-runtime 1.8.1
• Backend: DigitalOcean Spaces (nyc3)

Happy to send a PR — the dep bump alone is one line. Let me know which scope you'd prefer.

[paulbalaji]

adding a relayer-side angle to this, since the fixes proposed in the issue body only close half of it.

a few code-level points that i think should shape the fix:

1. the announced storage_location carries no endpoint, by design

S3Storage::announcement_location emits exactly s3://<bucket>/<region>/<folder>:

// hyperlane-base/src/types/s3_storage.rs
fn announcement_location(&self) -> String {
    match self.folder.as_deref() {
        None | Some("") => format!("s3://{}/{}", self.bucket, self.region),
        Some(folder_str) => {
            format!("s3://{}/{}/{}", self.bucket, self.region, folder_str)
        }
    }
}

and on the relayer side, CheckpointSyncerConf::from_str for the s3 scheme parses it back into just (bucket, region, folder):

// hyperlane-base/src/settings/checkpoint_syncer.rs
"s3" => {
    let url_components = suffix.split('/').collect::<Vec<&str>>();
    let (bucket, region, folder) = match url_components.len() {
        2 => Ok((url_components[0], url_components[1], None)),
        3.. => Ok((url_components[0], url_components[1], Some(url_components[2..].join("/")))),
        _ => Err(..)
    }?;
    Ok(CheckpointSyncerConf::S3 { bucket: bucket.into(), folder, region: .. })
}

no endpoint field anywhere in the roundtrip. even if the proposed validator-side fixes (bump aws-config, wire --checkpointSyncer.endpoint into S3Storage) land, any stock relayer that parses the announcement will still point reads at real aws s3. for origin chains whose validators announce s3 urls for a non-aws s3-compatible bucket, every stock relayer gets NoSuchBucket from aws and cannot consume those checkpoints at all.

2. the anonymous reader client is cached globally per-region

// hyperlane-base/src/types/s3_storage.rs
static ANONYMOUS_CLIENT_CACHE: OnceLock<DashMap<Region, OnceCell<Client>>> = OnceLock::new();

one cached client per region, shared across every origin chain a relayer serves. extending the relayer to use per-announcement endpoints would require either re-keying the cache by (region, endpoint) or dropping the global cache, so it's not purely additive.

3. multisig threshold doesn't route around it

relayers tolerate unreadable validators up to threshold-1, but that's per-chain. if a whole chain's validator set publishes to the same non-aws backend, all reads fail identically. this is a "pick a reachable backend per origin chain" problem, not a "pick a reachable validator" problem.

design suggestion

the announcement url itself needs to carry backend/endpoint info, otherwise relayers have to know out of band which origin chains use which s3-compatible backend, which doesn't scale as more non-aws validators appear.

the existing CheckpointSyncerConf::Gcs + gs:// scheme is a clean in-repo precedent. it's its own variant, own url prefix, own builder, own auth shape, and the relayer dispatches on the prefix with zero s3 plumbing:

// hyperlane-base/src/settings/checkpoint_syncer.rs
pub enum CheckpointSyncerConf {
    LocalStorage { path: PathBuf },
    S3 { bucket: String, folder: Option<String>, region: Region },
    Gcs {
        bucket: String,
        folder: Option<String>,
        service_account_key: Option<String>,
        user_secrets: Option<String>,
    },
}

two plausible shapes for s3-compatible backends that follow that precedent:

• extend CheckpointSyncerConf::S3 with endpoint: Option<String> and a url form that carries it, e.g. s3+custom://<endpoint>/<bucket>/<region>/<folder>. backwards compatible with every existing s3:// announcement; validators on aws keep emitting the old form.
• add a new variant per backend with its own scheme (spaces://, r2://, b2://, minio://...) that hardcodes or carries the endpoint. mirrors how gs:// is done today and sidesteps url escape edge cases, at the cost of one variant per backend.

either way the announcement becomes the source of truth for the endpoint, and the relayer can pick the right client without operator-side config.

tl;dr: the two bugs in the issue body are real and the fixes are right, but without a third change on the announcement/relayer-read path, the workaround in the issue body (hostAliases + sigproxy) stays load-bearing for anyone consuming these checkpoints, not just for the validator host. happy to take a crack at a PR for either shape above if there's a preferred direction.