Skip to content

Bug: Etcd publisher spams logs and fails to recover after auth token expiry #5707

Description

@voidint

Description:

When etcd authentication is enabled and the publisher runs longer than the
etcd auth token TTL, the following error floods the log at a
rate of roughly once per second and never stops:

{"@timestamp":"2026-07-29T09:37:39.546+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:40.710+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:41.784+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:42.850+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}

After investigating the code in keepAliveAsync, we found two underlying
issues that together cause both the log flood and the inability to recover.

1. Recovery is impossible when the error is Unauthenticated

The cached EtcdClient in connManager holds an expired auth token. Because
doKeepAlive always reuses the same cached client, every subsequent Watch
call immediately fails again with the same error, producing the log flood
described above.

connManager (a ResourceManager) has no way to invalidate a single cached
entry, so the stale client is never replaced and the publisher can never
recover on its own.

2. Missing revoke on Watch error (lease leak)

When the KeepAlive channel closes, the code correctly calls p.revoke(cli)
before re-registering. The Watch error path does not:

// KeepAlive channel — revoke is called
case _, ok := <-ch:
    if !ok {
        p.revoke(cli)
        p.doKeepAlive()
        return
    }

// Watch channel — revoke is missing
case c := <-wch:
    if c.Err() != nil {
        p.doKeepAlive()  // Grant creates a new lease; old lease is never revoked
        return
    }

Environments:

  • OS: Linux
  • go-zero version: v1.10.0
  • goctl version: v1.9.2

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions