Description:
When etcd authentication is enabled and the publisher runs longer than the
etcd auth token TTL, the following error floods the log at a
rate of roughly once per second and never stops:
{"@timestamp":"2026-07-29T09:37:39.546+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:40.710+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:41.784+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
{"@timestamp":"2026-07-29T09:37:42.850+08:00","caller":"discov/publisher.go:143","content":"etcd publisher watch: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token","level":"error"}
After investigating the code in keepAliveAsync, we found two underlying
issues that together cause both the log flood and the inability to recover.
1. Recovery is impossible when the error is Unauthenticated
The cached EtcdClient in connManager holds an expired auth token. Because
doKeepAlive always reuses the same cached client, every subsequent Watch
call immediately fails again with the same error, producing the log flood
described above.
connManager (a ResourceManager) has no way to invalidate a single cached
entry, so the stale client is never replaced and the publisher can never
recover on its own.
2. Missing revoke on Watch error (lease leak)
When the KeepAlive channel closes, the code correctly calls p.revoke(cli)
before re-registering. The Watch error path does not:
// KeepAlive channel — revoke is called
case _, ok := <-ch:
if !ok {
p.revoke(cli)
p.doKeepAlive()
return
}
// Watch channel — revoke is missing
case c := <-wch:
if c.Err() != nil {
p.doKeepAlive() // Grant creates a new lease; old lease is never revoked
return
}
Environments:
- OS: Linux
- go-zero version: v1.10.0
- goctl version: v1.9.2
Description:
When etcd authentication is enabled and the publisher runs longer than the
etcd auth token TTL, the following error floods the log at a
rate of roughly once per second and never stops:
After investigating the code in
keepAliveAsync, we found two underlyingissues that together cause both the log flood and the inability to recover.
1. Recovery is impossible when the error is
UnauthenticatedThe cached
EtcdClientinconnManagerholds an expired auth token. BecausedoKeepAlivealways reuses the same cached client, every subsequent Watchcall immediately fails again with the same error, producing the log flood
described above.
connManager(aResourceManager) has no way to invalidate a single cachedentry, so the stale client is never replaced and the publisher can never
recover on its own.
2. Missing
revokeon Watch error (lease leak)When the KeepAlive channel closes, the code correctly calls
p.revoke(cli)before re-registering. The Watch error path does not:
Environments: