Correction (updated after further investigation). This issue originally attributed a specific production freeze to the missing deadlines described below. That attribution was wrong: the incident was traced to a separate defect in Steampipe itself — a single unparseable or duplicate connection block fails the whole config load, and the connection watcher returns before calling RefreshConnections, so connection syncing stops silently until restart (turbot/steampipe#5020, fix in turbot/steampipe#5021). I have rewritten this issue to describe only what is directly verifiable in the code. Apologies for the noise.
Describe the bug
The unary administrative RPCs issued by the plugin client in grpc/shared/grpc.go — GetSchema, SetConnectionConfig, SetAllConnectionConfigs, UpdateConnectionConfigs, GetSupportedOperations, SetCacheOptions, SetConnectionCacheOptions, SetRateLimiters, GetRateLimiters — are issued on the long-lived plugin context (c.ctx), which is only cancelled when the plugin process ends. None of them carries a deadline, and no timeout is applied at any layer above them.
The consequence is that if a plugin process stops answering one of these calls — without exiting, so the context stays live — the caller blocks indefinitely. There is no error to surface, no retry, and nothing in the logs.
Why this is worth bounding
The blast radius in Steampipe is larger than a single stuck call, because RefreshConnections (pkg/connection/refresh_connections.go) is single-flight with exactly one queued slot: an execute lock plus a queue lock, where any caller that cannot take the queue lock returns immediately and silently. An admin RPC that never returns inside that path would hold the execute lock indefinitely, fill the queued slot, and turn every subsequent refresh trigger into a no-op — so connection config changes would stop being applied for as long as the plugin stayed unresponsive, with no visible error.
This is reachable whenever a plugin can become unresponsive while its process stays alive, which is exactly the case an unbounded RPC cannot defend against.
Expected behaviour
An administrative RPC to an unresponsive plugin should fail with a bounded, visible error that the caller can surface and retry on the next trigger, rather than blocking indefinitely.
Proposed fix
Bound the administrative unary RPCs with a generous default deadline — well above the 240s default plugin start timeout, so legitimately slow operations such as SetAllConnectionConfigs across hundreds of connections are unaffected — and make it overridable via an environment variable. The streaming RPCs (Execute, EstablishMessageStream) are long-lived by design and should stay unbounded.
PR: #957
Version
SDK v5.14.0 (observed in Steampipe v2.4.4)
Describe the bug
The unary administrative RPCs issued by the plugin client in
grpc/shared/grpc.go—GetSchema,SetConnectionConfig,SetAllConnectionConfigs,UpdateConnectionConfigs,GetSupportedOperations,SetCacheOptions,SetConnectionCacheOptions,SetRateLimiters,GetRateLimiters— are issued on the long-lived plugin context (c.ctx), which is only cancelled when the plugin process ends. None of them carries a deadline, and no timeout is applied at any layer above them.The consequence is that if a plugin process stops answering one of these calls — without exiting, so the context stays live — the caller blocks indefinitely. There is no error to surface, no retry, and nothing in the logs.
Why this is worth bounding
The blast radius in Steampipe is larger than a single stuck call, because
RefreshConnections(pkg/connection/refresh_connections.go) is single-flight with exactly one queued slot: an execute lock plus a queue lock, where any caller that cannot take the queue lock returns immediately and silently. An admin RPC that never returns inside that path would hold the execute lock indefinitely, fill the queued slot, and turn every subsequent refresh trigger into a no-op — so connection config changes would stop being applied for as long as the plugin stayed unresponsive, with no visible error.This is reachable whenever a plugin can become unresponsive while its process stays alive, which is exactly the case an unbounded RPC cannot defend against.
Expected behaviour
An administrative RPC to an unresponsive plugin should fail with a bounded, visible error that the caller can surface and retry on the next trigger, rather than blocking indefinitely.
Proposed fix
Bound the administrative unary RPCs with a generous default deadline — well above the 240s default plugin start timeout, so legitimately slow operations such as
SetAllConnectionConfigsacross hundreds of connections are unaffected — and make it overridable via an environment variable. The streaming RPCs (Execute,EstablishMessageStream) are long-lived by design and should stay unbounded.PR: #957
Version
SDK v5.14.0 (observed in Steampipe v2.4.4)