When a Restate node shuts down, the partition processor will fail all pending rpc requests (including ingestion requests from an ingress). If the request is not idempotent, it cannot be retried by the ingress and it results in a 5xx error code. If ClusterFeature::ControlledIdempotentSharding is enabled, then all non-idempotent invocations get an idempotency key assigned which mitigates the problem as long as the ingress is running on a separate node.
If the ingress is also shut down, then it's the responsibility of the caller to retry the call/connect to another node to attach to the invocation. If users use the Restate clients with idempotency keys and retries enabled, then this should work within the bounds set by the retry policy (by default 6 retries with a maximum delay of 2s).
When doing rolling upgrades or shutting down Restate nodes, we ideally don't cause errors for our external clients. Since connection losses can always happen these clients need to be able to retry/re-attach in some cases.
Part of this issue is also that retries are currently costly as they entail Bifrost writes. For invocation attachments that are bound to a specific connection (ephemeral) and thereby specific to a Restate peer, it should be possible to store the information purely in memory.
One idea could be to communicate back the chosen invocation id once the invocations gets applied by the partition processor. If the attachment calls don't cause any Bifrost writes, then retrying by attaching to the received invocation id, could be cheap. Our internal clients and sdk clients could follow the strategy to attach once a connection error is returned if they have received the invocation id. That way we would only be susceptible to 5xxs during the time until we receive the invocation id.
When a Restate node shuts down, the partition processor will fail all pending rpc requests (including ingestion requests from an ingress). If the request is not idempotent, it cannot be retried by the ingress and it results in a 5xx error code. If
ClusterFeature::ControlledIdempotentShardingis enabled, then all non-idempotent invocations get an idempotency key assigned which mitigates the problem as long as the ingress is running on a separate node.If the ingress is also shut down, then it's the responsibility of the caller to retry the call/connect to another node to attach to the invocation. If users use the Restate clients with idempotency keys and retries enabled, then this should work within the bounds set by the retry policy (by default 6 retries with a maximum delay of 2s).
When doing rolling upgrades or shutting down Restate nodes, we ideally don't cause errors for our external clients. Since connection losses can always happen these clients need to be able to retry/re-attach in some cases.
Part of this issue is also that retries are currently costly as they entail Bifrost writes. For invocation attachments that are bound to a specific connection (ephemeral) and thereby specific to a Restate peer, it should be possible to store the information purely in memory.
One idea could be to communicate back the chosen invocation id once the invocations gets applied by the partition processor. If the attachment calls don't cause any Bifrost writes, then retrying by attaching to the received invocation id, could be cheap. Our internal clients and sdk clients could follow the strategy to attach once a connection error is returned if they have received the invocation id. That way we would only be susceptible to 5xxs during the time until we receive the invocation id.