fix: add bounded jittered backoff to StartableToolSet retry path

Fixes issue #4060: RAG semantic-embeddings indexing triggered a
rate-limit retry storm — repeated toolset-start attempts had no pacing
after a 429 from the embedding provider.

Implementation:
- modelerrors.RetryableHTTPStatus(err): HTTP-status classifier that
  recognises 429, 408, and 5xx via *StatusError first, then falls back
  to statusCodeRegex. The toolset gate pre-filters to *StatusError via
  errors.As before calling it, so port numbers and chunk counts in
  plain error strings cannot arm the gate.
- pkg/tools/startable_backoff.go: bounded exponential backoff with
  additive 0-20% jitter (base=15s, cap=5min, delay∈[d,1.2d]).
- Gate in tryStartLocked (TryStart/TryStartWithTimeout only): blocking
  Start() bypasses it so mcpcatalog enable and skill startup are
  immediate.
- Gate adopts a live StartReporter after /toolset-restart without
  waiting for the window to expire.
- Wrap embedding errors in WrapHTTPError at openai/client.go and
  dmr/embed.go so a 429 from the embedding provider surfaces as
  *StatusError and correctly arms the gate.
- WithStartRetryJitter / WithStartRetryClock options via variadic
  NewStartable for deterministic test control.
- Stale 'retry on next turn' log messages updated in agent.go/mcp.go.
- Partial-start exemption documented (code-mode composites remain
  unpaced; follow-up at issue #4067).

Tests (same commit, covering the above):
- startable_backoff_test.go: unit tests for the gate (gate fires on
  429/408/5xx StatusError, not on plain text / context errors,
  blocking Start() ungated, concurrency, jitter bounds)
- startable_backoff_regression_test.go: consumer-shaped regression
  suite (RAG/MCP/LSP error shapes, no-goroutine/timer leak, latch)
- rag_backoff_test.go: real-toolset integration test via rag.New +
  fake clock

Docs:
- docs/tools/rag/index.md: 'Indexing failures, retries and backoff'
  section with trigger table, parameters, and troubleshooting.
- docs/tools/mcp/index.md, docs/tools/lsp/index.md: lifecycle notes
  confirming local startup failures fail fast.

Scope: DefaultStartTimeout (30s) unchanged — deferred.

Author: aheritier

docs/community/troubleshooting/index.md

@@ -199,6 +199,8 @@ MCP tools using stdio transport must complete the initialization handshake befor
 
 If a toolset keeps crashing in a tight loop, tune the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block on the toolset (e.g. raise `backoff.initial`, lower `max_restarts`, or switch to the `best-effort` profile) so a flaky dependency does not amplify into a restart storm.
 
+If a **RAG knowledge base** is failing to index because the embedding provider is rate-limiting requests (HTTP 429), Docker Agent automatically backs off and retries — see [Indexing failures, retries and backoff](../../tools/rag/index.md#indexing-failures-retries-and-backoff) for the retry schedule and the `max_indexing_concurrency` / `max_embedding_concurrency` knobs that control how much concurrent load is generated.
+
 ## Configuration Errors
 
 ### YAML syntax issues

docs/tools/lsp/index.md

@@ -197,6 +197,8 @@ Available Capabilities:
 
 LSP toolsets are managed by the same supervisor as MCP toolsets, so a crashed `gopls` (or any other language server) is reconnected automatically with exponential backoff. Use the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block to tune the policy per toolset — for example, mark `gopls` as `strict` if your CI flow requires it to be available, or use `/toolset-restart gopls` from the TUI to force a reconnect when the server gets stuck.
 
+**Startup failure behaviour:** local LSP server startup failures (missing binary, server-unavailable) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to LSP server startup.
+
 ```yaml
 toolsets:
   - type: lsp

docs/tools/mcp/index.md

@@ -258,6 +258,8 @@ toolsets:
 
 See [Toolset Lifecycle](../../configuration/tools/index.md#toolset-lifecycle) for all profiles and tuning knobs, and [`/toolset-restart`](../../features/tui/index.md) to force a reconnect from the TUI.
 
+**Startup failure behaviour:** local MCP startup failures (missing binary, connection refused, authentication error) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to MCP server startup.
+
 ## Combined Example
 
 ```yaml

docs/tools/rag/index.md

@@ -174,6 +174,92 @@ chunking:
 >
 > Currently supports Go (`.go`) files. More languages will be added. Falls back to plain text chunking for unsupported file types.
 
+## Indexing failures, retries and backoff
+
+When a knowledge-base fails to start — because the embedding provider is rate-limiting
+your requests or returning a transient server error — Docker Agent spaces out retry
+attempts with bounded exponential backoff instead of hammering the provider on
+every agent turn.
+
+### What triggers backoff
+
+Backoff applies only to **HTTP 429 rate-limit** responses from the embedding or
+model provider — the one signal that reliably reaches the toolset gate. Other
+errors (5xx, 408) are handled per-file within the indexing run and do not arm
+the gate. These are the current gate triggers:
+
+| Failure kind | Behaviour |
+|---|---|
+| HTTP 429 (rate limit) | Backoff: next attempt delayed |
+| Other failures (5xx, 408, config errors, auth) | Fail fast: retried every turn with no added delay |
+| Context cancellation or agent shutdown | Immediate: no delay |
+
+> [!NOTE]
+> 5xx and 408 errors from the embedding provider are retried per-file and do not
+> propagate to the toolset gate. Only 429 (rate-limit) terminates the indexing run
+> early and surfaces the gate so Docker Agent can pace the next attempt.
+
+### Retry policy and parameters
+
+The backoff is **bounded exponential with additive jitter**:
+
+- **Base delay**: 15 seconds
+- **Maximum delay**: up to ~6 minutes (5-minute cap plus up to 20% additive jitter)
+- **Growth**: doubles after each consecutive retryable failure (15s → 30s → 1m → 2m → 4m → 5m)
+- **Jitter**: each wait is a random value in `[nominal, 1.2×nominal]` (additive 0–20%)
+  so concurrent knowledge-base sources spread their retries and avoid
+  hammering the provider together
+- **Retry-After override**: if the embedding provider responds with a `Retry-After`
+  header, that hint overrides the computed delay (capped at the 5-minute maximum,
+  with the same additive jitter applied to spread concurrent retries)
+
+The gate is a lightweight wall-clock check — it creates no background threads or
+timers. A Stop command or agent shutdown takes effect immediately regardless of
+how much of the backoff window remains.
+
+### Operational impact
+
+**Before**: a rate-limited knowledge base was re-indexed on every agent turn —
+`max_indexing_concurrency × max_embedding_concurrency` concurrent provider calls
+could relaunch within milliseconds, easily tripping rate limits for both the
+knowledge base and the agent's own model calls.
+
+**After**: retries are spaced out and jittered so the provider has room to recover
+before the next attempt. The agent continues working with any other toolsets that
+are not affected.
+
+### What you will see
+
+- Docker Agent logs a single warning when a knowledge base first fails to start.
+  Repeated failures in between are logged at debug level only, so you are not
+  flooded with alerts on every turn. Recovery is intentionally silent — the
+  tool appearing in the agent's tool list is the signal that indexing succeeded.
+- The knowledge-base tool does not appear in the agent's tool list until indexing
+  succeeds. A successful start is silent — the tool is listed and the agent uses it.
+
+### Troubleshooting repeated 429 errors
+
+If you see persistent `429` errors in the logs:
+
+1. **Check provider rate limits.** Your embedding API key may have a low requests-per-minute
+   quota. Upgrading the plan or using a different API key can help.
+2. **Reduce concurrency.** The chunked-embeddings and semantic-embeddings strategies
+   accept `max_indexing_concurrency` (default `3`) and `max_embedding_concurrency`
+   (default `3`) parameters. Lowering these reduces simultaneous requests:
+
+   ```yaml
+   rag:
+     docs:
+       docs: [./knowledge-base]
+       strategies:
+         - type: chunked-embeddings
+           max_indexing_concurrency: 1
+           max_embedding_concurrency: 1
+   ```
+
+3. **Use a model with a higher quota.** Some providers offer higher rate limits on
+   specific embedding model tiers.
+
 ## Debugging RAG
 
 Enable debug logging to see retrieval details:
@@ -218,6 +304,7 @@ Look for log tags: `[RAG Manager]`, `[Chunked-Embeddings Strategy]`, `[BM25 Stra
 | `limit`                     | int    | `5`                 | Max results from this strategy                               |
 | `embedding_batch_size`      | int    | `50`                | Chunks per embedding request                                 |
 | `max_embedding_concurrency` | int    | `3`                 | Max concurrent embedding requests                            |
+| `max_indexing_concurrency`  | int    | `3`                 | Max concurrent file-indexing tasks                           |
 | `chunking.size`             | int    | `1500`              | Chunk size in characters (`4000` when `code_aware` is set)   |
 | `chunking.overlap`          | int    | `75`                | Overlap between chunks in characters                         |
 | `chunking.code_aware`       | bool   | `false`             | AST-based chunking (Go files only)                           |

pkg/agent/agent.go

@@ -645,10 +645,10 @@ func (a *Agent) ensureToolSetsAreStarted(ctx context.Context) {
 			continue
 		}
 		if toolSet.ShouldReportFailure() {
-			slog.WarnContext(ctx, "Toolset start failed; will retry on next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.WarnContext(ctx, "Toolset start failed; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 			a.AddToolWarning(fmt.Sprintf("%s start failed: %v", desc, err))
 		} else {
-			slog.DebugContext(ctx, "Toolset still unavailable; retrying next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.DebugContext(ctx, "Toolset still unavailable; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 		}
 	}
 }

pkg/model/provider/dmr/embed.go

@@ -17,6 +17,7 @@ import (
 	"github.com/openai/openai-go/v3"
 
 	"github.com/docker/docker-agent/pkg/model/provider/base"
+	"github.com/docker/docker-agent/pkg/model/provider/oaistream"
 	"github.com/docker/docker-agent/pkg/rag/types"
 )
 
@@ -50,7 +51,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 		Model: c.ModelConfig.Model,
 	})
 	if err != nil {
-		return nil, fmt.Errorf("failed to create embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/model/provider/dmr/embed_test.go

@@ -0,0 +1,51 @@
+package dmr
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	openaisdk "github.com/openai/openai-go/v3"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/docker/docker-agent/pkg/config/latest"
+	"github.com/docker/docker-agent/pkg/modelerrors"
+)
+
+// TestCreateBatchEmbedding_429SurfacesAsStatusError proves that when the
+// embedding endpoint returns HTTP 429 the error propagates as
+// *modelerrors.StatusError so the StartableToolSet backoff gate can arm.
+func TestCreateBatchEmbedding_429SurfacesAsStatusError(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusTooManyRequests)
+		_, _ = w.Write([]byte(`{"error":{"message":"rate limit exceeded","type":"requests","code":"rate_limit_exceeded"}}`))
+	}))
+	t.Cleanup(srv.Close)
+
+	cfg := &latest.ModelConfig{
+		Provider: "dmr",
+		Model:    "ai/nomic-embed-text",
+		BaseURL:  srv.URL + "/v1",
+	}
+	client, err := NewClient(t.Context(), cfg, nil)
+	require.NoError(t, err)
+
+	_, err = client.CreateBatchEmbedding(t.Context(), []string{"hello"})
+	require.Error(t, err, "expected error from 429 response")
+
+	// The critical assertion: the OpenAI SDK error must be wrapped in
+	// *modelerrors.StatusError by WrapOpenAIError so the backoff gate can arm.
+	var se *modelerrors.StatusError
+	require.ErrorAs(t, err, &se,
+		"error must contain *modelerrors.StatusError so the backoff gate can arm")
+	assert.Equal(t, http.StatusTooManyRequests, se.StatusCode)
+
+	// Cross-check: the underlying SDK error is also reachable.
+	var apiErr *openaisdk.Error
+	assert.ErrorAs(t, err, &apiErr,
+		"underlying *openaisdk.Error must be reachable through the chain")
+}

pkg/model/provider/openai/client.go

@@ -1165,7 +1165,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 	response, err := client.Embeddings.New(ctx, params)
 	if err != nil {
 		slog.ErrorContext(ctx, "OpenAI batch embedding request failed", "error", err)
-		return nil, fmt.Errorf("failed to create batch embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create batch embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/model/provider/openai/embed_test.go

@@ -0,0 +1,90 @@
+package openai
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	openaisdk "github.com/openai/openai-go/v3"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/docker/docker-agent/pkg/config/latest"
+	"github.com/docker/docker-agent/pkg/modelerrors"
+)
+
+// TestCreateBatchEmbedding_429SurfacesAsStatusError mirrors the DMR embed test
+// for the OpenAI provider path (pkg/model/provider/openai/client.go:1168).
+// It proves that a 429 from the embedding endpoint propagates as
+// *modelerrors.StatusError so the StartableToolSet backoff gate can arm.
+func TestCreateBatchEmbedding_429SurfacesAsStatusError(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusTooManyRequests)
+		_, _ = w.Write([]byte(`{"error":{"message":"rate limit exceeded","type":"requests","code":"rate_limit_exceeded"}}`))
+	}))
+	t.Cleanup(srv.Close)
+
+	cfg := &latest.ModelConfig{
+		Provider:     "openai",
+		Model:        "text-embedding-3-small",
+		BaseURL:      srv.URL,
+		ProviderOpts: map[string]any{"api_type": "openai_chatcompletions"}, // custom provider → no auth required
+	}
+	client, err := NewClient(t.Context(), cfg, nil)
+	require.NoError(t, err)
+
+	_, err = client.CreateBatchEmbedding(t.Context(), []string{"hello"})
+	require.Error(t, err, "expected error from 429 response")
+
+	// The critical assertion: the error must contain *modelerrors.StatusError
+	// so the StartableToolSet backoff gate can arm.
+	var se *modelerrors.StatusError
+	require.ErrorAs(t, err, &se,
+		"error must wrap *modelerrors.StatusError so the backoff gate can arm")
+	assert.Equal(t, http.StatusTooManyRequests, se.StatusCode)
+
+	// The underlying SDK error must also be reachable.
+	var apiErr *openaisdk.Error
+	assert.ErrorAs(t, err, &apiErr,
+		"underlying *openaisdk.Error must be reachable through the chain")
+}
+
+// TestCreateBatchEmbedding_5xxDoesNotSurfaceAsStatusError verifies that 5xx
+// errors from the embedding endpoint are handled but do NOT propagate as
+// *modelerrors.StatusError (they are per-file retryable errors swallowed by
+// the indexing strategy, not gate-arming errors).
+func TestCreateBatchEmbedding_5xxSurfacesAsStatusError(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusServiceUnavailable)
+		_, _ = w.Write([]byte(`{"error":{"message":"service unavailable","type":"server_error"}}`))
+	}))
+	t.Cleanup(srv.Close)
+
+	cfg := &latest.ModelConfig{
+		Provider:     "openai",
+		Model:        "text-embedding-3-small",
+		BaseURL:      srv.URL,
+		ProviderOpts: map[string]any{"api_type": "openai_chatcompletions"}, // custom provider → no auth required
+	}
+	client, err := NewClient(t.Context(), cfg, nil)
+	require.NoError(t, err)
+
+	_, err = client.CreateBatchEmbedding(t.Context(), []string{"hello"})
+	require.Error(t, err)
+
+	// 5xx is also wrapped by WrapOpenAIError — it surfaces as *StatusError
+	// but is classified retryable-per-file by classifyModelCallError.
+	var se *modelerrors.StatusError
+	require.ErrorAs(t, err, &se)
+	assert.Equal(t, http.StatusServiceUnavailable, se.StatusCode)
+
+	// Verify the chain is intact.
+	var apiErr *openaisdk.Error
+	assert.ErrorAs(t, err, &apiErr)
+}

pkg/modelerrors/modelerrors.go

@@ -781,3 +781,18 @@ func scalarString(v any) string {
 		return fmt.Sprint(v)
 	}
 }
+
+// RetryableHTTPStatus reports whether err contains an HTTP status that warrants
+// backoff (429 rate-limit, 408 request-timeout, or a 5xx server error). It
+// checks for a *StatusError in the chain first; if none is found it falls back
+// to matching \b([45]\d{2})\b in the error message, which can produce false
+// positives for port numbers or similar numeric patterns. Callers that need
+// strict StatusError-only classification should pre-filter with errors.As.
+// No context-error handling is performed.
+func RetryableHTTPStatus(err error) bool {
+	code := extractHTTPStatusCode(err)
+	if code == 0 {
+		return false
+	}
+	return code == http.StatusTooManyRequests || isRetryableStatusCode(code)
+}