fix: add bounded jittered backoff to StartableToolSet retry path

Fixes issue #4060: RAG semantic-embeddings indexing triggered a
rate-limit retry storm — repeated toolset-start attempts had no pacing
after a 429 from the embedding provider.

Implementation:
- modelerrors.RetryableHTTPStatus(err): HTTP-status classifier that
  recognises 429, 408, and 5xx via *StatusError first, then falls back
  to statusCodeRegex. The toolset gate pre-filters to *StatusError via
  errors.As before calling it, so port numbers and chunk counts in
  plain error strings cannot arm the gate.
- pkg/tools/startable_backoff.go: bounded exponential backoff with
  additive 0-20% jitter (base=15s, cap=5min, delay∈[d,1.2d]).
- Gate in tryStartLocked (TryStart/TryStartWithTimeout only): blocking
  Start() bypasses it so mcpcatalog enable and skill startup are
  immediate.
- Gate adopts a live StartReporter after /toolset-restart without
  waiting for the window to expire.
- Wrap embedding errors in WrapHTTPError at openai/client.go and
  dmr/embed.go so a 429 from the embedding provider surfaces as
  *StatusError and correctly arms the gate.
- WithStartRetryJitter / WithStartRetryClock options via variadic
  NewStartable for deterministic test control.
- Stale 'retry on next turn' log messages updated in agent.go/mcp.go.
- Partial-start exemption documented (code-mode composites remain
  unpaced; follow-up at issue #4067).

Tests (same commit, covering the above):
- startable_backoff_test.go: unit tests for the gate (gate fires on
  429/408/5xx StatusError, not on plain text / context errors,
  blocking Start() ungated, concurrency, jitter bounds)
- startable_backoff_regression_test.go: consumer-shaped regression
  suite (RAG/MCP/LSP error shapes, no-goroutine/timer leak, latch)
- rag_backoff_test.go: real-toolset integration test via rag.New +
  fake clock

Docs:
- docs/tools/rag/index.md: 'Indexing failures, retries and backoff'
  section with trigger table, parameters, and troubleshooting.
- docs/tools/mcp/index.md, docs/tools/lsp/index.md: lifecycle notes
  confirming local startup failures fail fast.

Scope: DefaultStartTimeout (30s) unchanged — deferred.

Author: aheritier

docs/community/troubleshooting/index.md

@@ -199,6 +199,8 @@ MCP tools using stdio transport must complete the initialization handshake befor
 
 If a toolset keeps crashing in a tight loop, tune the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block on the toolset (e.g. raise `backoff.initial`, lower `max_restarts`, or switch to the `best-effort` profile) so a flaky dependency does not amplify into a restart storm.
 
+If a **RAG knowledge base** is failing to index because the embedding provider is rate-limiting requests (HTTP 429), Docker Agent automatically backs off and retries — see [Indexing failures, retries and backoff](../../tools/rag/index.md#indexing-failures-retries-and-backoff) for the retry schedule and the `max_indexing_concurrency` / `max_embedding_concurrency` knobs that control how much concurrent load is generated.
+
 ## Configuration Errors
 
 ### YAML syntax issues

docs/tools/lsp/index.md

@@ -197,6 +197,8 @@ Available Capabilities:
 
 LSP toolsets are managed by the same supervisor as MCP toolsets, so a crashed `gopls` (or any other language server) is reconnected automatically with exponential backoff. Use the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block to tune the policy per toolset — for example, mark `gopls` as `strict` if your CI flow requires it to be available, or use `/toolset-restart gopls` from the TUI to force a reconnect when the server gets stuck.
 
+**Startup failure behaviour:** local LSP server startup failures (missing binary, server-unavailable) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to LSP server startup.
+
 ```yaml
 toolsets:
   - type: lsp

docs/tools/mcp/index.md

@@ -258,6 +258,8 @@ toolsets:
 
 See [Toolset Lifecycle](../../configuration/tools/index.md#toolset-lifecycle) for all profiles and tuning knobs, and [`/toolset-restart`](../../features/tui/index.md) to force a reconnect from the TUI.
 
+**Startup failure behaviour:** local MCP startup failures (missing binary, connection refused, authentication error) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to MCP server startup.
+
 ## Combined Example
 
 ```yaml

docs/tools/rag/index.md

@@ -174,6 +174,89 @@ chunking:
 >
 > Currently supports Go (`.go`) files. More languages will be added. Falls back to plain text chunking for unsupported file types.
 
+## Indexing failures, retries and backoff
+
+When a knowledge-base fails to start — because the embedding provider is rate-limiting
+your requests or returning a transient server error — Docker Agent spaces out retry
+attempts with bounded exponential backoff instead of hammering the provider on
+every agent turn.
+
+### What triggers backoff
+
+Backoff applies only to **HTTP 429 rate-limit** responses from the embedding or
+model provider — the one signal that reliably reaches the toolset gate. Other
+errors (5xx, 408) are handled per-file within the indexing run and do not arm
+the gate. These are the current gate triggers:
+
+| Failure kind | Behaviour |
+|---|---|
+| HTTP 429 (rate limit) | Backoff: next attempt delayed |
+| Other failures (5xx, 408, config errors, auth) | Fail fast: retried every turn with no added delay |
+| Context cancellation or agent shutdown | Immediate: no delay |
+
+> [!NOTE]
+> 5xx and 408 errors from the embedding provider are retried per-file and do not
+> propagate to the toolset gate. Only 429 (rate-limit) terminates the indexing run
+> early and surfaces the gate so Docker Agent can pace the next attempt.
+
+### Retry policy and parameters
+
+The backoff is **bounded exponential with additive jitter**:
+
+- **Base delay**: 15 seconds
+- **Maximum delay**: 5 minutes
+- **Growth**: doubles after each consecutive retryable failure (15s → 30s → 1m → 2m → 4m → 5m)
+- **Jitter**: each wait is a random value in `[nominal, 1.2×nominal]` (additive 0–20%)
+  so concurrent knowledge-base sources spread their retries and avoid
+  hammering the provider together
+
+The gate is a lightweight wall-clock check — it creates no background threads or
+timers. A Stop command or agent shutdown takes effect immediately regardless of
+how much of the backoff window remains.
+
+### Operational impact
+
+**Before**: a rate-limited knowledge base was re-indexed on every agent turn —
+`max_indexing_concurrency × max_embedding_concurrency` concurrent provider calls
+could relaunch within milliseconds, easily tripping rate limits for both the
+knowledge base and the agent's own model calls.
+
+**After**: retries are spaced out and jittered so the provider has room to recover
+before the next attempt. The agent continues working with any other toolsets that
+are not affected.
+
+### What you will see
+
+- Docker Agent logs a single warning when a knowledge base first fails to start.
+  Repeated failures in between are logged at debug level only, so you are not
+  flooded with alerts on every turn. Recovery is intentionally silent — the
+  tool appearing in the agent's tool list is the signal that indexing succeeded.
+- The knowledge-base tool does not appear in the agent's tool list until indexing
+  succeeds. A successful start is silent — the tool is listed and the agent uses it.
+
+### Troubleshooting repeated 429 errors
+
+If you see persistent `429` errors in the logs:
+
+1. **Check provider rate limits.** Your embedding API key may have a low requests-per-minute
+   quota. Upgrading the plan or using a different API key can help.
+2. **Reduce concurrency.** The chunked-embeddings and semantic-embeddings strategies
+   accept `max_indexing_concurrency` (default `3`) and `max_embedding_concurrency`
+   (default `3`) parameters. Lowering these reduces simultaneous requests:
+
+   ```yaml
+   rag:
+     docs:
+       docs: [./knowledge-base]
+       strategies:
+         - type: chunked-embeddings
+           max_indexing_concurrency: 1
+           max_embedding_concurrency: 1
+   ```
+
+3. **Use a model with a higher quota.** Some providers offer higher rate limits on
+   specific embedding model tiers.
+
 ## Debugging RAG
 
 Enable debug logging to see retrieval details:
@@ -218,6 +301,7 @@ Look for log tags: `[RAG Manager]`, `[Chunked-Embeddings Strategy]`, `[BM25 Stra
 | `limit`                     | int    | `5`                 | Max results from this strategy                               |
 | `embedding_batch_size`      | int    | `50`                | Chunks per embedding request                                 |
 | `max_embedding_concurrency` | int    | `3`                 | Max concurrent embedding requests                            |
+| `max_indexing_concurrency`  | int    | `3`                 | Max concurrent file-indexing tasks                           |
 | `chunking.size`             | int    | `1500`              | Chunk size in characters (`4000` when `code_aware` is set)   |
 | `chunking.overlap`          | int    | `75`                | Overlap between chunks in characters                         |
 | `chunking.code_aware`       | bool   | `false`             | AST-based chunking (Go files only)                           |

pkg/agent/agent.go

@@ -645,10 +645,10 @@ func (a *Agent) ensureToolSetsAreStarted(ctx context.Context) {
 			continue
 		}
 		if toolSet.ShouldReportFailure() {
-			slog.WarnContext(ctx, "Toolset start failed; will retry on next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.WarnContext(ctx, "Toolset start failed; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 			a.AddToolWarning(fmt.Sprintf("%s start failed: %v", desc, err))
 		} else {
-			slog.DebugContext(ctx, "Toolset still unavailable; retrying next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.DebugContext(ctx, "Toolset still unavailable; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 		}
 	}
 }

pkg/model/provider/dmr/embed.go

@@ -17,6 +17,7 @@ import (
 	"github.com/openai/openai-go/v3"
 
 	"github.com/docker/docker-agent/pkg/model/provider/base"
+	"github.com/docker/docker-agent/pkg/model/provider/oaistream"
 	"github.com/docker/docker-agent/pkg/rag/types"
 )
 
@@ -50,7 +51,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 		Model: c.ModelConfig.Model,
 	})
 	if err != nil {
-		return nil, fmt.Errorf("failed to create embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/model/provider/dmr/embed_test.go

@@ -0,0 +1,51 @@
+package dmr
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	openaisdk "github.com/openai/openai-go/v3"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/docker/docker-agent/pkg/config/latest"
+	"github.com/docker/docker-agent/pkg/modelerrors"
+)
+
+// TestCreateBatchEmbedding_429SurfacesAsStatusError proves that when the
+// embedding endpoint returns HTTP 429 the error propagates as
+// *modelerrors.StatusError so the StartableToolSet backoff gate can arm.
+func TestCreateBatchEmbedding_429SurfacesAsStatusError(t *testing.T) {
+	t.Parallel()
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusTooManyRequests)
+		_, _ = w.Write([]byte(`{"error":{"message":"rate limit exceeded","type":"requests","code":"rate_limit_exceeded"}}`))
+	}))
+	t.Cleanup(srv.Close)
+
+	cfg := &latest.ModelConfig{
+		Provider: "dmr",
+		Model:    "ai/nomic-embed-text",
+		BaseURL:  srv.URL + "/v1",
+	}
+	client, err := NewClient(t.Context(), cfg, nil)
+	require.NoError(t, err)
+
+	_, err = client.CreateBatchEmbedding(t.Context(), []string{"hello"})
+	require.Error(t, err, "expected error from 429 response")
+
+	// The critical assertion: the OpenAI SDK error must be wrapped in
+	// *modelerrors.StatusError by WrapOpenAIError so the backoff gate can arm.
+	var se *modelerrors.StatusError
+	require.ErrorAs(t, err, &se,
+		"error must contain *modelerrors.StatusError so the backoff gate can arm")
+	assert.Equal(t, http.StatusTooManyRequests, se.StatusCode)
+
+	// Cross-check: the underlying SDK error is also reachable.
+	var apiErr *openaisdk.Error
+	assert.ErrorAs(t, err, &apiErr,
+		"underlying *openaisdk.Error must be reachable through the chain")
+}

pkg/model/provider/openai/client.go

@@ -1165,7 +1165,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 	response, err := client.Embeddings.New(ctx, params)
 	if err != nil {
 		slog.ErrorContext(ctx, "OpenAI batch embedding request failed", "error", err)
-		return nil, fmt.Errorf("failed to create batch embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create batch embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/modelerrors/modelerrors.go

@@ -781,3 +781,18 @@ func scalarString(v any) string {
 		return fmt.Sprint(v)
 	}
 }
+
+// RetryableHTTPStatus reports whether err contains an HTTP status that warrants
+// backoff (429 rate-limit, 408 request-timeout, or a 5xx server error). It
+// checks for a *StatusError in the chain first; if none is found it falls back
+// to matching \b([45]\d{2})\b in the error message, which can produce false
+// positives for port numbers or similar numeric patterns. Callers that need
+// strict StatusError-only classification should pre-filter with errors.As.
+// No context-error handling is performed.
+func RetryableHTTPStatus(err error) bool {
+	code := extractHTTPStatusCode(err)
+	if code == 0 {
+		return false
+	}
+	return code == http.StatusTooManyRequests || isRetryableStatusCode(code)
+}

pkg/modelerrors/modelerrors_test.go

@@ -944,3 +944,57 @@ func TestScalarStringEdgeCases(t *testing.T) {
 		})
 	}
 }
+
+func TestRetryableHTTPStatus(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name     string
+		err      error
+		expected bool
+	}{
+		// Retryable HTTP codes via *StatusError.
+		{name: "429 rate-limit StatusError", err: &StatusError{StatusCode: 429, Err: errors.New("rate limited")}, expected: true},
+		{name: "408 request timeout StatusError", err: &StatusError{StatusCode: 408, Err: errors.New("timeout")}, expected: true},
+		{name: "500 server error StatusError", err: &StatusError{StatusCode: 500, Err: errors.New("internal server error")}, expected: true},
+		{name: "503 unavailable StatusError", err: &StatusError{StatusCode: 503, Err: errors.New("service unavailable")}, expected: true},
+		{name: "529 overloaded StatusError", err: &StatusError{StatusCode: 529, Err: errors.New("overloaded")}, expected: true},
+		// Non-retryable HTTP codes via *StatusError.
+		{name: "400 bad request StatusError", err: &StatusError{StatusCode: 400, Err: errors.New("bad request")}, expected: false},
+		{name: "401 unauthorized StatusError", err: &StatusError{StatusCode: 401, Err: errors.New("unauthorized")}, expected: false},
+		{name: "403 forbidden StatusError", err: &StatusError{StatusCode: 403, Err: errors.New("forbidden")}, expected: false},
+		{name: "404 not found StatusError", err: &StatusError{StatusCode: 404, Err: errors.New("not found")}, expected: false},
+		// Plain string errors without HTTP codes: must stay non-retryable so
+		// MCP/LSP "connection refused" errors are not paced.
+		{name: "connection refused", err: errors.New("connection refused: dial tcp 127.0.0.1:9999"), expected: false},
+		{name: "no such host", err: errors.New("no such host: example.invalid"), expected: false},
+		// Plain text that DOES contain a retryable HTTP code: the regex fallback
+		// in extractHTTPStatusCode finds it. NOTE: RetryableHTTPStatus returns
+		// true here, but startBackoffRetryable (the toolset gate classifier)
+		// requires a *StatusError and returns false for the same input — the
+		// narrowing is deliberate to prevent port-number/chunk-count false positives.
+		{name: "503 in plain text", err: errors.New("upstream: 503 Service Unavailable"), expected: true},
+		{name: "429 in plain text", err: errors.New("provider said: 429 Too Many Requests"), expected: true},
+		// HTTP-status precedence: a StatusError{429} wrapped alongside
+		// context.DeadlineExceeded must return true — the HTTP signal wins.
+		{
+			name: "429 StatusError + DeadlineExceeded in chain",
+			err: fmt.Errorf("start budget exceeded: %w, provider said: %w",
+				context.DeadlineExceeded,
+				&StatusError{StatusCode: 429, Err: errors.New("rate limited")}),
+			expected: true,
+		},
+		// Bare context errors: no HTTP code, must return false.
+		{name: "bare DeadlineExceeded", err: context.DeadlineExceeded, expected: false},
+		{name: "bare Canceled", err: context.Canceled, expected: false},
+		{name: "nil error", err: nil, expected: false},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			got := RetryableHTTPStatus(tc.err)
+			assert.Equal(t, tc.expected, got)
+		})
+	}
+}