403 error still doesn't failover on bucket profiles

[acoliver]

What happened?

✕ [API Error: 403 {"type":"error","error":{"type":"permission_error","message":"OAuth token has been revoked. Please obtain a new token."},"request_id":"req_011CXBZxZCA5PAvdsNYjj5KJ"} (Status: 403)]

it isn't obvious if it tries to fail over

What did you expect to happen?

just fail over to the other bucket

Client information

About LLxprt Code

CLI Version 0.8.0-nightly.260115.5ea41cda7
Git Commit eabbb24
Model anthropic:claude-opus-4-5-20251101
Provider anthropic
Sandbox no sandbox
OS darwin
Auth Method provider

INFO: # LLxprt Diagnostics

Provider Information

• Active Provider: anthropic
• Current Model: claude-opus-4-5-20251101
• Current Profile: opusthinkingbucketed
• API Key: unavailable via runtime helpers

Model Parameters

• No custom model parameters set

Ephemeral Settings

• Compression & Context:

  • context-limit: 200000

• Other Settings:

• activeProvider: "anthropic"

  • tools: {}
  • shell-replacement: true
  • reasoning: {"enabled":true,"includeInContext":true,"includeInResponse":true,"stripFromContext":"allButLast"}
  • max_tokens: 10000
  • currentProfile: "opusthinkingbucketed"
  • emojifilter: "auto"

System Information

• Platform: darwin
• Node Version: v25.2.1
• Working Directory: /Users/acoliver/projects/seredes
• Debug Mode: Disabled
• Approval Mode: yolo

Compression

• Threshold: default
• Context Limit: 200000

Settings

• Theme: Green Screen
• Selected Auth Type: provider
• Default Profile: none
• Sandbox: disabled

IDE Integration

• IDE Mode: Disabled

LLxprt Diagnostics

Provider Information

• Active Provider: anthropic
• Current Model: claude-opus-4-5-20251101
• Current Profile: opusthinkingbucketed
• API Key: unavailable via runtime helpers

Model Parameters

• No custom model parameters set

Ephemeral Settings

• Compression & Context:
  • context-limit: 200000
• Other Settings:
  • activeProvider: "anthropic"
  • tools: {}
  • shell-replacement: true
  • reasoning: {"enabled":true,"includeInContext":true,"includeInResponse":true,"stripFromContext":"allButLast"}
  • max_tokens: 10000
  • currentProfile: "opusthinkingbucketed"
  • emojifilter: "auto"

System Information

• Platform: darwin
• Node Version: v25.2.1
• Working Directory: /Users/acoliver/projects/seredes
• Debug Mode: Disabled
• Approval Mode: yolo

Compression

• Threshold: default
• Context Limit: 200000

Settings

• Theme: Green Screen
• Selected Auth Type: provider
• Default Profile: none
• Sandbox: disabled

IDE Integration

• IDE Mode: Disabled
• IDE Client: Connected

MCP (Model Context Protocol)

• MCP Servers: None configured
• MCP Server Command: not set

Memory/Context

• User Memory: 758 characters
• Context Files: 1 files

Tools

• Available Tools: 22
• First 10 Tools: codesearch, delete_line_range, direct_web_fetch, replace, exa_web_search, glob, google_web_search, google_web_fetch, insert_at_line, list_subagents

Telemetry

• Usage Statistics: Enabled

LLxprt Diagnostics

Provider Information

• Active Provider: anthropic
• Current Model: claude-opus-4-5-20251101
• Current Profile: opusthinkingbucketed
• API Key: unavailable via runtime helpers

Model Parameters

• No custom model parameters set

Ephemeral Settings

• Compression & Context:
  • context-limit: 200000
• Other Settings:
  • activeProvider: "anthropic"
  • tools: {}
  • shell-replacement: true
  • reasoning: {"enabled":true,"includeInContext":true,"includeInResponse":true,"stripFromContext":"allButLast"}
  • max_tokens: 10000
  • currentProfile: "opusthinkingbucketed"
  • emojifilter: "auto"

System Information

• Platform: darwin
• Node Version: v25.2.1
• Working Directory: /Users/acoliver/projects/seredes
• Debug Mode: Disabled
• Approval Mode: yolo

Compression

• Threshold: default
• Context Limit: 200000

Settings

• Theme: Green Screen
• Selected Auth Type: provider
• Default Profile: none
• Sandbox: disabled

IDE Integration

• IDE Mode: Disabled
• IDE Client: Connected

MCP (Model Context Protocol)

• MCP Servers: None configured
• MCP Server Command: not set

Memory/Context

• User Memory: 758 characters
• Context Files: 1 files

Tools

• Available Tools: 22
• First 10 Tools: codesearch, delete_line_range, direct_web_fetch, replace, exa_web_search, glob, google_web_search, google_web_fetch, insert_at_line, list_subagents

Telemetry

• Usage Statistics: Enabled

OAuth Tokens

Provider Tokens

• anthropic:
  • Buckets: 3
  • acoliver@vybestack.dev:
    • Status: Authenticated
    • Expires: 2026-01-17T02:59:10.000Z
    • Time Remaining: 7h 58m
    • Refresh Token: Available
  • claudius@vybestack.dev:
    • Status: Authenticated
    • Expires: 2026-01-17T02:58:27.000Z
    • Time Remaining: 7h 58m
    • Refresh Token: Available
  • default:
    • Status: Authenticated
    • Expires: 2026-01-17T02:58:27.000Z
    • Time Remaining: 7h 58m
    • Refresh Token: Available

(note this is after /auth anthropic logout --all and "continue")

Login information

No response

Anything else we need to know?

No response

[acoliver]

@CodeRabbit look into this

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

⚠️ Possible Duplicate Issue

• ✕ [API Error: 403 {"type":"error","error":{"type":"permission_error","message":"OAuth token has been revoked. Please obtain a new token."},"request_id":"req_011CX6Cwz8rJqKAK3zdtL3aC"} (Status: 403)] #1123

🔗 Related PRs

#807 - Integrate bucket failover for AnthropicProvider and OpenAIProvider - fixes #686 [merged]
#900 - Fix OAuth logout bucket cleanup + 429 bucket failover (fixes #192) [merged]
#906 - OAuth: bucket-aware refresh + proactive renewals [closed]
#1009 - fix: trigger bucket failover immediately on 429/402/401 errors (#1006) [merged]
#1125 - fix: handle 403 OAuth token revoked errors with refresh retry and bucket failover [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch details of issue `#1151`
gh issue view 1151 --json title,body,author,state,labels,comments --jq '{title: .title, body: .body, author: .author.login, state: .state, labels: [.labels[].name], comments: [.comments[] | {author: .author.login, body: .body}]}'

Length of output: 5816

---

🏁 Script executed:

#!/bin/bash

# Search for bucket-related code and failover logic
echo "=== Searching for bucket failover logic ==="
rg -n "failover" -A 3 -B 3

echo -e "\n=== Searching for bucket selection/rotation ==="
rg -n "bucket" -i -A 2 -B 2 | head -100

echo -e "\n=== Searching for 403 error handling ==="
rg -n "403" -A 3 -B 3

echo -e "\n=== Searching for OAuth token revoked error handling ==="
rg -n "revoked|permission_error" -i -A 3 -B 3

echo -e "\n=== Finding authentication and profile related files ==="
fd -e ts -e js | rg -i "auth|bucket|profile" | head -20

Length of output: 281694

---

🏁 Script executed:

#!/bin/bash

echo "=== Searching for 403 permission_error handling ==="
rg -n "permission_error|403" packages/core/src/core/bucketFailoverIntegration.ts packages/core/src/utils/retry.ts -A 5 -B 5

echo -e "\n=== Checking shouldFailover function implementation ==="
rg -n "function shouldFailover|const shouldFailover" packages/core/src/core/bucketFailoverIntegration.ts -A 30

echo -e "\n=== Checking Anthropic provider failover integration ==="
rg -n "getBucketFailoverHandler|onPersistent429" packages/core/src/providers/anthropic/AnthropicProvider.ts -A 10 -B 3 | head -100

echo -e "\n=== Looking for issue 1151 or related fixes ==="
rg -n "issue.*1151|1151|fix.*403.*failover" --type md | head -30

echo -e "\n=== Check if 403 is in failover status codes ==="
rg -n "failover.*403|403.*failover" packages/core/src -A 2 -B 2 | head -50

Length of output: 10445

---

@acoliver thanks for the ping — I looked into this. The code path should fail over on 403 after one refresh retry, so this likely comes down to the failover handler not being wired in your runtime after re-auth, or logs not showing the attempt. Here’s a quick triage and what to try next.

What should happen

• retry.ts treats 401/403 as auth errors, does one refresh retry, then triggers bucket failover if the onPersistent429 failover handler is present.
• AnthropicProvider asks the runtime config for a BucketFailoverHandler, flushes the runtime auth scope, rebuilds the client, and continues with the next bucket on success.

What to try (fast checks)

1. Reload the profile in the same session after your re-auth

• After /auth anthropic login steps, run:
  /profile load opusthinkingbucketed
  This reattaches the bucket failover handler to the active runtime.

2. Turn on debug to confirm failover wiring

• Run your prompt with:
  LLXPRT_DEBUG=llxprt:*
  You should see one of these in the logs from the Anthropic provider:
• [issue1029] Failover handler found: enabled=..., currentBucket=...
• Attempting bucket failover on persistent 429
• Bucket failover successful: ->
  If instead you see:
• [issue1029] No failover handler found on any config...
  …then the handler isn’t wired for this session; reloading the profile (Step 1) should fix that.

3. Force a clean runtime auth scope

• If the error happened mid-session, the cached token may linger. Before re-running:
• Reload the profile as in Step 1 (this triggers an auth re-resolution).
• Alternatively, switch to a different profile then back, to ensure a fresh runtimeId is used.

4. Quick visibility tweak

• While this is handled automatically, you can add a temporary banner to see the decision point by enabling DEBUG and watching for:
• 401/403 error detected, retrying once to allow refresh before bucket failover
• Attempting bucket failover after status 403

Short-term configuration workaround (optional)

• If you’re also using a load balancer profile around the model profile, you can include 403 in LB failover triggers:
  /set failover_status_codes [400,401,402,403,404,429,500,501,502,503,504]
  /profile save loadbalancer failover
  This won’t affect bucket failover directly but provides an extra layer.

If you can share a debug snippet

• From a single failing call (with LLXPRT_DEBUG=llxprt:*), paste the lines around:
• [issue1029] Checking failover handler availability
• 401/403 error detected, retrying once...
• Attempting bucket failover…
  That will confirm whether the handler is missing vs. the error not being recognized as 403 on the second attempt.

Proposed follow-ups (I can put a plan together if you want this in a PR)

• Add a one-line debug banner on first 401/403 that says: “Will fail over to next bucket if auth refresh fails.”
• Add a CLI self-check: /diagnostics failover to print:
  • Current profile, bucket list/order
  • Handler installed: yes/no
  • Current bucket, next bucket
• Clarify docs that bucket failover also covers 401/403 (revoked/unauthorized), not only 429/402.

Want me to open a follow-up issue to add the /diagnostics failover and the extra debug banner?

[acoliver]

@CodeRabbit these are the diagnostics before a reauth

OAuth Tokens

Provider Tokens

• anthropic:
  • Buckets: 3
  • acoliver@vybestack.dev:
    • Status: Authenticated
    • Expires: 2026-01-17T02:59:10.000Z
    • Time Remaining: 7h 53m
    • Refresh Token: Available
  • claudius@vybestack.dev:
    • Status: Authenticated
    • Expires: 2026-01-17T03:05:17.000Z
    • Time Remaining: 7h 59m
    • Refresh Token: Available
  • default:
    • Status: Authenticated
    • Expires: 2026-01-17T02:58:27.000Z
    • Time Remaining: 7h 52m
    • Refresh Token: Available

(a different instance) -- wondering if a refresh would have fixed the token? Wondering what "revokes" it.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for the 403 handling and refresh retry logic
echo "=== Checking retry.ts for 403 handling and refresh behavior ==="
rg -n "403|revoked|shouldRetry|isAuthError" packages/core/src/utils/retry.ts -A 5 -B 2

echo -e "\n=== Checking OAuth refresh implementation ==="
rg -n "refreshToken|refresh.*token|attemptRefresh" packages/core/src -A 3 -B 2 | head -80

echo -e "\n=== Checking what triggers token refresh in auth flow ==="
rg -n "class.*Auth|refreshAccessToken|getValidToken" packages/core/src/auth -A 5 -B 2 | head -100

Length of output: 10468

---

@acoliver Good catch on the diagnostics — this perfectly illustrates the local vs. server-side token state mismatch.

Would a refresh have fixed it?

Almost certainly not. Here's why:

When Anthropic returns 403 "OAuth token has been revoked", they're saying the entire OAuth session (access token + refresh token) has been invalidated server-side. Your local diagnostics show:

• ✓ Access token exists and hasn't expired locally (7h+ remaining)
• ✓ Refresh token is available locally
• ✗ But the server has revoked the session

When you attempt to refresh a revoked token, Anthropic will also reject the refresh request with a 403 or similar error, because the refresh token is part of the same invalidated session.

What the code does automatically

Based on the retry logic I found, here's what happens on a 403:

1. First 403 error → Retry once (consecutiveAuthErrors = 1)

   • Logs: "401/403 error detected, retrying once to allow refresh before bucket failover"
   • This gives the auth layer a chance to attempt token refresh
   • In your case, the refresh attempt would also fail with 403

2. Second consecutive 403 error → Trigger bucket failover (consecutiveAuthErrors = 2)

   • Logs: "Attempting bucket failover after status 403"
   • Switches to next bucket with a fresh OAuth session

So yes, the code does try to refresh once, but when the session is revoked, that refresh attempt fails too, and then it correctly fails over to the next bucket.

What causes server-side revocation?

Common scenarios:

1. User logout elsewhere — You (or someone with access) logged out from Anthropic's web console or another client
2. Security policy — Anthropic detected suspicious activity or applied a security policy change
3. Rate/abuse detection — Too many requests or unusual patterns triggered automatic revocation
4. Session timeout — Long-running sessions sometimes get server-side invalidated even before expiry
5. Manual revocation — Admin or user explicitly revoked the app's access
6. OAuth app changes — If Anthropic changed OAuth client settings or policies

Why your local state still looked valid

Your client stores:

• Access token + expiry timestamp (calculated from expires_in when token was issued)
• Refresh token (opaque string)
• "Authenticated" status (based on local checks)

None of these tell you whether Anthropic's server still considers the session valid. The only way to know is by making an API call — which is when you got the 403.

Next steps

Since the automatic refresh → bucket failover is already working as designed (attempt refresh on first 403, failover on second 403), the question is: are you seeing the failover happen automatically, or do you still need to manually re-auth?

If failover isn't triggering:

• Double-check that you ran /profile load opusthinkingbucketed after the first 403
• Enable LLXPRT_DEBUG=llxprt:* and share the logs around the 403 errors
• We can verify whether the handler is wired and the second 403 is triggering failover

If you're curious about which specific action caused the revocation, you might check:

• Anthropic console: Recent activity / authorized apps
• Did you recently log out and back in from another device?
• Any security emails from Anthropic?

Let me know if you're seeing the automatic failover work, or if you're still getting stuck after the 403!

---

🧠 Learnings used

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2026-01-13T23:04:39.246Z
Learning: Treat Anthropic 403 permission_error "OAuth token has been revoked" as an expired/auth error: trigger a one-time refresh retry, then bucket failover if refresh fails. Avoid auto-switching providers; only switch buckets within the same provider.

---

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2025-12-13T16:04:07.079Z
Learning: Support multi-bucket failover on a single profile: ordered buckets tried in sequence; on 429 or 402 advance to next bucket; on 401 attempt auto-renew then retry once before advancing.

---

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2026-01-06T03:14:42.597Z
Learning: Auto-switching the active provider after OAuth authentication can conflict with multi-provider workflows: pre-authenticating backup providers for bucket failover, setting up load balancer profiles, authenticating GoogleWebSearch, and authenticating subagents. Users need explicit control in these scenarios.

[acoliver]

@CodeRabbit I'm wondering if it is getting revoked cause we're having multiple agents use the same oauth tokens? meaning different instances of llxprt?

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for OAuth token storage and management
echo "=== OAuth token storage implementation ==="
rg -n "class.*TokenStore|saveToken|loadToken|tokenStore" packages/core/src/auth -A 5 -B 2 | head -100

echo -e "\n=== File-based storage and concurrent access patterns ==="
rg -n "fs\.write|fs\.read|writeFile|readFile|lock|mutex" packages/core/src/auth -A 3 -B 2 | head -80

echo -e "\n=== OAuth session and instance tracking ==="
rg -n "sessionId|instanceId|clientId|deviceId" packages/core/src/auth -A 3 -B 2 | head -60

echo -e "\n=== Concurrent refresh handling ==="
rg -n "refreshInProgress|refreshing|concurrent.*refresh|race.*condition" packages/core/src -A 3 -B 2 | head -60

echo -e "\n=== Check for warnings about multi-instance usage ==="
rg -n "multiple.*instance|concurrent.*usage|shared.*token|parallel" packages/core/src/auth -A 3 -B 2

Length of output: 23243

---

🏁 Script executed:

#!/bin/bash
# Look at the token refresh implementation in detail
echo "=== Token refresh implementation ==="
rg -n "refreshAccessToken|attemptRefresh|refreshToken.*async|async.*refresh" packages/core/src/auth -A 10 -B 3 | head -120

echo -e "\n=== Check how providers refresh tokens ==="
rg -n "class.*Provider.*{" packages/core/src/providers/anthropic -A 5 | head -30

echo -e "\n=== Look for Anthropic OAuth refresh implementation ==="
fd -e ts -e js anthropic packages/core/src/auth packages/core/src/providers/anthropic | head -20

echo -e "\n=== Check for any documentation about multi-instance usage ==="
rg -n "concurrent|multi.*instance|parallel.*instance|same.*token" --type md | head -40

echo -e "\n=== Look at how tokens are cached in memory ==="
rg -n "cache.*token|token.*cache|in.*memory.*token" packages/core/src/auth packages/core/src/providers -A 3 -B 2 | head -60

Length of output: 15291

---

@acoliver Yes, that's almost certainly a major contributing factor. Great intuition! Let me explain why multiple instances sharing OAuth tokens can trigger revocation:

The Refresh Token Rotation Problem

Anthropic has adopted OAuth 2.1 as part of the Model Context Protocol (MCP) authorization spec, which enforces refresh token rotation where tokens are valid for a single use only.

Here's what happens when you run multiple llxprt instances:

The Race Condition

Scenario: Two llxprt instances (A and B) are using the same acoliver@vybestack.dev bucket tokens

1. Both tokens expire around the same time (they show ~8h remaining, so they'll expire together)
2. Instance A makes an API call → gets 403 → attempts refresh
3. Instance B makes an API call → gets 403 → attempts refresh
4. Instance A completes refresh first:
   • Gets new access_token + new refresh_token
   • Old refresh token is invalidated
   • Saves new tokens to ~/.llxprt/oauth/anthropic~acoliver@vybestack.dev.json
5. Instance B tries to refresh using the OLD refresh_token (it read it before A updated the file):
   • Anthropic rejects it — the refresh token was already used
   • Reuse detection monitors for multiple uses of the same refresh token, and if reused it signals a potential compromise, triggering safeguards like token invalidation
6. Anthropic's security response: "This looks like a stolen token attack" → revokes the entire OAuth session

Why Your Code Doesn't Prevent This

From my analysis, the codebase:

• ✓ Uses atomic file writes (temp + rename)
• ✓ Caches tokens in memory per instance
• ✗ No file locking for concurrent access
• ✗ No inter-process coordination
• ✗ No refresh-in-progress lock to prevent simultaneous refreshes

Each instance:

1. Reads token file into memory at startup
2. Caches it
3. When refresh is needed, refreshes independently
4. Writes back to shared file

Two instances never know about each other.

Evidence This Is Happening

Other users running multiple concurrent Claude Code instances over extended periods report token expiration halting all in-progress tasks simultaneously, which matches your experience.

Why This Looks Like an Attack to Anthropic

From Anthropic's perspective:

• Refresh token used → New tokens issued → Old token invalidated
• Same old refresh token used again → 🚨 Security violation detected
• Response: "Someone intercepted and is replaying this token" → Revoke everything

This is actually correct OAuth 2.1 security behavior — the goal is to make refresh tokens single-use or client-bound so they're useless to attackers if compromised.

Solutions & Mitigations

Short-term workarounds

1. Use separate buckets per instance

   • Instance 1: Use acoliver@vybestack.dev bucket
   • Instance 2: Use claudius@vybestack.dev bucket
   • Instance 3: Use default bucket
   • This avoids the shared token problem entirely

2. Stagger your instance startups

   • Start instances 30+ minutes apart
   • Their tokens will expire at different times
   • Reduces (but doesn't eliminate) collision probability

3. Use load balancer profiles instead of buckets

   • Create separate profiles for each instance
   • Each profile can have its own provider config with different buckets

Long-term fixes (would need code changes)

1. File locking during refresh

   // Use node-file-lock or similar
   async refreshWithLock(provider, bucket) {
     const lockFile = `${tokenPath}.lock`;
     await acquireLock(lockFile);
     try {
       // Check if token is still expired (another instance may have refreshed)
       const currentToken = await readToken();
       if (!isExpired(currentToken)) {
         return currentToken; // Someone else already refreshed
       }
       // Perform refresh
       return await doRefresh();
     } finally {
       await releaseLock(lockFile);
     }
   }

2. Inter-process communication

   • Use a local mutex/semaphore
   • Only one process refreshes at a time
   • Others wait and re-read the file

3. Token refresh coordination service

   • Background daemon that manages all tokens
   • All instances request tokens from the daemon
   • Daemon handles refresh coordination

4. Per-instance OAuth sessions

   • Each instance authenticates with its own OAuth session
   • No token sharing
   • More overhead, but eliminates race conditions

Immediate Action

For your current setup with 3 buckets, try this:

1. Assign each concurrent instance to a different bucket explicitly
2. If you're running more than 3 instances, you'll need to rotate or add more buckets

Want me to open an issue to track adding file locking or instance coordination for shared token scenarios? This is a real gap when running multiple instances.

---

✏️ Learnings added

Learnt from: acoliver
URL: https://github.com/vybestack/llxprt-code/issues/1151

Timestamp: 2026-01-16T21:11:21.213Z
Learning: Running multiple llxprt instances that share the same OAuth token files can trigger Anthropic token revocation due to refresh token rotation. When multiple instances try to refresh the same single-use refresh token simultaneously, Anthropic's reuse detection treats it as a security compromise and revokes the session. Mitigation: use separate buckets per instance, or implement file locking during token refresh operations.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

---

🧠 Learnings used

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2026-01-13T23:04:39.246Z
Learning: Treat Anthropic 403 permission_error "OAuth token has been revoked" as an expired/auth error: trigger a one-time refresh retry, then bucket failover if refresh fails. Avoid auto-switching providers; only switch buckets within the same provider.

---

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2026-01-06T03:14:42.597Z
Learning: Auto-switching the active provider after OAuth authentication can conflict with multi-provider workflows: pre-authenticating backup providers for bucket failover, setting up load balancer profiles, authenticating GoogleWebSearch, and authenticating subagents. Users need explicit control in these scenarios.