Skip to content

Fix model-aware fallback on first request#3

Open
FarisZR wants to merge 3 commits intomainfrom
fix/model-availability-fallback
Open

Fix model-aware fallback on first request#3
FarisZR wants to merge 3 commits intomainfrom
fix/model-availability-fallback

Conversation

@FarisZR
Copy link
Owner

@FarisZR FarisZR commented Mar 6, 2026

Summary

  • retry the same Copilot request against another eligible account when the first account reports that the requested model is unavailable
  • remember per-account unsupported models without blocking unrelated models, and keep agent stickiness on the working fallback account
  • add regression coverage for model-aware fallback plus docs for the new behavior

Testing

  • bun run test
  • bun run lint
  • bun run build
  • timeout 10 opencode . --print-logs

Summary by CodeRabbit

  • New Features

    • Automatic same-request model fallback: if a model isn’t available on the chosen account, the request retries on another eligible account and notifies you of the switch.
  • Documentation

    • README and architecture docs updated with guidance on model-aware fallback, per-request retry flow, and user-facing messages.
  • UI / Notifications

    • Account-selection toasts can show human-readable fallback messages identifying the chosen account and reason.
  • Tests

    • Expanded tests covering fallback, unsupported-model expiry, and selection behavior.

@coderabbitai
Copy link

coderabbitai bot commented Mar 6, 2026

📝 Walkthrough

Walkthrough

Adds model-aware same-request fallback: when a selected account reports a model unavailable, the account is marked unsupported for that model and the fetch logic retries the same request on another eligible account. Changes touch account selection, fetch retry flow, model availability cache, notifications, docs, and tests.

Changes

Cohort / File(s) Summary
Documentation
README.md, docs/ARCHITECTURE.md
Documented "Model-Aware Fallback" feature and updated architecture notes describing per-request retry flow, account marking, and user-facing messages.
Account Management
src/accounts/manager.ts
Added isAccountEligible(account, modelId, host, excludedAccountIds?); updated selectAccount signature to accept excludedAccountIds and delegate eligibility checks to the new helper.
Fetch Handling
src/fetch/copilot-fetch.ts
Reworked request loop to prepare selection, refresh tokens, detect model-unavailable responses (body + status), mark account unsupported, and retry using selectFallback/selectAccount. Enhanced headers, backoff, agent coordination, and notifier integration.
Model Availability Tracking
src/models/availability.ts
CacheEntry.models now nullable; added unsupportedModels array, markUnsupported() behavior, and public isUnsupported(account, modelId) to track per-account unsupported-model markers with TTL expiry.
Usage Notifications
src/observe/usage.ts
Extended UsageNotifier.accountSelected to accept optional message parameter; logging and toast display now prefer provided message.
Tests
test/accounts.test.ts, test/availability.test.ts, test/fetch.test.ts
Added tests for unsupported-model markers (including TTL expiry), selection behavior with unknown/empty model lists, and fetch retry behavior for model-unavailable responses; updated test helpers and mocked fetch/notifier usage.

Sequence Diagram(s)

sequenceDiagram
    participant Agent as Agent/Client
    participant Fetch as Copilot Fetch
    participant Manager as Account Manager
    participant Cache as Model Cache
    participant Notifier as Usage Notifier

    Agent->>Fetch: Request(modelId, host, initiator)
    Fetch->>Manager: selectAccount(modelId, host)
    Manager->>Cache: check models / isUnsupported(account, modelId)
    Cache-->>Manager: eligibility result
    Manager-->>Fetch: selected account

    Fetch->>Fetch: prepare headers, refresh token if needed
    Fetch->>CopilotAPI: send request
    alt model unavailable (400/403/404 + body)
        CopilotAPI-->>Fetch: model-unavailable response
        Fetch->>Cache: markUnsupported(account, modelId)
        Cache-->>Fetch: updated unsupportedModels
        Fetch->>Manager: selectAccount(modelId, host, excludedAccountIds={account})
        Manager->>Cache: check eligibility excluding previous
        Cache-->>Manager: fallback account
        Manager-->>Fetch: fallback account
        Fetch->>CopilotAPI: retry request on fallback
        CopilotAPI-->>Fetch: success 200
        Fetch->>Notifier: accountSelected(fallback, reason, message)
    else success
        CopilotAPI-->>Fetch: success 200
        Fetch->>Notifier: accountSelected(account, reason)
    end
Loading
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix model-aware fallback on first request' directly and clearly describes the main functionality added: handling model unavailability by falling back to another account on the initial request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/model-availability-fallback

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/accounts/manager.ts`:
- Around line 121-123: The code currently mutates account.models by filtering
out unsupported model entries; instead, revert that mutation and ensure
unsupported-model removals are recorded only in the ModelAvailabilityCache used
by isAccountEligible(). Locate the block that checks
Array.isArray(account.models) and remove the line that assigns account.models =
account.models.filter(...); instead update or call the ModelAvailabilityCache
API (the same mechanism used to mark TTL-scoped unsupported models) to record
the unsupported model for that account so account.models remains unchanged and
the exclusion expires with the cache.
- Around line 140-144: The current check treats an empty models array as
"unknown" by returning true when models.length === 0; update the logic in the
method that reads this.availability.get(account) (the block assigning
cachedModels and models and returning models.includes(modelId)) so that
null/undefined still means "unknown" (keep returning true), but an explicit
empty array (models.length === 0) is treated as "supports nothing" and returns
false; ensure the final return uses models.includes(modelId) only when models is
a non-empty array.

In `@src/fetch/copilot-fetch.ts`:
- Around line 169-175: The function isModelUnavailableResponse currently checks
only for 400 and 404, but must also treat 403 as a model-unavailable case;
update isModelUnavailableResponse (the async function
isModelUnavailableResponse(response: Response, modelId: string)) to include
response.status === 403 in the initial status check so that 403 responses are
routed to isModelUnavailableBody(modelId) and ultimately call
markModelUnsupported for that model instead of triggering account-wide
auth-failure handling.
- Around line 226-231: The current updateHostLock function lets a non-agent
request overwrite the sticky agent accountId even when the agent was recently
active; change updateHostLock (which updates lockByHost for host) to preserve
previous.accountId if previous.lastAgentAt indicates the agent was recently
active and the current request is not an agent (i.e., if !isAgent and
previous?.lastAgentAt is recent/agentRecentlyActive), otherwise update accountId
as before; still update lastAgentAt only when isAgent (Date.now()) or carry
previous.lastAgentAt when not an agent.
- Line 150: Add explicit TypeScript return type annotations to the six new
helper functions to satisfy the repo rule: annotate isModelUnavailableBody with
: boolean; isModelUnavailableResponse with : Promise<boolean>; updateHostLock
with : void; prepareSelection with : Promise<NonNullable<ReturnType<typeof
manager.selectAccount>>>; buildFallbackMessage with : string; and selectFallback
with : { fallback: NonNullable<ReturnType<typeof manager.selectAccount>>;
message: string } | null. Locate the functions by name in copilot-fetch.ts
(isModelUnavailableBody, isModelUnavailableResponse, updateHostLock,
prepareSelection, buildFallbackMessage, selectFallback) and add the specified
return type annotations to their declarations.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a6787498-9e7f-43ae-b6df-fafcb3a467c1

📥 Commits

Reviewing files that changed from the base of the PR and between 74483c2 and 6b82a25.

📒 Files selected for processing (9)
  • README.md
  • docs/ARCHITECTURE.md
  • src/accounts/manager.ts
  • src/fetch/copilot-fetch.ts
  • src/models/availability.ts
  • src/observe/usage.ts
  • test/accounts.test.ts
  • test/availability.test.ts
  • test/fetch.test.ts

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/accounts/manager.ts`:
- Around line 125-130: The new public methods lack explicit return types: add an
explicit ": boolean" return type to isAccountEligible(account: CopilotAccount,
modelId: string, host: string, excludedAccountIds: Set<string> = new Set()) and
annotate selectAccount(...) with ": AccountSelection | null" (the widened
signature around selectAccount in the block roughly corresponding to lines
187-191) so the manager's public API is not inferred; update both function
signatures accordingly and ensure any helper/internal functions keep their
existing types.

In `@src/fetch/copilot-fetch.ts`:
- Around line 150-176: The current heuristic in isModelUnavailableBody is too
permissive; change it so we only classify a miss when the body either (A)
contains the requested modelId (modelId !== 'unknown') together with one of the
error phrases, or (B) contains a stronger model-specific phrase where "model"
appears adjacent to the phrase (e.g. "model not found", "model is not
available", "model unsupported") — implement this by replacing the simple
includes checks with: require modelId presence plus phrase, or test a regex that
matches /model.{0,10}(not found|does not exist|not available|unsupported|no
access)/ (case-insensitive). Keep isModelUnavailableResponse behavior but call
the tightened isModelUnavailableBody; this prevents markModelUnsupported() from
caching unrelated 400/403/404 errors that merely mention "model" or generic "not
found".

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2876b2d5-64db-49fa-aee2-6f176e4fbe70

📥 Commits

Reviewing files that changed from the base of the PR and between 6b82a25 and e72eada.

📒 Files selected for processing (4)
  • src/accounts/manager.ts
  • src/fetch/copilot-fetch.ts
  • test/accounts.test.ts
  • test/fetch.test.ts

Comment on lines +125 to +130
isAccountEligible(
account: CopilotAccount,
modelId: string,
host: string,
excludedAccountIds: Set<string> = new Set()
) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

Add explicit return types to the new public methods.

isAccountEligible() and the widened selectAccount() signature are now part of the manager’s public surface, but both still rely on inference. Please annotate them explicitly (: boolean and : AccountSelection | null) so future edits cannot silently widen the API.

♻️ Proposed fix
   isAccountEligible(
     account: CopilotAccount,
     modelId: string,
     host: string,
     excludedAccountIds: Set<string> = new Set()
-  ) {
+  ): boolean {
   selectAccount(
     modelId: string,
     host: string,
     excludedAccountIds: Set<string> = new Set()
-  ): AccountSelection | null {
+  ): AccountSelection | null {

As per coding guidelines, "Prefer explicit type annotations over inference".

Also applies to: 187-191

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/accounts/manager.ts` around lines 125 - 130, The new public methods lack
explicit return types: add an explicit ": boolean" return type to
isAccountEligible(account: CopilotAccount, modelId: string, host: string,
excludedAccountIds: Set<string> = new Set()) and annotate selectAccount(...)
with ": AccountSelection | null" (the widened signature around selectAccount in
the block roughly corresponding to lines 187-191) so the manager's public API is
not inferred; update both function signatures accordingly and ensure any
helper/internal functions keep their existing types.

Comment on lines +150 to +176
function isModelUnavailableBody(bodyText: string, modelId: string): boolean {
const normalized = bodyText.toLowerCase();
const mentionsModel =
normalized.includes('model') ||
(modelId !== 'unknown' && normalized.includes(modelId.toLowerCase()));

if (!mentionsModel) return false;

return [
'not found',
'does not exist',
'not available',
'not supported',
'unsupported',
'no access to model',
'access to this model',
].some((phrase) => normalized.includes(phrase));
}

async function isModelUnavailableResponse(response: Response, modelId: string): Promise<boolean> {
if (response.status !== 400 && response.status !== 403 && response.status !== 404) return false;
const bodyText = await response
.clone()
.text()
.catch(() => '');
return isModelUnavailableBody(bodyText, modelId);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Tighten the model-unavailable classifier before caching the miss.

Right now any 400/403/404 body that contains the word model and a generic phrase like not found or not available returns true. That will misclassify unrelated errors such as {"error":"not found","model":"gpt-5.4"} and then markModelUnsupported() will wrongly exclude that account for the TTL window. Please require either the requested model id or a stronger model-specific phrase before treating the response as a model miss.

💡 Narrow the heuristic
 function isModelUnavailableBody(bodyText: string, modelId: string): boolean {
   const normalized = bodyText.toLowerCase();
-  const mentionsModel =
-    normalized.includes('model') ||
-    (modelId !== 'unknown' && normalized.includes(modelId.toLowerCase()));
+  const mentionsRequestedModel =
+    modelId !== 'unknown' && normalized.includes(modelId.toLowerCase());
+  const mentionsModel = normalized.includes('model') || mentionsRequestedModel;
 
-  if (!mentionsModel) return false;
+  if (!mentionsModel) return false;
+
+  const strongModelSpecificMatch = [
+    'not supported',
+    'unsupported',
+    'no access to model',
+    'access to this model',
+  ].some((phrase) => normalized.includes(phrase));
+  if (strongModelSpecificMatch) return true;
 
   return [
     'not found',
     'does not exist',
     'not available',
-    'not supported',
-    'unsupported',
-    'no access to model',
-    'access to this model',
-  ].some((phrase) => normalized.includes(phrase));
+  ].some((phrase) => mentionsRequestedModel && normalized.includes(phrase));
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/fetch/copilot-fetch.ts` around lines 150 - 176, The current heuristic in
isModelUnavailableBody is too permissive; change it so we only classify a miss
when the body either (A) contains the requested modelId (modelId !== 'unknown')
together with one of the error phrases, or (B) contains a stronger
model-specific phrase where "model" appears adjacent to the phrase (e.g. "model
not found", "model is not available", "model unsupported") — implement this by
replacing the simple includes checks with: require modelId presence plus phrase,
or test a regex that matches /model.{0,10}(not found|does not exist|not
available|unsupported|no access)/ (case-insensitive). Keep
isModelUnavailableResponse behavior but call the tightened
isModelUnavailableBody; this prevents markModelUnsupported() from caching
unrelated 400/403/404 errors that merely mention "model" or generic "not found".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant