Skip to content

Fix Openrouter functionality#6

Merged
zackarychapple merged 32 commits intomainfrom
fix/openrouter
Oct 28, 2025
Merged

Fix Openrouter functionality#6
zackarychapple merged 32 commits intomainfrom
fix/openrouter

Conversation

@nitink23
Copy link
Copy Markdown
Contributor

@nitink23 nitink23 commented Oct 27, 2025

What's the issues or discussion related to this PR ?

Prior to this PR, when users selected an OpenRouter model in interactive mode and executed batch benchmarks, the OpenRouter adapter would always fall back to the default model (minimax/minimax-m2:free) instead of using the selected model. This occurred because the batch execution logic in executeMultipleBenchmarks() was only passing model parameters to anthropic and claude-code agents, while explicitly passing undefined to all other agents including openrouter.
Users reported seeing warnings like:

What's added in this PR?

Fixed OpenRouter Model Selection in Batch Execution: Updated executeMultipleBenchmarks() function to include openrouter in the list of agents that receive model parameters, preventing it from always falling back to the default model

Enhanced Model Source Tracking: Added modelSource property and getModelSource() method to OpenRouterAdapter to track whether model came from parameter, environment variable, or default fallback

Improved Cost Calculation: Implemented proper USD cost calculation based on actual OpenRouter API pricing data instead of hardcoded zero values, including pricing cache and intelligent fallback pricing for unknown models

Silent Execution Mode: Removed verbose console.log statements from OpenRouterAdapter to match AnthropicAdapter's clean execution behavior, showing only essential error logging and final CLI summary

Web Dashboard Model Name Prioritization: Updated all dashboard pages (Agents, Overview, Runs, Batch Details) to display specific model names (e.g., mistralai/devstral-small-2505:free) as primary identifiers instead of generic agent types like "openrouter"

Enhanced User Feedback: Added comprehensive model source detection with clear visual indicators and actionable guidance when default models are used

What are the steps to test this PR?

Here are the instructions for environment variables setup:

Create a .env file in the root directory of the project (*\ze-benchmarks\.env):

# OpenRouter API Key
OPENROUTER_API_KEY=your_openrouter_api_key_here

# Anthropic API Key  
ANTHROPIC_API_KEY=your_anthropic_api_key_here

# Optional: Set default OpenRouter model (overrides default)
OPENROUTER_MODEL=openai/gpt-4o-mini

pnpm bench (select the openrouter models)

Documentation update for this PR (if applicable)?

not needed

(Optional) What's left to be done for this PR?

(Optional) What's the potential risk and how to mitigate it?

Who do you wish to review this PR other than required reviewers?

@zackarychapple @Nsttt

Examples:

  • Does this PR introduce breaking changes?
  • Does this include backend implementation that affects plugins, side panels and frontend?
  • Is this a UI changes that should be added to documentation?
  • Is this a UI changes on components that will make application look/behave inconsistent?

(Required) Pre-PR/Merge checklist

  • I have added/updated our documentation to cover this new behavior
  • I have added an explanation of my changes
  • I have written new tests (if applicable)
  • I have tested this locally (standing from a first time user point of view, never touch this app before)
  • I have mentioned the related person or team responsible for reviewing proposed changes
  • I have/will run tests, or ask for help to add test

@nitink23 nitink23 marked this pull request as ready for review October 27, 2025 23:20
Copy link
Copy Markdown
Member

@Nsttt Nsttt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for now

@zackarychapple zackarychapple merged commit 5f29d91 into main Oct 28, 2025
3 of 8 checks passed
@zackarychapple zackarychapple deleted the fix/openrouter branch October 28, 2025 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants