Fix Openrouter functionality#6
Merged
zackarychapple merged 32 commits intomainfrom Oct 28, 2025
Merged
Conversation
Tool calls and oracle
zackarychapple
approved these changes
Oct 28, 2025
zackarychapple
approved these changes
Oct 28, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's the issues or discussion related to this PR ?
Prior to this PR, when users selected an OpenRouter model in interactive mode and executed batch benchmarks, the OpenRouter adapter would always fall back to the default model (minimax/minimax-m2:free) instead of using the selected model. This occurred because the batch execution logic in
executeMultipleBenchmarks()was only passing model parameters to anthropic and claude-code agents, while explicitly passing undefined to all other agents including openrouter.Users reported seeing warnings like:
What's added in this PR?
• Fixed OpenRouter Model Selection in Batch Execution: Updated
executeMultipleBenchmarks()function to includeopenrouterin the list of agents that receive model parameters, preventing it from always falling back to the default model• Enhanced Model Source Tracking: Added
modelSourceproperty andgetModelSource()method toOpenRouterAdapterto track whether model came from parameter, environment variable, or default fallback• Improved Cost Calculation: Implemented proper USD cost calculation based on actual OpenRouter API pricing data instead of hardcoded zero values, including pricing cache and intelligent fallback pricing for unknown models
• Silent Execution Mode: Removed verbose console.log statements from
OpenRouterAdapterto matchAnthropicAdapter's clean execution behavior, showing only essential error logging and final CLI summary• Web Dashboard Model Name Prioritization: Updated all dashboard pages (Agents, Overview, Runs, Batch Details) to display specific model names (e.g.,
mistralai/devstral-small-2505:free) as primary identifiers instead of generic agent types like "openrouter"• Enhanced User Feedback: Added comprehensive model source detection with clear visual indicators and actionable guidance when default models are used
What are the steps to test this PR?
Here are the instructions for environment variables setup:
Create a
.envfile in the root directory of the project (*\ze-benchmarks\.env):pnpm bench(select the openrouter models)Documentation update for this PR (if applicable)?
not needed
(Optional) What's left to be done for this PR?
(Optional) What's the potential risk and how to mitigate it?
Who do you wish to review this PR other than required reviewers?
@zackarychapple @Nsttt
Examples:
(Required) Pre-PR/Merge checklist