Skip to content

feat(models): per-provider model catalogs and model-derived context budget - #3087

Open
chimyves wants to merge 1 commit into
zhayujie:masterfrom
chimyves:feat/model-catalog-v2
Open

feat(models): per-provider model catalogs and model-derived context budget#3087
chimyves wants to merge 1 commit into
zhayujie:masterfrom
chimyves:feat/model-catalog-v2

Conversation

@chimyves

Copy link
Copy Markdown
Contributor

Fixes #3085

What

  • New models/model_catalog.py: per-provider model catalog persisted under provider_model_catalog in config.json, keyed by provider id ("zhipu", "custom:<id>"). Entries carry capability tags plus optional context_window / max_output_tokens:
    • text marks a conversational model: text-tagged entries are the main-model (chat) candidates and what the session switcher offers. vision / video / image / embedding / asr / tts route entries into the matching capability cards. A model may hold several tags (text + vision for a VL model).
    • A saved catalog replaces the vendor's preset list; an absent or empty catalog keeps today's behavior (fully backward compatible; custom_providers[].model stays as the default model).
    • Writers keep conf() and config.json in sync, so runtime reads never hit disk. Reads heal stale data (old/unknown tags, empty tag sets) to current semantics.
  • POST /api/models gains a save_catalog action (full-list replace with validation) and GET /api/models attaches each provider's catalog plus a seed: the vendor's preset models pre-typed with their real capabilities and researched specs (context window / max output for ~50 built-in models, from official docs; native-multimodal models the preset lists lagged on — Kimi K3, MiniMax M3/M2.7, ERNIE 5.0 — get their vision tag). The vendor and custom-provider modals seed the editable model table from catalog → seed → presets.
  • Capability model lists (chat / vision / asr / tts / embedding / image) are filtered per tag server-side; the session model switcher (_session_model_catalog) offers the provider's text-tagged models while keeping the "active model stays visible" behavior.
  • Context budget: the global agent_max_context_tokens is removed. Agent._get_model_context_window() prefers the catalog's context_window (falling back to the existing name heuristics) and _get_output_reserve_tokens() uses the catalog's max_output_tokens when set, so the trimming budget follows the effective model automatically — a session that switched models uses that model's window, message channels the main model's. Sub-agents keep inheriting an explicit cap (never above the input ceiling).
  • max_output_tokens is passed through to the LLM request only when configured, so gateways that reject an explicit max_tokens are unaffected.
  • Desktop app and web console: the vendor and custom-provider modals gain a model-catalog table (name, capability tags, context window, max output, add/remove rows); the stale "Max Context Tokens" setting is removed from both UIs.

Why

One-model-per-provider makes multi-model setups impossible to maintain, and the global context budget ignores the active model (small-window models 400 with an oversized budget; large windows get silently capped). This is also the prerequisite for model routing/fallback (#2747) and gives the session model switcher (#2995) a real list to switch between.

Notes

  • No config migration needed; an existing agent_max_context_tokens value is simply ignored.
  • Existing test suite untouched and green; new behavior verified with local scripts.

@zhayujie

zhayujie commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Thanks for this — the direction is great. Per-provider catalogs plus a model-derived context budget is exactly the right foundation, and the catalog → seed → presets fallback keeps existing setups working. A few things before this can be merged:

  1. Rebase onto latest master (currently CONFLICTING). The branch was cut before several master changes landed, so there are 5 conflicts, all of the "same region edited upstream" kind — none are architectural. To resolve, keep master's newer logic and layer the catalog changes on top:
  • config.py — master added agent_bindings right where you remove agent_max_context_tokens. Keep agent_bindings; drop the token key.
  • channel/web/web_channel.py — master made a custom provider's API key optional (configured = bool(raw_base) or cls._is_real_key(raw_key)). Keep that line and just add your catalog = catalog_map.get(f"custom:{pid}") or [] alongside it.
  • desktop/src/renderer/src/types.ts — the set_capability action now also accepts 'chat_fallback' + its fields. Keep that union member and add your save_catalog action as a separate member.
  • desktop/src/renderer/src/pages/settings/ModelsTab.tsx — master's key-dirty handling changed (if (keyDirty && !MASK_RE.test(apiKey)) payload.api_key = apiKey.trim()). Keep that, and wrap your save_catalog follow-up call around it.
  • desktop/src/renderer/src/pages/settings/CapabilityCard.tsx — two small conflicts in the same vein; keep upstream, re-apply the catalog bits.
  1. Please reconsider fully removing agent_max_context_tokens (my main concern). Deriving the budget from the model is the right default, but a hard removal is a breaking change for anyone who set it, and for downstream integrations that push this key into config (it silently becomes a no-op). Could we make it an optional manual override instead of deleting it? i.e. resolution order: catalog context_window → explicit agent_max_context_tokens (if set) → name heuristics. That keeps the new behavior as the default while staying backward compatible.

  2. Minor XSS check on the catalog editor. renderCatalogEditor builds rows via innerHTML from user-entered model names. You added escapeAttr for attribute values (nice) — just double-check every interpolated user value is escaped, since model names are free text.

  3. Small ask: a screenshot/GIF of the vendor + custom-provider modals with the catalog table would make review much easier, and confirming the "untouched editor doesn't overwrite the saved catalog" (catalogDirty) path in the description is reassuring.

Overall really nice work — this also unblocks a context-budget usage indicator and one-click compact/clear, which is why the accurate per-model context_window matters. Happy to help review once it's rebased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Per-provider model catalogs with per-model metadata, and derive the context budget from the active model

2 participants