Skip to content
This repository was archived by the owner on May 25, 2026. It is now read-only.
This repository was archived by the owner on May 25, 2026. It is now read-only.

feat: Add model benchmarking and latency tracking #41

Closed

Description

Summary

Add built-in benchmarking to track model performance, latency, and reliability over time.

Motivation

Users want to know:

  • Which models are fastest for their use case
  • Which free models have the best availability
  • Historical reliability of different models

Proposed Implementation

1. Metrics to track

interface ModelMetrics {
  model: string;
  requestCount: number;
  successCount: number;
  failureCount: number;
  avgLatencyMs: number;
  p50LatencyMs: number;
  p95LatencyMs: number;
  avgTokensPerSecond: number;
  lastUsed: Date;
  errorTypes: Record<string, number>;
}

2. CLI commands

# Show model stats
npx opencode-puter-auth stats

# Show stats for specific model
npx opencode-puter-auth stats claude-opus-4-5

# Reset stats
npx opencode-puter-auth stats --reset

3. OpenCode tool

Add puter-stats tool to view metrics from within OpenCode.

4. Storage

Store metrics in ~/.config/opencode/puter-metrics.json

Technical Notes

  • Track metrics in FallbackManager and PuterChatLanguageModel
  • Use rolling windows for averages
  • Consider privacy (don't log prompts/responses)

Priority: Low

Activity

  1. chindris-mihai-alexandru commented on Feb 15, 2026

    @chindris-mihai-alexandru
    OwnerAuthor

    Consolidated into #48 to keep roadmap tracking in one place.

  2. chindris-mihai-alexandru commented on Feb 15, 2026

    @chindris-mihai-alexandru
    OwnerAuthor

    Reopening to track this item as a standalone issue while also keeping #48 as the roadmap index.

  3. chindris-mihai-alexandru commented on Mar 13, 2026

    @chindris-mihai-alexandru
    OwnerAuthor

    Implemented model metrics tracking (latency + reliability), puter-auth stats CLI and puter-stats tool, stored in puter-metrics.json. See commit fbb2428.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestroadmapTracked in roadmap/meta issuesstatus:backlogPlanned but not in progress

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions