Skip to content

Comments

Add Metal memory stats to MLLMScheduler#78

Open
janhilgard wants to merge 1 commit intowaybarrios:mainfrom
janhilgard:fix/mllm-metal-memory-stats
Open

Add Metal memory stats to MLLMScheduler#78
janhilgard wants to merge 1 commit intowaybarrios:mainfrom
janhilgard:fix/mllm-metal-memory-stats

Conversation

@janhilgard
Copy link
Collaborator

@janhilgard janhilgard commented Feb 12, 2026

Summary

  • MLLMScheduler.get_stats() was missing Metal memory metrics (metal_active_memory_gb, metal_peak_memory_gb, metal_cache_memory_gb) that the regular Scheduler already reports (scheduler.py:1818-1822)
  • SimpleEngine.get_stats() was also missing Metal memory metrics entirely
  • BatchedEngine.get_stats() nested MLLM scheduler stats under a mllm_scheduler key, but /v1/status looks for metal_* at top-level — so Metal stats were always null for vision models even after the scheduler fix
  • All three issues together caused monitoring dashboards to show no Metal memory data for vision models (e.g. Qwen3-VL on BatchedEngine)

Changes

  1. mllm_scheduler.py — Added import mlx.core as mx and Metal memory stats block to get_stats() (identical to scheduler.py:1818-1822)
  2. engine/simple.py — Added Metal memory stats to SimpleEngine.get_stats()
  3. engine/batched.py — Promote metal_*_memory_gb keys from nested mllm_scheduler dict to top-level, so /v1/status can find them (consistent with how non-MLLM stats are merged via update())

Test plan

  • Verify curl http://localhost:1236/v1/status returns non-null metal.active_memory_gb, metal.peak_memory_gb, metal.cache_memory_gb for an MLLM model (Qwen3-VL-8B — confirmed 5.76 GB)
  • Verify no regression on regular LLM scheduler stats

🤖 Generated with Claude Code

- MLLMScheduler.get_stats() was missing metal_active/peak/cache_memory_gb
- SimpleEngine.get_stats() was missing Metal memory stats entirely
- BatchedEngine.get_stats() nested MLLM stats under mllm_scheduler key,
  but /v1/status expects metal_* at top-level — promote them

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@janhilgard janhilgard force-pushed the fix/mllm-metal-memory-stats branch from a675c17 to a0a8e9c Compare February 14, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant