Skip to content

fix(monitoring): remove --pid host, consolidate cpu smoke test into c… - #111

Open
sbates130272 wants to merge 1 commit into
mainfrom
bug/compose-fix
Open

fix(monitoring): remove --pid host, consolidate cpu smoke test into c…#111
sbates130272 wants to merge 1 commit into
mainfrom
bug/compose-fix

Conversation

@sbates130272

Copy link
Copy Markdown
Collaborator

…ompose

spur-authz blocks --pid=host on all SPUR nodes. The cpu smoke test was using raw docker run with --pid host for node-exporter and nvme-exporter, causing a fatal exit 125. The monitoring compose had the same flag, which has been silently failing in all CI tests where monitoring is best-effort.

  • Remove pid: host from node-exporter and nvme-exporter in docker-compose.monitoring.yml; hsa-snoop retains it (kprobe requirement)
  • Add exporters-cpu profile covering node/nvme/rdma but not GPU exporters (amdgpu-exporter, hsa-snoop), so the cpu smoke test can use start_monitoring() without attempting to launch GPU-only services
  • Switch docker-compose.emulator.yml from bridge+published port to network_mode: host, consistent with the monitoring stack's localhost scraping
  • Rewrite spur-monitoring-cpu-smoke.sh SRUN_BODY to use compose + monitoring-lib.sh (start_monitoring/stop_monitoring) instead of reimplementing container orchestration inline with raw docker run
  • Add AIC_CPU_SMOKE=1 flag to mon_profile() in monitoring-lib.sh to select the exporters-cpu profile; add aic-vllm-emulator to MON_CONTAINERS

…ompose

spur-authz blocks --pid=host on all SPUR nodes. The cpu smoke test was
using raw docker run with --pid host for node-exporter and nvme-exporter,
causing a fatal exit 125. The monitoring compose had the same flag, which
has been silently failing in all CI tests where monitoring is best-effort.

- Remove pid: host from node-exporter and nvme-exporter in
  docker-compose.monitoring.yml; hsa-snoop retains it (kprobe requirement)
- Add exporters-cpu profile covering node/nvme/rdma but not GPU exporters
  (amdgpu-exporter, hsa-snoop), so the cpu smoke test can use start_monitoring()
  without attempting to launch GPU-only services
- Switch docker-compose.emulator.yml from bridge+published port to
  network_mode: host, consistent with the monitoring stack's localhost scraping
- Rewrite spur-monitoring-cpu-smoke.sh SRUN_BODY to use compose +
  monitoring-lib.sh (start_monitoring/stop_monitoring) instead of
  reimplementing container orchestration inline with raw docker run
- Add AIC_CPU_SMOKE=1 flag to mon_profile() in monitoring-lib.sh to select
  the exporters-cpu profile; add aic-vllm-emulator to MON_CONTAINERS

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
@sbates130272 sbates130272 self-assigned this Jul 29, 2026
@sbates130272 sbates130272 added the bug Something isn't working label Jul 29, 2026
@sbates130272

Copy link
Copy Markdown
Collaborator Author

/run-ci

@github-actions

Copy link
Copy Markdown

❌ dist-build failed for f7d9244. View logs.

@amd-ivaganev

Copy link
Copy Markdown
Collaborator

IIRC --pid host was required for vllm/lmcache to share GPU memory across containers. Won't that break the purpose of the smoke testing?

@github-actions

Copy link
Copy Markdown

✅ Hardware CI passed for f7d9244 (dist-build + smoke-test + tiny-test).

@sbates130272

Copy link
Copy Markdown
Collaborator Author

IIRC --pid host was required for vllm/lmcache to share GPU memory across containers. Won't that break the purpose of the smoke testing?

It seems that --pid <container-name> works and is more isolated than using --ipc host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants