Skip to content

Queue output downloads before waiting on the CPU - #199

Closed
kvark wants to merge 2 commits into
perf/scalar-matmul-autotune-2026-09-20from
perf/queued-output-readback-2026-09-20
Closed

kvark wants to merge 2 commits into
perf/scalar-matmul-autotune-2026-09-20from
perf/queued-output-readback-2026-09-20

Conversation

@kvark

@kvark kvark commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Stacked on #198. Adds Session::wait_read_output: when the existing readback probe selected staging, enqueue the copy before the CPU waits for the graph. Mapped reads and initial qualification still wait first. The measured readback choice and bit checks are unchanged; every command buffer is freshly recorded.

The normal GGUF helper uses the combined operation. CPU stages are now decode_record_finish_ms, because finish includes both waiting and host copying; these are not kernel/barrier measurements. The existing broad regression issues fresh GPU work for each read size, including empty reads and a partial final 16 MiB staging chunk. No new test executable.

A source-only controlled experiment compares both call sequences within exactly the same tuned sessions. Three fresh processes per GPU, order reversed in the middle process. Decode improves in every pair: 1.17-1.65% on RTX 5070 and 2.75-4.01% on B570. All twelve saved logit sets match bit for bit between arms and retain all 33 CPU-reference choices.

Fresh default-invocation comparisons, three processes per engine/GPU with rotated engine order (milliseconds; median of process medians):

GPU / phase Meganeura llama.cpp
RTX 5070 prefill 7.210 7.126
RTX 5070 decode/token 1.389 1.361
Arc B570 prefill 14.048 15.650
Arc B570 decode/token 3.666 3.431

This is close, not stable parity: decode process medians span 1.346-1.503 ms / 3.656-3.957 ms, including different attention-split and submission selections. All twelve fresh logit sets are finite and retain all CPU-reference choices. Neither kernel search budget is exhausted. Preparation remains 9.61/31.45 s versus llama.cpp's 0.25/0.51 s with existing driver caches. Full methodology and limits: bench/queued-readback.md.

Separate Nsight Systems Vulkan/OS-runtime captures confirm the ordering change: all final 32 serial decode steps contain a host poll between graph/readback submissions, versus none with queued readback. These tune-disabled, single-submission captures are ordering diagnostics, not latency or barrier-cost evidence.

All 449 active library tests and all-target/all-feature Clippy pass. Both existing device-local/readback regressions pass on both physical GPUs. Implementation CI passes all six jobs, including host coverage. Known Naga Workgroup ArrayStride diagnostics remain. The final follow-up only adds the result report and updates helper documentation; its CI is separate.

No raw results, binary artifacts, Inferena changes, paper changes or command-buffer replay.

@kvark
kvark marked this pull request as ready for review September 20, 2026 02:28

kvark commented Sep 20, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #200 as requested: one simplified PR against current main with focused commits. The existing branch and measured revisions remain available for reproducibility. Closing this PR so review can continue in one place.

@kvark kvark closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant