[phippsrtyler]

Depends on #2189 (--gpu-normalize); review/merge after it.

In one sentence: A short doc shows how to run 2–4 infer.py processes on one GPU, which inks the same windings 1.2–1.4× faster.

One real example: On a Radeon AI PRO R9700, six Reader v3 windings take 74.7 s with one process and 51.9 s with four processes (1.44×), and the output is byte-identical.

Before: One process leaves the GPU idle between batches, while CPU workers read and normalize tiles.

After this PR: ink-detection/koine_machines/inference/CONCURRENCY.md gives measured times per card and a copy-paste loop that splits the segments into one --folder per process. No code changes.

Proof:

GPU	1 process	2 processes	4 processes	Output vs 1 process
RTX 3080 10 GB	40.8 s	33.1 s (1.23×)	34.4 s (1.19×)	byte-identical
Radeon AI PRO R9700 32 GB	74.7 s	64.4 s (1.16×)	51.9 s (1.44×)	byte-identical
Instinct MI50 32 GB	102.2 s	81.0 s (1.26×)	77.7 s (1.32×)	byte-identical

Six Reader v3 windings (PHerc0800, 0125, 0211 and 0268 surface volumes), --stride 192 --amp-dtype fp16 --no-compile, MIOpen off on the AMD cards, windings split round-robin over the processes, time until every process is done. Each value is the median of 3 interleaved rounds after a discarded warm-up, with nothing else running on the machine.

--gpu-normalize gave no gain in these runs: within 3% on the R9700 and MI50, and 1–4% slower on the RTX 3080. --batch-size 8 with --amp-dtype fp16 gave no gain on any card.

R9700 rows come from one fair re-take (ABBA x3, byte-identical outputs), which agrees with an earlier run. The card is on loan to another project, so no further R9700 re-takes are planned for this PR.

Why / where this is useful:

One infer.py process leaves the GPU idle while tiles are being read and normalized. Running 2–4 processes per card inks the same windings 1.2–1.4× faster with no code changes, which makes inking a whole scroll on one GPU much faster.

• I personally verified that the example and proof above were produced by this PR on the stated data.

Details

Parity check for one segment. SEG is the segment to check; OTHER1–OTHER3 are three other segments that keep the GPU shared during the concurrent run.

mkdir -p solo part0 part1 part2 part3
ln -s /path/to/SEG solo/
ln -s /path/to/SEG part0/
ln -s /path/to/OTHER1 part1/
ln -s /path/to/OTHER2 part2/
ln -s /path/to/OTHER3 part3/
python infer.py --folder solo --checkpoint-path model.ckpt --gpu-normalize --output-prefix solo
for i in 0 1 2 3; do
  python infer.py --folder part$i --checkpoint-path model.ckpt --gpu-normalize --output-prefix multi &
done
wait
cmp /path/to/SEG/preds/solo_*.tif /path/to/SEG/preds/multi_*.tif

[vercel]

@phippsrtyler is attempting to deploy a commit to the scroll Team on Vercel.

A member of the Team first needs to authorize it.

[bruniss]

while currently true the gpu not being fed is a problem of the ink pipeline which should be addressed directly rather than running concurrent preds. this also will likely become out of date as the pipeline becomes more optimized.

[phippsrtyler]

Makes sense, thanks. I've been working on the feeding side directly too: #2226 reads remote full-3D ink patches through vc_reader and is about 2x faster on 45 PHerc Paris 4 patches (36.6 → 18.1 s on a 3080, similar on an R9700 and MI50), with all 45 patches identical to main. It's ready for review if you have time to take a look.