Question. PHerc0841 (w00, ag896, ag405) is this repository's held-out test, but since 9 October it has said the scroll is "possibly not held out from Hecate" (log). Settle it from public sources, and for PHerc0841, PHerc0139 w045 and w043, and PHerc1447 w058 and w060 say which released reader trained on each. Nothing was rerun on a GPU, and nobody was asked.
Answer. Hecate is not a held-out reader on PHerc0841 (Interpretation, from the Sourced and Community lines below), but the exposure is not where the 9 October log first looked. Hecate's own training stages, as published, would never draw a PHerc0841 training patch: every supervised pixel of the three PHerc0841 label sets lies inside the dataset's validation mask, so the published split rule gives each segment zero training patches (Sourced fact for the code and masks; our computation; that the released run used this rule's split is Interpretation, item 1 below). The exposure comes through Hecate's starting weights (ink_canonical_2um, whose committed trainer lists all three segments as training fragments). Two weaker channels may add to it: checkpoint selection on a validation score that would average PHerc0841 in, and its Paris 4 3D teacher, whose first stage trained on eight scrolls' labels including PHerc0841 but which only made targets for Paris 4, so it is indirect at most. Team members also said on the community Discord in September 2026 that Hecate's training used almost all of the available ink labels and that the released label sets had been at least partly used to train the published models (Community reports, paraphrased). So Hecate's PHerc0841 numbers, and the team's new_canon_autoresearch_recipe maps there, stay possibly in-sample; every other released reader's PHerc0841 status is unchanged (held out).
Further facts for the other segments. w043, which kit/autobatch.py's comment calls "held-out" when it picks Hecate 2.4 um for fine scans, is in the training lists of ink_9um, d9v2 and Reader v2, and very likely in Hecate's: it has no validation mask in Hecate's dataset, and the published rule puts 74.5 % of its supervised pixels under training patches (68.2 % to 93.7 % over seeds 0 to 39). It is not in the canonical model's committed fragment list. w045 is in the canonical list, and the published Hecate rule puts 86.8 % of its supervised pixels under training patches. PHerc1447 is in neither Hecate's dataset nor the canonical list.
Labels follow ../NOVELTY.md. This log supersedes the held-out wording of the 2026-10-09 log, which stays as a record.
"Trained" means the reader's own published training list, config or split names the segment as training data; "inherited" means its starting weights did. Sources S1 to S14 are in the next section; all read 16 October 2026.
| Segment | ink_9um |
v8in (rev d89166b) | v8in-1447 | d9v2 | Reader v2 | ink_canonical_2um (team new_canon maps) |
Hecate, own stages | Hecate as released |
|---|---|---|---|---|---|---|---|---|
| PHerc0841 w00, ag896, ag405 | No (S7) | No (S8) | No (S8, S9) | No: all three in test.txt (S10) |
No (S11) | Listed as training (S3); release from that list: Interpretation | Validation only: 0 training patches (S4, S5, M1) | Not held out: base weights (S1, S3), team statements (S12, S13); weak: checkpoint choice (item 4), 3D teacher stage 1 (S6, indirect); Interpretation |
PHerc0139 w045 (dataset names w029, w029_202601261946) |
Trained, as pherc0139-w029 (S7) | No (S8) | No | Own: no (test.txt); inherited from ink_9um (S10) |
Trained twice (native and pherc0139-w029) and inherited (S11) | Listed as training, w029_202601261946 (S3) |
Trained: 86.8 % of supervised pixels under training patches, 13.2 % validation (M1) | Trained (own stages and base) |
| PHerc0139 w043 | Trained, pherc0139-w043 (S7) | No (S8) | No | Trained (train.txt) and inherited (S10) |
Trained twice (pherc0139-w043 and native) and inherited (S11) | Not in the committed list (S3) | Very likely trained: no validation mask; generated holdout of 2 blocks (15.3 % of supervision at the default seed); 74.5 % under training patches (M1) | Very likely trained (own stages); the base list does not name it |
| PHerc1447 w058, w060 | No (S7) | No (S8) | Trained (S9) | No (S10) | No; its w058 is PHerc0139 w058 (S11) |
Not in the committed list (S3) | Not in the dataset: no 1447 folder (S5) | No public source names it (Interpretation) |
Label per cell: entries from a reader's own card, config or committed list are Sourced fact for what the list says; that a community author's list is the list actually trained on (d9v2, Reader v2) is Community report; "release from that list" and the last column are Interpretation; M1 percentages are our computation on the published masks.
- Its own stages. The Hecate trainer the team published (villa branch
distilled_3d_ink, 5ff00ca0, S4) builds its split from the ink dataset the Hecate card cites (S1, S5). For each segment it uses the validation mask as the holdout when it has any pixel (multiteacher.py:186-195), keeps every 256 px training patch at least 256 px clear of that holdout (:111-134), and recordsvalidation_onlywhen no training patch is left (:214). Training draws come only from segments with training patches (:285-293). For PHerc0841 the threevalidation_mask_v2files contain every supervised pixel (M1), so all three are validation-only. Both trainers in that branch (the multi-teacher one with human 2D supervision, and the resolution distillation that produced the 9.6 um model from the 2.4 um one) read the same manifest. Limit: the trainer's guide says to use "the original audited manifest" and that a new one "creates a new split, not a reproduction of an archived run" (vesuvius/docs/hecate.md:19); that manifest is not public, so that the released run used this split is Interpretation. Two community checks of the open-data copies of these masks found the same 100 % (S13). - Its starting weights. The card: "We started from the canonical 2 µm ink detector" and, for the 2.4 um model, "We preserved as much of the canonical model's architecture and pretrained weights as possible" (S1). The canonical checkpoint's committed trainer lists w00, ag405 and ag896 in
TRAIN_FRAGMENT_IDS(lines 200-207), reads them from2um_dataset/841_2um(lines 292-293), holds out onlyl5_0_crop(line 794) and reads no validation mask (tile loop, lines 384-418) (S3). One caveat: that loop keeps a tile only where the segment's own{id}_maskfile is nonzero over the whole tile (line 413, loaded bytrain_resnet3d_lib/data_ops.read_image_mask, lines 178-185, in the same commit), and those files in the team's2um_datasetare not public, so whether they cut anything out of the PHerc0841 segments is unknown. Its checkpoint name, epoch 13 and 8,089 steps per epoch (113,246 total) match the released file (S2, S3). Whether the 9.6 um student started from canonical weights is not stated (the guide says student weights are random unless a checkpoint is given,hecate.md:36); its targets came from the 2.4 um model (S1), so it inherits that model's exposure either way (Interpretation). - Its teachers (weak, indirect). The Paris 4 3D teacher's card says it was trained in stages, the first a "teacher on 8-scroll surface-conditioned ink labels (PHerc. Paris 4, 0139, 0500P2, 0814, 0841, 1667, MAN5, 9B)"; its final config trains on Paris 4 only (S6). Hecate used it for Paris 4 3D targets, and the trainer gives Paris 4 segments a teacher index of their own (
teacher_id0, every other scroll 1;data/multiteacher.py:208), so Hecate never saw PHerc0841 CT through this teacher. At most, weights shaped in part by PHerc0841 labels shaped Hecate's Paris 4 targets: an indirect, weak channel (Interpretation). Whether that first stage held PHerc0841's validation mask out is not stated. - Checkpoint choice (weak). The multi-teacher trainer keeps
best.pthbyval/macro/pr_auc(training/multiteacher.py:543, 613), a mean over scrolls' validation patches, which would include PHerc0841's. The card says only "We then froze selected student snapshots" (S1). Selecting on PHerc0841 validation is a weak exposure, if it happened (Interpretation). - What the team said. Asked on 25 September whether PHerc0841 was held out from
hecate_9.6um, a team member answered that Hecate's training used almost all of the available ink labels, and advised making new segmentations to test on material a model has not seen (S12). On 28 September, in a community thread about the PHerc0841 validation masks, a team member advised mostly ignoring the validation set and said the label sets were published as a first batch that had been at least partly used to train the published models (S13). Community reports, paraphrased (team members on the community Discord, not a training record).
| Id | What | URL, revision | Dated | Label |
|---|---|---|---|---|
| S1 | Hecate card: base_model: scrollprize/ink_canonical_2um; starts from the canonical detector; "Strong 2D supervision from the existing ink annotations"; dataset link |
scrollprize/hecate, rev 9cb86e5; repo holds no manifest | created and last modified 2026-09-15 | Sourced fact |
| S2 | ink_canonical_2um: recipe new_canon_autoresearch_recipe, epoch 13, step 113,246, file renamed from ...i3depoch=13.ckpt; config.json total_steps 8089 |
scrollprize/ink_canonical_2um, rev 075855b | last modified 2026-07-07 | Sourced fact |
| S3 | Canonical trainer ink-detection/train_resnet3d_3d_decoder.py: TRAIN_FRAGMENT_IDS (200-207) has w00, ag405, ag896, w029_202601261946, 46527_2um_try2 and 19 others, no w043 and no PHerc1447 segment; paths 264-325; tile loop 384-418, per-segment mask test 413; validation 794; model 829; checkpoint name 845 |
villa 94602c91 (Youssef Nader, #820) | authored 2026-04-22 | Sourced fact |
| S4 | Hecate trainer: vesuvius/src/vesuvius/ink_detection/data/multiteacher.py (scroll list line 28; split 47-53, 98-134, 166-228; teacher index 208; draws 285-293), training/multiteacher.py (543, 613), vesuvius/docs/hecate.md (5, 19, 36, 48) |
villa 5ff00ca0 on branch distilled_3d_ink (Giorgio Angelotti), not merged into villa main (ccb43181, 2026-10-15) |
authored 2026-09-17, committed 2026-09-19 | Sourced fact |
| S5 | The ink dataset: folders ink/{0009b,0139,0500p2,1667,814,841,man5,phercparis4} plus unused, the trainer's eight scroll ids; no 1447 folder. PHerc0841's three folders carry validation_mask_v2; 0139 w043, w035, w039, w040, w041 and w044 carry no validation mask |
scrollprize/datasets, ink, listing via the bucket API |
mask files uploaded 2026-07-07 | Sourced fact |
| S6 | Paris 4 3D teacher: stage 1 "teacher on 8-scroll surface-conditioned ink labels (PHerc. Paris 4, 0139, 0500P2, 0814, 0841, 1667, MAN5, 9B)"; final config.json datasets Paris 4 only |
scrollprize/ink_3d_dino_guided, rev 73a7952 | last modified 2026-08-05 | Sourced fact |
| S7 | ink_9um label dataset README: 24 aligned segments (9 PHerc0139 incl. pherc0139-w029 = public w045 and pherc0139-w043 = public w043; 6 PHerc1667; 8 Paris 4; PHerc0814 46527) and 5 native PHerc0139 segments; no PHerc0841, PHerc0009B or PHerc1447 |
scrollprize/datasets, ink_9um/README.md |
file mtime 2026-08-14 | Sourced fact |
| S8 | v8in training/configs/v8in.json and patch pack segments.json: 16 training segments (Scroll 1 x2, Scroll 5 x6, PHerc1667 1, PHerc0139 w033, w035, w041, w044, PHerc0814 46527, PHerc0500P2 front, Frag1); validation 0009B_ag132115, Frag3, Frag4 |
ink-8um-v8in at d89166b; patch pack at d67b1c1 | 2026-09-28 | Sourced fact |
| S9 | v8in-1447 training/configs/loo_w062.json: v8in fine-tuned on PHerc1447 w058 and w060 |
ink-8um-v8in-pherc1447-loo-w062 at 2bf9f42 | 2026-09-28 | Sourced fact |
| S10 | d9v2 reader/train.txt (124 segments: PHerc0009B 17, 0139 36 incl. w039, w043, w044, 0343P 8, 0500P2 45, 0814 18) and reader/test.txt (8 segments, among them w045 and the three PHerc0841) |
TAUIL-Abd-Elilah/pherc0826-first-letters-search at c156944 | read 2026-10-16 | Community report |
| S11 | Reader v2 train_config.json: 13 dataset blocks; PHerc0139 includes pherc0139-w029 and pherc0139-w043 and natively 20260126000000-w045_2026012619, 20260112000000-w043_2026011217; no PHerc0841 or PHerc1447 |
domenicor046/reader-v2 at 72633d8 | 2026-09-27 | Community report |
| S12 | A team member's reply on the community Discord to a question about hecate_9.6um on PHerc0841, PHerc0009B and PHerc0500P2 |
Vesuvius Challenge Discord, #general (not linked) | 2026-09-25 | Community report |
| S13 | A community thread on what the validation masks mean for PHerc0841 and PHerc0500P2: two community members measured them (validation covers about 70 % of the canvas and all of supervision on the three PHerc0841 sets and on PHerc0500P2; 100 % on all three PHerc0841 sets), and a team member replied twice | Vesuvius Challenge Discord, #ink-detection (not linked) | 2026-09-23 to 2026-09-28 | Community report |
| S14 | The open-data bucket holds the PHerc0841 human labels this repository scores: PHerc0841/segments/20260220213127-w00/ink-labels/2.403um-volume-20260319124803/20260918/{inklabels,supervision,validation}.zarr |
bucket listing | read 2026-10-16 | Sourced fact |
scripts/heldout/hecate_split.py ports the split functions of S4 (numpy only; tests in tests/test_hecate_split.py, including a brute-force check of the max filter, hand-counted cases that pin the 5 % threshold, the two scan strides and the validation rule, and a check that the rule can produce training patches). It read the masks below from the ink dataset (kept outside git), CPU only, a few minutes in all. Our computation on published inputs (Sourced fact for inputs and code; the numbers are a measurement, not a model output). The trainer picks the highest label version that has both ink labels and a supervision mask, here _v2 except w043 (unversioned).
| Segment (trainer identity) | Supervised px | In validation | Training patches | Validation patches | Supervised px under training patches | Supervised ink px under training patches |
|---|---|---|---|---|---|---|
| 841/w00 | 24,531,374 | 100 % | 0 | 3,871 | 0 % | 0 of 5,445,369 |
| 841/auto_grown_20260220144552896 (ag896) | 14,896,863 | 100 % | 0 | 4,745 | 0 % | 0 of 4,816,246 |
| 841/auto_grown_20260220174252405 (ag405) | 19,402,619 | 100 % | 0 | 3,662 | 0 % | 0 of 4,397,914 |
| 0139/w029_202601261946 (public w045) | 8,763,974 | 13.2 % | 627 | 26 | 86.8 % | 1,901,994 of 2,234,011 |
| 0139/w043_2026011217 (no validation mask) | 6,882,172 | 15.3 % in 2 generated blocks | 432 | 21 | 74.5 % | 2,259,608 of 3,045,398 |
| 814/814_46527_2um_try2 (control) | 9,433,800 | 27.3 % | 415 | 57 | 62.6 % | 1,933,863 of 2,958,069 |
The rows w045 and PHerc0814 are the check that the rule can fail: on segments with a small validation block it does give training patches. w043's generated holdout depends on the seed (27, the default) and on the segment identity; under that seed its two 1024 px blocks are rows 14,336 to 15,360, columns 24,576 to 25,600 and rows 16,384 to 17,408, columns 8,192 to 9,216 of its 23,760 x 31,680 canvas. Over seeds 0 to 39 the holdout is always 2 of w043's 23 blocks, but those two hold anywhere from 0.9 % to 17.8 % of its supervised pixels, and the share under training patches runs from 68.2 % to 93.7 %, so "very likely trained" holds whatever the archived seed was (our computation, 16 October).
Reproduce, with each folder holding the *_supervision_mask*.tif, *_validation_mask*.tif and *_inklabels*.tif files of https://huggingface.co/buckets/scrollprize/datasets/resolve/ink/<scroll>/<folder>/<file>:
python3 scripts/heldout/hecate_split.py DIR841 w00:841/w00 auto_grown_20260220144552896:841/auto_grown_20260220144552896 auto_grown_20260220174252405:841/auto_grown_20260220174252405
python3 scripts/heldout/hecate_split.py DIRW043 w043_2026011217:0139/w043_2026011217 --suffix "" # add --seed N for another seedMask SHA-256 (first 12 hex digits; supervision then validation): w00 473fa4f95847 and fe3ae9268658; ag896 226a3f640ba9 and 711e50d921d0; ag405 87b6d1164651 and a1105426782a; w029 1ccaab08702a and a0d054be393b; PHerc0814 8216d94b7b18 and 916abe3a772e; w043 supervision c3880f79d194.
- README "Novel findings". Findings 3, 4 and 6 use
ink_9um, its soup and d9v2 on PHerc0841: unchanged. Findings 7 to 9 use the team'snew_canon_autoresearch_recipemaps of PHerc0841 and already say they are probably in-sample; this log adds a second line of evidence (S13) and no change of wording is needed. Finding 1 (w045) is unchanged. kit/autobatch.py, lines 47 to 51. Its comment says the reader choice was "measured on held-out PHerc0139 w043" and its printed reason reads "fine scan: Hecate 2.4 um (best on held-out w043)". w043 is inink_9um's list and very likely in Hecate's own training (74.5 % of supervision under training patches by the published rule), but not in the canonical model's committed list, so that comparison favours Hecate over canonical 2um rather than measuring generalization. The reader choice is Chase's (16 Oct) and may well stand; the words "held-out" do not.kit gateranksink_9um, d9v2, Reader v2 and v8in on the PHerc0841 crops, never Hecate: unaffected.
Brought in line with this log (wording only, no number changed): docs/results.json (held_out.PHerc0841 and held_out.w045 rewritten, held_out.w043 and held_out.PHerc1447 added, notes on the two community Hecate rows), the glossary rows, rule 6 of docs/AI-AGENTS.md, docs/STANDARDS.md, docs/tricks.md, and the Hecate caveats in the docstrings of kit/auc.py and kit/ensemble.py and in kit plan's printed First Letters plan, which said "possibly not" or "may not be" held out. Dated logs keep their wording.
- Which fragment list, and which per-segment
_maskfiles, the run behind the releasedink_canonical_2umused, and the archived Hecate manifest. Only the authors can say; asking them is an outward step and needs Chase's go-ahead. - How the 9.6 um student was initialized, and whether the Paris 4 teacher's first stage held PHerc0841's validation mask out.
- The w045 crop's position relative to Hecate's training patches (it sits on another grid; not mapped).
- Indirect exposure of d9v2 and Reader v2 through the team's dense targets (unchanged from the 2026-10-09 log; Untested idea).
- Untested idea: score Hecate and canonical 2um on w043 separately on Hecate's two generated holdout blocks (above) and on the pixels under its training patches. A large Hecate drop on the blocks would show its w043 lead is in-sample. It needs a preregistration first, and the blocks move with the seed.