Skip to content

Commit 58e3764

Browse files
committed
seo: harden backfill-openapi-lastmod.js arg parsing; clarify digest verification scope
Addresses two low-confidence findings from pre-merge review on d48c756: - --cache-dir/--seed-ledger now reject a missing value instead of silently setting undefined (which surfaced later as an opaque path.join TypeError, or worse, a silent fallback to git history for --seed-ledger). - Clarified in code comments and the log line that only the CDX API's reported digest is checked against the recorded value, not the fetched body -- Wayback's digest is computed over the raw compressed bytes as captured, which fetch() transparently decompresses, so a body-level re-hash was attempted and confirmed to always mismatch on legitimate content. Not worth a broken check for a one-time, human-reviewed script whose only output is a committed diff.
1 parent d48c756 commit 58e3764

1 file changed

Lines changed: 21 additions & 3 deletions

File tree

scripts/backfill-openapi-lastmod.js

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -127,8 +127,18 @@ async function fetchSnapshot(obs, cacheDir) {
127127
"Refusing to trust unverified snapshot content -- investigate before updating OBSERVATIONS.",
128128
);
129129
}
130-
console.log(` ${obs.date} (${obs.timestamp}): digest verified, fetching content...`);
130+
console.log(` ${obs.date} (${obs.timestamp}): CDX metadata verified, fetching content...`);
131131
const contentUrl = `https://web.archive.org/web/${obs.timestamp}id_/https://${SPEC_HOST_PATH}`;
132+
// Note on what is and isn't verified here: the CDX digest checked above
133+
// is the base32 SHA-1 of the *raw captured bytes* as transmitted over the
134+
// wire (confirmed by comparing a fetched snapshot's decoded length against
135+
// the CDX API's own `length` column, which is smaller -- the archive
136+
// stores the original gzip-compressed response). `fetch()` transparently
137+
// decompresses, so the JSON text below cannot be rehashed against that
138+
// digest; what's verified is that the archive's own metadata for this
139+
// exact timestamp still matches what was recorded on 2026-08-25, not the
140+
// bytes of this particular fetch. That is an acceptable bar for a
141+
// one-time, human-reviewed script whose only output is a committed diff.
132142
const res = await fetchWithBackoff(contentUrl);
133143
const text = await res.text();
134144
const spec = JSON.parse(text);
@@ -276,13 +286,21 @@ function updateMeta(meta, { totalNewlyDated, totalDateless }) {
276286
return next;
277287
}
278288

289+
function requireValue(argv, i, flag) {
290+
const v = argv[i + 1];
291+
if (v === undefined || v.startsWith("--")) {
292+
throw new Error(`${flag} requires a value`);
293+
}
294+
return v;
295+
}
296+
279297
function parseArgs(argv) {
280298
const args = { cacheDir: path.join(ROOT, ".openapi-wayback-cache"), dryRun: false, seedLedger: null };
281299
for (let i = 0; i < argv.length; i++) {
282300
const a = argv[i];
283301
if (a === "--dry-run") args.dryRun = true;
284-
else if (a === "--cache-dir") args.cacheDir = argv[++i];
285-
else if (a === "--seed-ledger") args.seedLedger = argv[++i];
302+
else if (a === "--cache-dir") args.cacheDir = requireValue(argv, i++, "--cache-dir");
303+
else if (a === "--seed-ledger") args.seedLedger = requireValue(argv, i++, "--seed-ledger");
286304
else throw new Error(`unknown argument: ${a}`);
287305
}
288306
return args;

0 commit comments

Comments
 (0)