Skip to content

AGENTS.md

Latest commit

 

History

History
677 lines (544 loc) · 46 KB

AGENTS.md

File metadata and controls

677 lines (544 loc) · 46 KB
Raw

Agent Instructions: Dataset Processing

You work in data-workflows, which uses cng-datasets to convert geospatial data into cloud-native formats on Kubernetes. This file carries the boundaries you must never cross and the shape of the pipeline; the procedural detail lives in skills that load on demand.

Skills — load the one that matches what you are doing

These are .claude/skills/<name>/SKILL.md in this repo. They work in Claude Code and opencode alike (both read .claude/skills/), and they are plain markdown, so any agent can read the file directly if it has no skill support.

skill load it when
stac-authoring writing or editing any stac-collection.json or dataset README (Steps 5–6)
raster-hexing ingesting or re-hexing a GeoTIFF/COG — resolution, reducer, mosaicking
hex-tuning a hex job OOMs, or choosing native/parent H3 resolutions and chunk sizes
job-troubleshooting a job fails, hangs, or a published parquet will not read
dataset-recipes starting an ingest that resembles a worked example
armada-pipeline armadactl will not authenticate, a k8s job hits the 200-completion cap, or you are choosing between the k8s and Armada pathways
pod-preemption indexed-job pods die and retry for no apparent reason, you are working out which indexes actually need re-running, you are choosing a priority class or sizing requests (NRP usage policy), or before submitting any job whose pods run over an hour

Everything below is always in force, skill or no skill.

⛔ HARD BOUNDARY: Scoping decisions live in the GitHub issue, NEVER in memory

Every decision that defines what a dataset task delivers — spatial extent (full upstream coverage vs. a regional clip), H3 resolution, reducer, source product/version, target bucket and naming — MUST be written into the GitHub issue before you act on it. Sessions are disposable: the laptop dies, agent memory is wiped, a different agent picks up the task. The issue is the single source of truth that survives all of that. Treat yourself like any professional developer — nobody is expected to carry scope "in their head," and you must not rely on agent memory (~/.claude/.../memory) for scope. Memory is for how you work, never what a task delivers.

  • Before building: if the issue leaves any of the scope fields above unstated, it is underspecified. The request form asks only for a link, so a fresh request always is: scoping it is your job, not the requester's. Stop, propose the scope, get agreement, and edit the issue to record it (gh issue edit / a scoping comment) before running cluster jobs. A vague label like "the wyoming group" is not a scope.
  • When scope changes mid-task (e.g. "extend to the full upstream extent, not the regional clip"): update the issue body in the same turn you learn it — don't just remember it.
  • When the user tells you a fact only they hold — a dataset's origin, why a licence is what it is, who contributed something, what an upstream field means — write it into the issue in the same turn, before using it. These facts are unrecoverable: nobody can re-derive them from the data, so a fact left in session context is destroyed when the session ends, and the next agent asks the same question again. If you are about to ask the user something, first search the repo, git history, and the issues for the answer: being asked twice is the symptom that the first answer was never recorded (#579).
  • Never infer scope from the bucket name, an existing clipped COG, or a prior build's resolution. Those are artifacts of how an earlier (possibly wrong) pass happened to run, not statements of intent.
  • A reviewer (human or agent) must be able to read the issue alone and know exactly what to build. If you found yourself reconstructing scope from code, S3 layout, or memory, that is the signal the issue needs updating.
  • Scope is WHAT, never HOW. The issue states the template's fields and nothing more — not which scripts to write, not cng-datasets vs a custom job, not what to model the pipeline on, not whether to wait on another repo's issue. Implementation is decided by whoever builds it, against the tooling as it stands that day. An agent-authored directive in an issue body is indistinguishable from a considered decision and outlives the session that guessed it: #669 carried a fabricated "do not wait on it and do not implement it in the tool repo" that sent a later session off to hand-roll a Zarr reader. File new dataset issues with gh issue create --template dataset-import.yml. The form asks only for a link (description and licence are optional) and has no title pattern, so requesters outside the lab can use it; the scope fields above are added when the request is scoped, by editing the issue.
  • Write the scope, not an essay. The issue must carry the scope fields — that is the part that survives the session, and it stays as long as it needs to be. Everything around it is noise: do not restate the plan or narrate progress. The same goes for PR reviews and bug reports — lead with the evidence (the measurement, the repro, the query someone can re-run) and cut the prose around it. Issue traffic is read by every agent that touches the task afterwards, so verbosity is a cost they all pay.
  • The reviewer and CI set the bar — never the issue. Do not write "acceptance criteria" into an issue, and do not review a PR against them or block it for lacking them; they are not a scope field. An issue author asking for work does not get to define what passes review, any more than students write their own exam. Issue authors (often agents outside this repo) also lack the guidance in this file and its skills, so judge a PR on its merits against that guidance and on CI.
  • Close a duplicate as a duplicate. When an issue repeats work another issue already covers, close it with the reason duplicate, never not planned: "not planned" tells the requester we declined their request. gh issue close offers only completed and not planned, so set the reason through the API (gh api -X PATCH repos/boettiger-lab/data-workflows/issues/<n> -f state=closed -f state_reason=duplicate), after a comment that names the issue covering the work (#815, #816).
  • External requesters are our guests. Collaborators and users outside the lab who file requests (through the request form or directly on GitHub) get white-glove treatment: respectful, professional, and written for them, not for us. Explain what happens to their request in plain words; leave out internal jargon, tooling names and process detail they do not need. When their issue needs reshaping, do the reshaping yourself rather than sending it back to them.

Default scope = faithful, comprehensive representation of upstream

When scoping a dataset, default to representing the upstream product comprehensively: every format it ships (vector and raster), every year/epoch/layer, and its companion tables (e.g. national statistics). Narrow only for a concrete reason — a cost that is truly prohibitive, or a layer that is redundant — and put that trade-off to the user.

Format or epoch choices the user did not make are open questions under "Scope is WHAT, never HOW" above: assess them against the upstream data and settle them with the user (GMW #443: the issue said "prefer the GeoTIFF path" and "latest single year"; upstream ships 41 annual vector and raster layers).

⛔ HARD BOUNDARY 0: Big-data compute runs on the cluster, NOT your laptop

For ANY query/scan/aggregation over S3 parquet — catalog data and large intermediate/build files (e.g. a 24 GB consolidated GeoParquet) — use the mcp__duckdb-geo__query MCP server. It runs on generously-provisioned cluster metal with the internal NRP S3 endpoint and a 100 Gb/s network, and DuckDB streams (larger-than-memory spills to disk) — so it does not hit the RAM limits or the slow public endpoint (~12 MB/s) that bottleneck the laptop.

  • ❌ Do NOT shell out to local duckdb/python -c "import duckdb"/uv run --with duckdb.
  • ❌ Do NOT spin up k8s jobs running pyarrow/geopandas/duckdb to scan big parquet, and never read big parquet over https://s3-west.nrp-nautilus.io/... (public endpoint) — use s3:// (internal) inside jobs and the MCP for queries.
  • ✅ Verify counts/schema, scan for pathological geometries/bbox/validity, distinct counts, ST_/h3_ → mcp__duckdb-geo__query.
  • The tool is deferred: ToolSearch select:mcp__duckdb-geo__query first. If ToolSearch returns no match the MCP server is disconnected — ask the user to reconnect it; do NOT fall back to the slow RAM-bound path.
  • Heavy writes/transforms the MCP can't do still run as cluster jobs, but over the internal endpoint (rook-ceph-rgw-nautiluss3.rook, AWS_HTTPS=false), never the laptop.

⛔ Serialization standard: keep parquet row groups small

A single parquet column chunk in the ~2.8–2.88 GB compressed range makes DuckDB's httpfs reader abort with SQL Error: stoi — a reader bug on the https:// path, not corrupt data (fully diagnosed in datasets #106).

Prevention is the fix: use --row-group-size 2000 as the default for geometry-heavy datasets, so no geometry column chunk approaches ~1 GB compressed. The default 100000 can pack a single chunk past the cliff on national-scale multipolygons.

Full diagnosis, what does not matter (CRS tag, writer, one bad geometry), and how to rescue an already-published file: skill job-troubleshooting.

⛔ HARD BOUNDARY 1: NRP S3 is Canonical for STAC and Data

Canonical STAC/data live on NRP S3 (s3://public-<bucket>/, public URL https://s3-west.nrp-nautilus.io/<bucket>/...). This repo does NOT contain STAC JSON or README files. Never create catalog/*/stac/, never git add STAC JSON or README.

  • Read: curl https://s3-west.nrp-nautilus.io/<bucket>/stac-collection.json
  • Write: edit in /tmp/, then rclone copyto /tmp/... nrp:<bucket>/...

You publish to NRP. Downstream copies are NOT your job.

NRP is where your work ends. Backup and mirror tiers — source.coop, MinIO, and any other copy of this data anywhere — are owned by a different agent, and nothing about them is in scope for this repo. You do not add buckets to a mirror list, decide which prefixes are excluded from one, record a redistribution verdict in an inventory, or open a PR against the repo that holds those scripts. Put the data in the bucket and let the other agent do its job.

What is yours is the licence metadata on the collection: an accurate SPDX license and a {"rel": "license"} link (Step 7). The mirror tier reads those to make its own routing decisions. Getting the licence right is therefore the whole of your contribution to that question — the routing built on top of it is not yours to make.

Two things will tempt you across this line, and both have:

  • Old issue bodies. Plenty of open issues still say "add to REPOS", "EXCLUDES sub-path", "MinIO-only, exclude from source.coop", or name gen-source-sync.sh / gen-minio-sync.sh. They were written when this was the repo's job. They are stale, those two script names no longer exist, and #678 tracks purging them. Treat such an instruction as out of scope and say so in the issue rather than acting on it.
  • The sibling checkouts in ~. glen-ops and friends sit next to this repo and are readable, so going looking will always turn up the real machinery. Finding it is not authorisation to touch it — Hard Boundary 2 covers every other repo, not just cng-datasets.

If a task genuinely seems to require a mirror-side change, that is a signal to stop and hand it over, not to make the change.

Git workflow

Use a worktree for every dataset task. Sessions in this repo frequently open on a stale in-flight branch (e.g. last week's paused ingest); committing there silently mixes unrelated work and forces cherry-picks later. Before staging changes:

  1. git branch --show-current — if it isn't the branch you want, invoke superpowers:using-git-worktrees to spin up an isolated workspace off origin/main.
  2. Work in the worktree, commit, push, open the PR from there.
  3. Read-only investigation can stay on whatever branch is checked out — the gate is committing, not exploring.

⛔ HARD BOUNDARY 2: Do NOT Touch the cng-datasets Tool Repo

Work only in data-workflows. Do not edit/commit/push/PR to boettiger-lab/datasets or any other repo. If cng-datasets has a bug: file a GitHub issue on boettiger-lab/datasets with a minimal reproducible example (see below), tell the user, and wait. Prior hotfixes have caused production failures — the tool has tests and a deploy pipeline, and unreviewed hotfixes bypass them.

Bug reports must include a tested MRE

Every bug report MUST include code you have actually run that reproduces the error. Circumstantial evidence (e.g. "the S3 parquet has bad coords, so the tool is broken" — S3 could be stale) is not an MRE.

The MRE must:

  1. Isolate to the tool — verify the upstream input is correct first (e.g. ogrinfo on the raw source).
  2. Run the tool locally and capture bad output — actually execute cng-convert-to-parquet/cng-datasets. Don't infer from downstream artifacts.
  3. Show expected vs actual with concrete values.
  4. Be minimal — one feature, one file.

Template:

# 1. Input is correct:
ogrinfo /vsizip/source.zip -al -where "NAME='X'" | grep 'POLYGON\|Extent'
# 2. Run the tool:
cng-convert-to-parquet source.zip /tmp/output.parquet
# 3. Show the bug:
python3 -c "import duckdb; c=duckdb.connect(); c.execute('LOAD spatial;');
print(c.execute(\"SELECT ST_AsText(ST_Envelope(geom)) FROM read_parquet('/tmp/output.parquet') WHERE NAME='X'\").fetchdf())"
# → wrong: xmin=38.32 (lat) instead of -120.07 (lon)

⛔ HARD BOUNDARY 3: Jobs run in geo-workflows, not biodiversity

Status (2026-07-05): MIGRATION IN PROGRESS (glen-ops#21). geo-workflows is provisioned as the least-privilege job-submission namespace; biodiversity keeps the fleet apps + MCP + LLM proxy (and its 26 app/LLM/private secrets). Existing biodiversity job manifests are legacy artifacts — reshape them by regenerating with --namespace geo-workflows (below). Use geo-workflows for all new and re-run work; any residual biodiversity reference in an old manifest is legacy, not a target.

Why. geo-workflows holds ONLY NRP-canonical credentials (aws + rclone-config). A confused or compromised job there can damage only recoverable NRP-canonical data — it cannot reach the app / LLM / oauth / private-data secrets that live in biodiversity. That confinement is the whole point; never reintroduce other credentials into geo-workflows.

How to target it — the CLI already supports it, just pass the flag:

cng-datasets workflow        --namespace geo-workflows  ...   # all other args unchanged
cng-datasets raster-workflow --namespace geo-workflows  ...

It stamps the namespace on every generated manifest (and, for armada, the queue). Apply and monitor with -n geo-workflows:

kubectl apply -n geo-workflows -f catalog/<dataset>/k8s/<name>/workflow-rbac.yaml   # once
kubectl apply -n geo-workflows -f .../configmap.yaml -f .../workflow.yaml
kubectl -n geo-workflows get jobs | grep <name>

What's already provisioned there (manifest: glen-ops k8s/geo-workflows-provisioning.yaml):

  • Secrets aws (NRP S3) + rclone-config (nrp-only). Nothing else.
  • Orchestrator SAs cng-datasets-workflow / workflow-runner (create/watch/delete child Jobs).
  • Child job pods run as the default SA, hardened to mount no k8s token (they need only S3).
  • rechunk-scratch PVC (2Ti RWX) for >50Gi scratch (binds once Ceph is healthy).
  • No enforced ResourceQuota — but keep the good-practice targets anyway: ≤200 simultaneous pods and ~50Gi ephemeral per job (limits.ephemeral-storage; a default, not an enforced max — see below), especially on large-completion fan-out jobs. Oversubscribing is antisocial on shared nodes.

Credentials you have, and the only two you ever need:

  • Public buckets (almost everything). Build jobs read and write NRP only, with the aws + rclone-config secrets already in geo-workflows. Reading public data needs no credential at all. Stage raw under s3://<bucket>/raw/ (Step 1b). A workflow-namespace rclone-config holding only [nrp] is correct, not a missing-remote bug (internal endpoint http://rook-ceph-rgw-nautiluss3.rook, upload_concurrency=16, chunk_size=64Mi).
  • Private buckets (private-wyoming, private-tpl) live only on MinIO. Mount a scoped, single-bucket, on-demand EXPIRING mint for that one bucket (mc admin user svcacct add <parent> --policy <one-bucket> --expiry …; the wyoming-publish model) — never a standing broad key. This is the only MinIO access a build job ever has.

⛔ Never add any other remote or secret to a workflow namespace. If a job appears to need one, it is out of scope — stop and ask.

What You Produce

Vector datasets (GDB, Shapefile, GeoPackage, GeoParquet):

Format File Use
GeoParquet dataset.parquet DuckDB/Polars queries
PMTiles dataset.pmtiles Web map visualization
H3 Hex Parquet dataset/hex/h0={cell}/data_0.parquet Spatial joins/aggregation

Raster datasets (GeoTIFF, COG):

Format File Use
COG dataset-cog.tif Cloud-optimized raster (titiler etc.)
H3 Hex Parquet dataset/hex/h0={cell}/data_0.parquet Spatial aggregation

Rasters produce COG + hex only (no GeoParquet/PMTiles). Workflow: create WGS84 COG first, then hex from the NRP S3 COG (not the original source URL).

Geometry support: polygon, point, line. Check before hexing:

SELECT ST_GeometryType(geom), COUNT(*) FROM read_parquet('s3://...') GROUP BY 1
  • Points/MultiPoints: each point → one H3 cell. At coarse resolutions (6–8) many points collapse to the same cell. Warning emitted during hex. Document in STAC.
  • Lines/MultiLines: supported as of cng-datasets PR #69. Each line is buffered by the H3 circumradius at the target resolution before polyfill. Default resolution 8 (auto-detected). Document: "Line features were hexed to H3 resolution 8 by buffering each segment by the H3 cell circumradius before polyfill."

ALWAYS use the k8s workflow. Local env lacks tools and permissions.

Local env (YAML generation only)

cng-datasets does from osgeo import gdal at import time, so even the pure-templating workflow/raster-workflow subcommands won't load without the Python GDAL binding — you get ModuleNotFoundError: No module named 'osgeo'. libgdal is already installed on the standard image (gdal-config --version works); you only need the matching Python binding. Use the committed bootstrap, which pins the binding to the system libgdal so it works on any clone:

scripts/setup-venv.sh          # creates .venv with gdal==$(gdal-config --version) + cng-datasets
source .venv/bin/activate

Or by hand:

uv venv && source .venv/bin/activate
uv pip install "gdal==$(gdal-config --version)"   # match system libgdal (auto-detected via gdal-config)
uv pip install git+https://github.com/boettiger-lab/datasets.git

How to Process a Dataset

Step 1: Verify source URLs

Always verify before generating workflows. Do not assume naming patterns.

curl -I https://example.com/data.zip                         # single file
curl -s https://.../TIGER2024/TRACT/ | grep '.zip' | head    # directory listing
ogrinfo /vsicurl/<source-url>                                # multi-layer file

Common patterns: Census TIGER = per-state (tl_2024_{STATEFP}_tract.zip); protected areas = per-region or national; rasters may be tiled.

💡 To read a schema, a lookup table, or one layer's coverage out of a multi-GB remote zip, use GDAL range reads (/vsizip//vsicurl/…) instead of localizing the archive — a small job, seconds to minutes, no PVC. Worked examples: skill dataset-recipes.

Step 1b: Copy raw to s3://<bucket>/raw/ FIRST

External downloads are slow/rate-limited; restart from S3 if conversion fails. Subsequent jobs read s3://<bucket>/raw/<file> (or /vsicurl/https://s3-west.nrp-nautilus.io/<bucket>/raw/<file> for GDAL).

⛔ Staging the raw is only half of it — record WHEN you pulled it, in the STAC (Step 5). The access date is the field that makes a provenance chain resolvable to an actual edition, and it is unrecoverable after the fact: publishers overwrite in place (FWS republishes the NWI every May and October at the same URL, with no dated archive) and links rot (the CAL FIRE FRAP raster has no stable public download URL at all). When either happens, your staged copy plus its checksum ARE the provenance — there is nothing else left to point at. Capture, on the collection: the upstream landing page (stable, not a fragile direct-download link), the edition label if upstream publishes one, the access date, and the staged raw's path + size + checksum. Recompute the checksum from the object, never transcribe it. Worked examples: wetlands/nwi and cwhr/cwhr13 (#417).

A caveat that has bitten twice: never let a number that isn't an edition stand in for one. A temporal extent is content coverage, a product-line name (nwi-v2 = FWS post-2012 mapping) is not a release, and a derived file's mtime is the conversion, not the pull. Consumers read all three as edition stamps and published three claims that had to be retracted. If the edition is genuinely not determinable, say so — do not synthesise one.

⚠️ Brand-new bucket? Run the *-setup-bucket job BEFORE any stage-raw/download job. The bucket does not exist until cng-datasets storage setup-bucket creates it, and rclone ... --s3-no-check-bucket will happily scrape/build for many minutes and then fail the upload with NoSuchBucket (404) — wasting the whole download. For an existing bucket this ordering doesn't matter (the orchestrator runs setup-bucket first anyway); it bites only the first dataset in a new bucket, where stage-raw runs outside the orchestrator.

command: [bash, -c, |
  curl -L --retry 5 -o /tmp/data.zip "$SOURCE_URL"
  rclone copy /tmp/data.zip nrp:<bucket>/raw/
]

Step 1c: Preprocessing multi-file zipped datasets

cng-convert-to-parquet rejects multiple .zip URLs. For per-state/per-region zips, run a preprocess job that downloads in parallel, unzips, and passes the shapefiles (the tool merges them automatically). Worked k8s job: skill dataset-recipes.

Step 2: Generate the pipeline

Vector

cng-datasets workflow \
  --dataset <name> --source-url <url> --bucket <bucket> \
  --h3-resolution 10 --parent-resolutions "9,8,0" \
  --hex-memory 32Gi --max-completions 200 --max-parallelism 50 \
  --output-dir catalog/<dataset>/k8s/<name>

Add --layer <LayerName> for multi-layer sources (one workflow per layer). The / in --dataset creates hierarchical S3 paths; - is used in k8s job names.

Prefer --backend armada for hex-heavy workflows. k8s backend caps at 200 indexed completions and 200 pods namespace-wide. Armada removes both and allows --chunk-size 1 (one feature per job), critical when polygon sizes vary wildly (a single Russia/China chunk can run for hours while 199 others finish in minutes):

cng-datasets workflow ... --h3-resolution 8 --parent-resolutions "0" \
  --max-completions <N-features> --backend armada ...

With --backend armada the CLI emits armada YAMLs for all steps alongside k8s manifests. Submit in order manually (no orchestrator):

armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-setup-bucket.yaml
armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-convert.yaml
armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-plan.yaml      # if emitted (cells-per-chunk plan, #124)
armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-pmtiles.yaml
armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-hex.yaml
# wait for hex, then:
armadactl submit catalog/<dataset>/k8s/<name>/armada-<name>-repartition.yaml

Monitor: https://armada-lookout.nrp-nautilus.io

⛔ Load this before your first armadactl command, not after it hangs — auth is a device-code flow, not the documented PKCE one, and the code expires in 60 seconds. That, install/config, the queue-to-namespace mapping, priority classes, and when to microslice: skill armada-pipeline.

Convert an existing k8s hex YAML to Armada (for rechunking):

from cng_datasets.k8s.armada import k8s_indexed_job_to_armada, save_armada_yaml
import yaml
with open('<name>-hex.yaml') as f: job_spec = yaml.safe_load(f)
armada_spec = k8s_indexed_job_to_armada(job_spec, queue='geo-workflows', job_set_id='<name>-hex')
save_armada_yaml(armada_spec, 'armada-<name>-hex.yaml')

Raster

cng-datasets raster-workflow \
  --dataset <name> --source-url <cog-url> --bucket <bucket> \
  --h3-resolution 8 --parent-resolutions "0" --value-column <band_name> \
  --hex-memory 32Gi --max-parallelism 61 \
  --output-dir catalog/<dataset>/k8s/<name>

Rasters produce COG + hex only (no GeoParquet/PMTiles); a WGS84 COG is always created on NRP S3 first and the hex reads from that COG. Completions default to 122 (one per h0 cell); a regional source should pass --h0-cells (H3 base-cell numbers) or --h0-subset (h0 grid positions) — the two both run 0–121, so a base-cell list given to --h0-subset silently builds the wrong cells (datasets#191, #213).

⛔ Two choices here silently corrupt the data if you get them wrong, and both are measurements, not preferences:

  • --h3-resolution — match the source pixel. Finer than the source adds no information at ~7× cost per step. raster-workflow defaults to 8, not to the pixel; pass it explicitly.
  • --hex-resampling — the reducer (sum / mean / mode / fractions / max / min, default mean). Summing a density is the recurring error: the reducer follows the source units, not the conceptual quantity.

Resolution anchors, the reducer decision table, many-tile mosaicking, and the re-hex campaign checklist: skill raster-hexing.

How --dataset controls naming

The last path segment drives PMTiles source-layer, S3 path, and k8s job prefix:

--dataset value S3 prefix PMTiles source-layer k8s job prefix
calenviroscreen-5-0/ces5 calenviroscreen-5-0/ces5 ces5 calenviroscreen-5-0-ces5
padus-4-1/fee padus-4-1/fee fee padus-4-1-fee
census-2024/tract census-2024/tract tract census-2024-tract

The PMTiles source-layer that MapLibre needs = last segment of --dataset. It does NOT come from the GDB/source layer name.

⚠️ No dots in dataset names. k8s pod names must match [a-z0-9][a-z0-9-]*[a-z0-9]. cng-datasets rejects dotted names at YAML generation. Encode versions without dots (e.g. overture-2026-02-18, not 2026-02-18.0). Pre-existing dotted YAMLs fail pod scheduling — rename before applying.

Underscores are allowed, and the two names then differ. Unlike a dot, an underscore in --dataset is fine: cng-datasets hyphenates it for the k8s job names while the S3 path, the --layer value and the PMTiles source-layer keep it verbatim. So --dataset a/csb_struct produces the job a-csb-struct-workflow alongside s3://…/a/csb_struct.parquet. Useful when a source's own layer names carry underscores and you want the published name to match them — but derive the job name with tr '_' '-' in any script that waits on it, or kubectl wait reports NotFound on a job that is running perfectly well.

Step 3: Apply to the cluster

One-time RBAC (likely already done): kubectl apply -f catalog/<dataset>/k8s/<name>/workflow-rbac.yaml

Per workflow:

kubectl apply -f catalog/<dataset>/k8s/<name>/configmap.yaml \
              -f catalog/<dataset>/k8s/<name>/workflow.yaml

Orchestrator DAG:

  • Vector: setup-bucket → convert → pmtiles + hex (parallel) → repartition
  • Raster: setup-bucket → hex (+ preprocess-cog before hex for multi-tile)

You can also apply individual job YAMLs (<name>-setup-bucket.yaml, <name>-convert.yaml, <name>-hex.yaml, etc.) for step-by-step control.

Step 4: Monitor

kubectl get jobs | grep <name>
kubectl logs job/<name>-convert       # conversion
kubectl logs job/<name>-workflow      # orchestrator

A ~300K-feature vector run typically takes 1–2 hours.

Step 5: Document

Write STAC/README to /tmp/ only — never to catalog/<dataset>/stac/. Upload directly:

rclone copyto /tmp/README.md nrp:<bucket>/README.md
rclone copyto /tmp/stac-collection.json nrp:<bucket>/<dataset>/stac-collection.json

⛔ Do not hand-check the STAC rules — run the gate.

scripts/verify-stac.py --no-data /tmp/stac-collection.json       # pre-publish (static)
scripts/verify-stac.py --bucket <bucket> --dataset <dataset>     # post-cluster (full, data-backed)

Fix every HARD finding before rclone copyto; the post-cluster run must exit 0. CI runs the same verifier on the PR, deriving collections from the s3:// paths in changed catalog/** YAMLs — so a RED check at PR-open is correct, and you re-fire it after the cluster jobs land.

Everything about what to write — the user-facing description register, table:columns placement, SPDX license rules, categorical values, hex per-feature duplication notes, h3:* declarations, point/line notes — is in skill stac-authoring. Load it before writing or editing any stac-collection.json or README.

Step 6: Register in the parent sub-catalog

The STAC catalog is a TREE — datasets belong in their domain/bucket sub-catalog, not the root. Only touch the root when adding a new top-level sub-catalog. Procedure: skill stac-authoring.

Step 7: Get the licence right

This is a metadata step and nothing else. Every collection carries an accurate SPDX license plus a {"rel": "license"} link in its STAC (Step 5). Getting it wrong is how data ends up over- or under-published.

  • Record what upstream actually granted, with evidence.
  • The test is decidability: a reader must be able to tell whether we may redistribute, and under what restrictions. An accurate SPDX id (CC-BY-4.0, CC-BY-NC-4.0, public-domain) settles it by itself. A placeholder (other / various / proprietary) names no terms, so it settles nothing on its own — it needs either a licence link that IS the terms document (WDPA's protectedplanet.net/en/legal) or a description that states the terms. Either satisfies the rule; boilerplate for its own sake does not. verify-stac.py reports license-terms-not-stated (ADVISORY) when neither is present (#651).
  • If the terms are genuinely unconfirmed or no licence was granted, say so plainly in the description — including "no grant has been located", when that is the honest answer. Never assert an SPDX id upstream did not grant, and never name a provider as licensor when no licence was granted. This has gone wrong in both directions — rivers/american-rivers/* asserted CC-BY-4.0 with no licensor behind it, and hazard/mid-century-habitat-climate-exposure asserts CC-BY-4.0 while its own description says the terms "require confirmation".
  • One terms page grants one thing. Two collections citing the same licence URL must declare the same license; verify-stac.py reports license-inconsistent-for-terms (ADVISORY) when they do not — wdpa (proprietary) and wdoecm-may-2026 (other) cite the same Protected Planet legal page.
  • Do not invent a terms URL to clear the gate. If no public terms page exists, the finding stands; record the facts in the description and say so in the issue.
  • Restrictive terms are enforced by policy, not by access control. Match the provider's access model; never exceed it. Licence clauses such as "no redistribution", "not downloadable", "non-commercial only" or "academic use only" are not a reason to keep data out of a public bucket. If the provider serves open downloads and relies on its written terms alone (no login, no credential check — UNEP-WCMC, Protected Planet/WDPA), we host it in an ordinary public-* bucket and enforce the same way: the licence travels with the data (the license field, the licence link, and the terms written into the description). We never build a stronger gate than the provider uses. When the provider itself requires a login or authorisation before the first download, the data goes behind an equivalent gate here, and that is a scoping question for the issue. Whether such data may go on an open-reuse mirror like source.coop is the mirror agent's decision, not yours (see "You publish to NRP" above).
  • Obligations a licence places on the licensee are Carl's, as PI. Examples are "notify the Director", "request permission for commercial use" and "cite as …". State the obligation in the collection description so users see it. Do not raise it as a blocker on the build, treat a published layer as a breach, or ask whether "the ingest" triggers it. Note it in the issue for Carl.
  • One sanctioned exception, and only this one: data contributed directly to us, where no terms were ever published. Here "no public terms page" is not a gap to be researched — it is the complete answer, so a permanently-red finding would be recording a fact as a defect. Publish the statement of terms beside the data (s3://<bucket>/<dataset>/LICENSE.md) and link it. The document must grant nothing: it records what was and was not given. verify-stac.py reports license-terms-self-hosted (ADVISORY) whenever a licence link resolves to our own bucket, so this can never be mistaken for a verified upstream grant. Worked example: high-seas/mpa-candidates (#579). This does not licence guessing anywhere else — for data with a real publisher, an absent terms page still leaves the finding standing.

Hex sizing and resolution

RAM is driven by the H3 cell count of the single largest feature in a chunk — not dataset size, not bounding box. 10,000 small features at 8Gi can be fine while one continent-scale polygon OOMs at 32Gi. OOMs are a tuning signal, not a failure.

h8 is the catalog's universal join key. Default native resolution is 8; every dataset finer than 8 (native h9/h10) MUST carry h8 as a parent, and h0 always (it is the hive partition key and the coarsest common join). Go coarser than 8 only when feature size or source pixel genuinely forces it — such a dataset cannot carry h8, which is a sanctioned case that must be stated in the hex asset description so an agent does not conclude the data is broken.

Always record the chosen native + parent resolutions in the GitHub issue (scope) and in the hex asset's h3:native_resolution / h3:parent_resolutions.

Full memory model, resolution and parent-set decision tables, vector/raster workflow parameters, and reprocessing failed chunks: skill hex-tuning (raster specifics: raster-hexing).

Pod-count good practice (geo-workflows)

Keep ≤200 simultaneous pods across all your jobs. geo-workflows has no enforced ResourceQuota (unlike the legacy biodiversity namespace, which hard-capped at 200), but ≤200 is the good-practice target — oversubscribing is antisocial on shared nodes. Applies to the k8s backend only — armada queues externally and is not constrained.

Rule: never submit more than one k8s hex workflow at a time. With --max-parallelism 50, a single hex job can take 50 pods; on the legacy biodiversity namespace concurrent hex workflows tripped its quota (exceeded quota: reached-quota, used: pods=200); on geo-workflows there is no wall, so self-limit. Run sequentially:

for d in dataset-a dataset-b dataset-c; do
  kubectl apply -n geo-workflows -f catalog/.../k8s/$d/configmap.yaml -f catalog/.../k8s/$d/workflow.yaml
  kubectl wait -n geo-workflows job/$d-workflow --for=condition=complete --timeout=7200s
done

With armada, submit freely — chunk-size 1 (one pod per feature) eliminates "one slow pod blocks everything."

⛔ Pod count is not what decides whether those pods survive. At opportunistic priority a long pod is preempted rather than finished, and retries hide it — a job reports Complete=True while indexes are killed and restarted, and a retry-exhausted index leaves stale data, not missing data. Before submitting anything whose pods run over an hour: skill pod-preemption.

⛔ Default priority has its own cost: NRP's usage policy (#804). NRP checks every non-opportunistic pod older than 30 minutes against its request. Memory is checked when the request is over 2 GiB and must be 20–150% used. CPU is checked when the request is over 1 core and must be 20–200% used. More than 4 out-of-range pods across the account makes NRP deny every Deployment change on Carl's account. On 2026-10-26, a 16Gi-request backfill using 1–2.5Gi blocked a production rollout this way. A separate policy caps limit/request at 1.2. So:

  • Pods under ~1 hour (setup-bucket, daily CronJob runs, short fan-out units) use priorityClassName: opportunistic.
  • Split long work into short units where practical.
  • Pods that must run long use default priority, sized to measured use, not to the generator's defaults. Check kubectl top pod ~30 minutes after launch.

Full rule: skill pod-preemption.

S3 Bucket Layout

# Vector                                # Raster
bucket/                                 bucket/
├── raw/                                ├── raw/
├── dataset.parquet                     ├── dataset-cog.tif
├── dataset.pmtiles                     ├── dataset/hex/h0={cell}/data_0.parquet
├── dataset/hex/h0={cell}/data_0.parquet├── README.md
├── README.md                           └── stac-collection.json
└── stac-collection.json

Troubleshooting

Diagnose before resubmitting. kubectl -n geo-workflows describe pod <pod> first — the same resources will fail the same way. ⛔ Never kubectl delete pod --force / --grace-period=0; let the control plane reap it (skill k8s-never-force-delete).

Causes and fixes — OOMKilled, evictions, ContainerStatusUnknown, flaky-node hangs, ephemeral-storage limits and PVC scratch, pod-quota errors, 404s on convert, blank PMTiles in MapLibre, DuckDB stoi — plus how to reprocess failed chunks: skill job-troubleshooting.

⛔ Generate manifests with cng-datasets; declare every deviation

cng-datasets is tested and has a deploy pipeline. A manifest typed by hand is neither, and it drifts from the tool in ways review does not catch — a missing --nodata, a stale reducer, a namespace that silently resolves to default.

Generate first, always:

cng-datasets raster-workflow ... --output-dir catalog/<dataset>/k8s/<name>   # raster
cng-datasets workflow        ... --output-dir catalog/<dataset>/k8s/<name>   # vector

The generated output carries a # Generation command: cng-datasets ... banner. Keep it — it is how a reviewer, and scripts/check-generated-manifests.py, tells generated from hand-written.

Deviating is allowed. Deviating silently is not. Some shapes the generator genuinely cannot emit yet. When you must, say so in the manifest that deviates:

# HAND-ROLLED: <why the generator cannot emit this> (<tracking issue>)

and for an edit on top of generated output:

# DEVIATION from generated output (<tracking issue>): <what and why>

If the reason is a gap in the tool, file it on boettiger-lab/datasets with a tested MRE (Hard Boundary 2) and cite the issue, so the deviation is temporary rather than permanent. Known open gaps that legitimately force a hand edit today:

gap issue
multi-dimension fan-out (h0 × year/layer/reducer) for per-year COGs (zarr/netCDF cubes: mdim-workflow, #181) datasets#172
Armada converter keeps k8s pod shape, drops retry datasets#183
no --convert-memory; convert fixed at 8Gi datasets#275
PMTiles TIPPECANOE_MAX_THREADS aborts on ≥256-CPU nodes datasets#273

No longer gaps — use the flag, not a hand edit: --namespace (#190), --h0-cells/--h0-subset (#191, #213), --hex-retries/--max-failed-indexes and default priority (#201). Armada steps are still submitted in order by hand; that is the design (#39 closed), not a deviation.

The gate. CI runs scripts/check-generated-manifests.py --base origin/main over the recipes your PR touches. It ratchets rather than blocking on history: the ~33 pre-existing undeclared recipes are grandfathered until someone touches them.

Why this is a hard rule. The catalog currently runs ~90 generated raster recipes against ~26 hand-rolled, and the hand-rolled ones are disproportionately the large, expensive builds — because hand-rolling starts exactly where the work gets hard. The pattern is self-propagating: an agent copies the neighbouring recipe, and a reviewer unsure whether the deviation was deliberate does not challenge it. A stated reason turns "should I push back on this?" into "the file says why", which is the whole point.

What NOT To Do

  • Do not process data locally. CLI generates YAML; the cluster does the work.
  • Do not modify cng_datasets/ source. File an issue (see Hard Boundary 2).
  • Ephemeral is requestable well past the 50Gi default, but the two eviction modes differ — only one is fixed by raising it. A limit eviction (usage exceeds the total limit of containers <N>) means the pod passed its own declared limit: unconditional, priority irrelevant, raise the limit. A node DiskPressure eviction means the node ran short, and the kubelet picks victims in priority order — so a large ephemeral request and priorityClassName: opportunistic compound badly. If you raise ephemeral, do not also run at opportunistic (see the pod-preemption skill). Verified 2026-09-13 by server-side admission: 80Gi, 200Gi and 1Ti are all admitted in both geo-workflows and biodiversity; neither namespace's LimitRange carries a max, and no ResourceQuota in either mentions ephemeral. The real ceiling is node free space, not policy.
  • Coming: Armada on 4x GB10 nodes, each with ~4 TB of local/ephemeral storage, largely free. That changes the calculus recorded here. Today, scratch-heavy raster work is constrained by shared-node disk and must be sized carefully; on that hardware the constraint largely disappears, and Armada becomes the natural home for scratch-heavy and finely-chunked work (datasets#173) rather than a way to escape the k8s completion cap. Until then, size ephemeral from the intermediate and keep priority at default.
  • ⛔ An ephemeral eviction is indistinguishable from an OOM in kubectl get pods. Both report exit 137 with ContainerStatusUnknown or Error. Only kubectl describe pod reveals it: Reason: Evicted / Pod ephemeral local storage usage exceeds the total limit of containers. If pods die at 137 while sitting far below their memory limit, check ephemeral before touching memory — on #515 five fvc pods died at 21–23 GiB of a 128 GiB limit and the memory request was raised twice for nothing. Size hex ephemeral from the localized COG plus margin. The full-grid fill-collapse intermediate that evicted #515 (34 GB per pod on a 4.3 GB LANDFIRE COG) is gone since datasets#209: with several --nodata codes an integer source is collapsed as a VRT lookup table (no raster written), and only a float source materialises a ZSTD-compressed copy; a single nodata code needs no collapse (_collapsed_aggregation_input, cng_datasets/raster/cog.py). The multi-tile mosaic (preprocess-cog) has written compressed, sparse intermediates since 0.11.3 (datasets#277).
  • Keep ephemeral-storage modest. Generator defaults: hex 10Gi (vector) / 20Gi (raster, --hex-storage), preprocess-cog 50Gi (--cog-storage), repartition 50Gi, raster merge 100Gi (--merge-storage). 50Gi is a good default and what most jobs need. It is not a hard cap in geo-workflows: the LimitRange there sets default: 50Gi with max empty, so a larger request is legal and passes validation. (The 50Gi clamp was the legacy biodiversity namespace, which these jobs no longer run in.) Go above 50Gi only with a reason recorded in the manifest — wrc-2 warps a striped CONUS source at 60Gi (#592) — and keep it courteous on shared nodes.
  • Do not use multiple .zip URLs with cng-datasets workflow. Preprocess first.
  • Do not record operational/how-to-work lessons in agent memory (~/.claude/.../memory). This repo is cloned and run by students — and soon by always-on headless agents (Hermes/openclaw). Anything that should shape how tasks run here belongs in this AGENTS.md or a local skill (.claude/skills/), so every clone and headless run behaves the same. A lesson saved only to one VM's memory silently diverges your experience from everyone else's. (Memory remains fine for genuinely personal, non-shareable session context.)

Reference Examples

Worked end-to-end ingests to copy from — PAD-US multi-layer GDB (5 spatial layers), Census TIGER per-state zips with a preprocess job, and a single-COG carbon raster: skill dataset-recipes.

Checking for User Dataset Requests

Request form: https://data-requests.nrp-nautilus.io/ (per-app routes, e.g. /tpl). Submissions stored as JSON in s3://public-requests/dataset-requests/.

Check at session start:

curl -s https://data-requests.nrp-nautilus.io/api/requests | python3 -m json.tool
rclone ls nrp:public-requests/dataset-requests/

Each submission: app, timestamp, plus user fields (dataset name, description, contact).

Triage:

  1. Review; decide if in scope.
  2. If yes: file a GitHub issue on boettiger-lab/data-workflows with gh issue create --template dataset-import.yml, then edit in the scope fields.
  3. Tag with the app if it came from a specific form.

Form deployment: dataset-requests/ — see dataset-requests/README.md for adding routes / redeploying.