Wave: 1 · Depends on: none · Part of: #309

Summary

Deliver a public library API for token counting across providers: tiktoken cl100k_base and o200k_base for OpenAI-family models, an offline approximation for Anthropic models, and a conservative heuristic fallback for everything else. Today skim's only tokenizer is cl100k_base, hard-wired as a pub(crate) function inside the rskim binary crate — nothing outside the CLI can count tokens, and everything inside it counts in one encoding regardless of the target model. Every Layer-3 component built after this — per-block never-inflate checks, budget truncation, savings accounting, stale-content compaction — needs to count tokens in the encoding the target model actually uses; counting in the wrong encoding produces savings numbers that are estimates dressed as measurements. This ticket supplies the counting substrate as a Wave-1 foundation with no dependencies, composing with the already tokenizer-agnostic rskim_core::truncate_to_token_budget. Network token-counting APIs (e.g., Anthropic's count-tokens endpoint) are permitted only as an explicit opt-in construction, never reachable from any default or hot path — and that opt-in network counter is delivered by this ticket (behind a non-default cargo feature; see constraint 15), not deferred to the proxy ticket.

Why now — evidence

PRISM-derived (all claims verified in the 2026-06-09 competitive study):

• PRISM L3 follow-up: extract shared log-line heuristics into a reusable library #327 (35 comments, their most-discussed open issue): real sessions see only 0–15% savings because file-read outputs and aged messages — the dominant agentic token pool — go uncompressed; the maintainer concedes the gap. This pool is the justification for skim's entire Layer-3 slate, and every downstream compression/budget decision in that slate consumes token counts. Tokenization is the first brick.
• PRISM #645 (independent benchmark, publishing ~2026-06-17): measured ~10.2% average savings at +0.9s/turn against PRISM's vendor-claimed fleet median of 4.8%. The lesson: savings claims that aren't grounded in real per-provider token counts don't survive independent measurement. skim's observability ticket (L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305) records only real measured raw−compressed counts — which requires this API.
• User demand for savings-accounting transparency: PRISM feat: add cargo test parser with three-tier degradation (#46) #83, #491, Temporal arms: make membership consistent with the FileManifest walked+tracked union (follow-up to #408) #454.

What PRISM got right here: deterministic local tiktoken counting (cl100k for GPT, o200k for newer models) exposed as a library-reusable interface with lazy initialization. The study's reliability filter found PRISM's deterministic components have no bug cluster against them — every PRISM reliability failure traces to ML, scoring heuristics, or proxy state. Tokenization belongs to the safe lane. Also instructive: PRISM #552 (their unknown-model-snapshot fallback fix) shows they had to retrofit graceful handling of unknown model identifiers after shipping — a lesson this ticket adopts up front as an acceptance criterion (unknown IDs resolve to documented defaults and never error).

What PRISM got wrong here: (1) Anthropic counting that leans on a network count-tokens API places a network call near the hot path; skim's binding stance is offline approximation by default, network strictly opt-in. (2) A SentencePiece fallback and similar heavyweight dependencies feed PRISM's install-reliability bug cluster (PRISM #525: macOS Intel unsupported due to ONNX; #355: native extension missing from wheel; #408/#593: unusable on a current Python release). skim ships a single static binary across 7 targets; this ticket replaces SentencePiece with a dependency-free conservative heuristic.

skim-internal evidence (audited at commit e3dec42, 2026-06-09):

• crates/rskim/src/tokens.rs: cl100k_base only (lines 11, 18); count_tokens is pub(crate) (line 22) in the binary crate, not rskim-core — not reusable as a library API. Lazy OnceLock<CoreBPE> init (lines 14–19) calls .expect(), i.e., panics on init failure. Counting goes through encode_with_special_tokens (line 24), so special-token text such as <|endoftext|> is counted as ordinary text — behavior the new API must preserve (constraint 13). A workspace-wide search found no other tokenizer (no o200k, no Anthropic tokenizer, no SentencePiece); tiktoken-rs 0.7 is the only tokenization dependency.
• rskim_core::truncate_to_token_budget (lib.rs:376) already takes an injected counting closure F: Fn(&str) -> usize — core is tokenizer-agnostic by design. The composition point exists; only the supply side is missing.

Scope

In

• A public library API exposing token counting for: OpenAI cl100k_base, OpenAI o200k_base (both available in the existing tiktoken-rs 0.7 workspace dependency — no new tokenizer crates), Anthropic via deterministic offline approximation, and a conservative heuristic fallback for unknown providers. All four counters must be available under default cargo features.
• A model-ID → encoding mapping with a two-tier fallback rule, stated here as binding: an unknown model ID within a known provider family resolves to that family's documented default encoding (OpenAI family default: o200k_base — an unrecognized OpenAI ID is most likely a model newer than the table; Anthropic family default: the offline approximation); an unknown provider resolves to the conservative heuristic. The table is the single source of truth consumed by the public API and is exposed via a public lookup function so L3: LLM message model — parse/serialize Anthropic & OpenAI request bodies + content-block classification #302 and L3: HTTP proxy foundation — provider routing, SSE/chunked streaming passthrough, stateless fail-open core #303 can reuse it without duplication. (This adopts the lesson of PRISM #552 before shipping rather than retrofitting it.)
• Closure adapters satisfying Fn(&str) -> usize so any provider's counter plugs into rskim_core::truncate_to_token_budget and the never-inflate guardrail's token comparison.
• An explicitly-constructed, clearly-separated opt-in network counter for Anthropic's count-tokens endpoint, with bounded timeout and bounded retries, unreachable from any default constructor or existing code path. This deliverable is unconditionally in scope for this ticket (not contingent on L3: HTTP proxy foundation — provider routing, SSE/chunked streaming passthrough, stateless fail-open core #303); its interface contract and dependency policy are pinned in constraint 15.
• Migrating crates/rskim/src/tokens.rs internals to consume the new API, with zero behavior change to existing commands (cl100k remains the CLI default).
• If the crate-placement open question resolves to a new publishable crate: the release-pipeline updates that creation entails — crates.io publish order in the release workflow (rskim-core → new crate → rskim), version sync in the release-prep process, and workspace metadata — are in scope for this ticket.

Out (decided exclusions, not oversights)

• SentencePiece / Google / Cohere / Mistral tokenizers — not in the adopted provider scope (Anthropic + OpenAI first); heavyweight deps are PRISM's install-bug lineage (#525/Wave 4 / Search: lexical query results are irrelevant — bigram index yields near-uniform BM25F scores and misses exact matches #355).
• ML-based token estimation — violates the slate-wide no-ML / deterministic / single-binary principles. ANTI-FEATURE.
• Default-on or hot-path network counting — binding stance: network counting is explicit opt-in only.
• Async (tokio) wrapper for the network counter — this ticket ships a synchronous blocking client only (constraint 15). Async adaptation is deferred until a consumer exists — no ticket in the slate needs it today: L3: HTTP proxy foundation — provider routing, SSE/chunked streaming passthrough, stateless fail-open core #303 explicitly excludes plumbing tokenization into the proxy, and L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305 runs all token counting synchronously on its analytics background thread.
• Provider detection from request URL/headers — belongs to L3: LLM message model — parse/serialize Anthropic & OpenAI request bodies + content-block classification #302 and L3: HTTP proxy foundation — provider routing, SSE/chunked streaming passthrough, stateless fail-open core #303; this API takes explicit provider selection only.
• Per-provider analytics recording / schema migration — owned by L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305.
• Importance scoring / budget-driven message dropping — PRISM deleted ~10K LOC of it (2026-05-01 REALIGNMENT: "wrong mental model"); skim compresses in place, never drops turns. ANTI-FEATURE slate-wide.
• Model pricing / model-metadata registry — deferred (nice-to-have in the study's capability matrix).
• Quantitative accuracy benchmarking campaign — explicitly deferred by owner decision; PRISM #645's independent benchmark (~2026-06-17) is a calibration input for later. (A minimal directional quality bar for the Anthropic approximation is still in scope — constraint 14 — because without it the counter's defining property is untestable.)

Relationship to existing specs

• v3 Vision: Three-Layer Context Optimization Platform #98 (v3 Vision: Three-Layer Context Optimization Platform): implements a Layer-3 foundation component of the vision. No amendment.
• v3: CCR — Compress-Cache-Retrieve for Reversible Compression #100 (CCR spec; CCR is skim's on-demand reversible compression — a content-addressed cache plus retrieve tool): untouched by this ticket. CCR will later consume this API for budget/eligibility decisions; any amendment of v3: CCR — Compress-Cache-Retrieve for Reversible Compression #100 is owned by L3: CCR — on-demand reversible compression: content-addressed cache + retrieve tool #308.
• v3: Layer 3 — LLM Message Compression Proxy (skim proxy) #101 (Layer-3 proxy spec):
  • Absorbs: the "Provider-Aware Tokenization" design section, and the "Multi-provider tokenization" checklist item from v3: Layer 3 — LLM Message Compression Proxy (skim proxy) #101 Phase 3 ("Advanced", weeks 9–10) — pulled forward to Wave 1 because L3: Per-content-type compression of message content blocks (reuse existing engines) #304 and L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305 depend on it.
  • Amends: v3: Layer 3 — LLM Message Compression Proxy (skim proxy) #101's Integration Points list assumes count_tokens() is callable from rskim-core; it is not (it is pub(crate) in the binary crate). This ticket makes that assumption true by shipping the public library API.
  • Drops: Google/SentencePiece from v3: Layer 3 — LLM Message Compression Proxy (skim proxy) #101's provider list (see exclusions); the "detect from request URL/headers and use correct tokenizer" behavior (detection moves to L3: LLM message model — parse/serialize Anthropic & OpenAI request bodies + content-block classification #302/L3: HTTP proxy foundation — provider routing, SSE/chunked streaming passthrough, stateless fail-open core #303; this API is selection-only). This ticket also carries none of v3: Layer 3 — LLM Message Compression Proxy (skim proxy) #101's Phase 2 importance-scoring language — that concept is excluded from the entire slate.

Design constraints & invariants

Each stated as a testable constraint:

1. No network in default paths. Constructing any default counter and counting text opens zero sockets. The network counter is a distinct, explicitly-named construction behind a non-default cargo feature (constraint 15); no default constructor, CLI path, guardrail check, or truncation path can reach it. Verifiable under a network-denied test environment.
2. Deterministic. Identical input yields an identical count across calls, threads, processes, and platforms. No wall-clock, RNG, locale, or environment dependence in any counting path. Property test: N repeated counts of the same corpus, all equal; golden vectors checked on all CI platforms.
3. Bounded. Counting is a single pass, O(input length). The opt-in network path has a fixed request timeout and a fixed retry cap. No unbounded loops or retries anywhere.
4. Result types, no panics. Tokenizer initialization and all fallible operations return Result; no unwrap/expect outside tests. This fixes the existing get_tokenizer() .expect() panic path when the code migrates.
5. Single binary, no config files. Vocabulary data is embedded at compile time; no runtime downloads; no .skimrc-style configuration. Provider selection happens only through API arguments.
6. Stateless and pure. Counters are pure &str → count functions after construction; lazy OnceLock-style one-time init is permitted (existing precedent in tokens.rs), mutable global state is not.
7. Thread-safe. Counters are Send + Sync (consumed by rayon parallel CLI paths today, and later — inside the tokio-based proxy process — by L3: Per-content-type compression of message content blocks (reuse existing engines) #304's per-block never-inflate checks and L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305's analytics background thread). Closure adapters satisfy Fn(&str) -> usize to match the truncate_to_token_budget<F> bound.
8. Encoding consistency for never-inflate. Any before/after comparison (guardrail token slow path) must use the same encoding on both sides; the API design must make mixed-encoding comparison structurally awkward (e.g., one counter instance owns one encoding and counts both sides).
9. Conservative fallback direction — pinned semantics. When the provider is unknown, the heuristic must not undercount. Binding test semantics: per document — for every document in the AC6 conservatism corpus, the heuristic's count must be ≥ max(cl100k count, o200k count) for that same document; corpus-aggregate compliance is not sufficient. Rationale: an overcount wastes a sliver of budget; an undercount overflows the real context window after budget truncation. The bound is binding; the formula is the planner's.
10. Existing behavior frozen. cl100k remains the default for every place the CLI counts today (--show-stats, analytics, guardrail); all existing tests pass unmodified; recorded analytics values stay comparable with history. Per-provider recording arrives only with L3: Proxy observability — analytics schema v4, per-provider/model savings accounting #305.
11. Latency budget. Binding slate-wide constraint (self-contained restatement): total added proxy latency must stay within single-digit milliseconds per turn, so counting must never dominate that budget. Concretely: counting a 100 KB input after initialization must complete in <5 ms locally (target) and <25 ms as the hard CI assertion (margin for shared runners); regressions are gated by a criterion benchmark with a tracked baseline rather than by wall-clock assertions alone.
12. Zero warnings; rustdoc on every public item (repo policy; design decisions documented in code).
13. Special-token semantics pinned. All tiktoken-backed counters (cl100k, o200k) count via encode_with_special_tokens, matching the current tokens.rs behavior (line 24): special-token strings such as <|endoftext|> are counted as ordinary text, never dropped and never an error. This choice propagates to the Anthropic approximation's basis and to both sides of any guardrail before/after comparison.
14. Anthropic offline approximation — minimal quality bar. Same conservatism direction as constraint 9, bound to cl100k: for every document in the AC3/AC5 golden corpus, the Anthropic offline count must be ≥ the cl100k count for the same text. Basis: Anthropic's public token-counting guidance states that tiktoken-style counts undercount Claude tokens by roughly 15–20% on typical text (more on code and non-English input), so an Anthropic estimate below cl100k is known-wrong in the unsafe direction. The approximation must apply a documented uplift over its cl100k/byte basis, and rustdoc must state the basis and its honesty limits (it is an approximation, not a measurement). Accuracy beyond this directional bar is deferred with the benchmarking campaign (see open question 3).
15. Network counter interface contract and dependency policy. Pinned for the planner:
    • Placement: lives in this ticket's library behind a non-default cargo feature; downstream crates that ever need it enable the feature explicitly (no slate ticket does today). The default build of the library and of the skim binary carries no HTTP client or TLS stack — verified by a dependency-tree check in CI.
    • Auth: API key read from the ANTHROPIC_API_KEY environment variable at construction (env var only — no config files per constraint 5; never logged). Missing key ⇒ construction returns Err.
    • Payload mapping: Anthropic's count-tokens endpoint (POST /v1/messages/count_tokens) counts a model+messages request body, not raw text. The adapter wraps the input &str as a single user message, so the returned count includes the endpoint's message-envelope overhead relative to bare text. Rustdoc must state this and that network counts are therefore not directly comparable to offline raw-text counts.
    • Model parameter: a required explicit argument at construction — no default model ID baked in.
    • Execution model: a synchronous blocking call honoring the fixed timeout and bounded retry cap of constraint 3. This library takes no tokio dependency; async adaptation is deferred until a consumer exists (no slate ticket needs it today — see Scope/Out).

Reusable skim assets

• crates/rskim/src/tokens.rs — current implementation to migrate: cl100k_base via tiktoken-rs (lines 11, 18), pub(crate) count_tokens (line 22), encode_with_special_tokens call (line 24), lazy OnceLock<CoreBPE> (lines 14–19), TokenStats (line 30).
• crates/rskim/tests/cli_tokens.rs — existing --show-stats integration tests are pattern-based (contains assertions), not exact golden output; the behavior-freeze acceptance criterion below requires capturing an exact golden run from current main before migration starts.
• crates/rskim-core/src/lib.rs:376 — truncate_to_token_budget<F: Fn(&str) -> usize>: AST-aware, priority-based truncation with an injected counting closure; documented invariant that a budget of 0 (or smaller than the omission marker) returns an empty string rather than violating the budget.
• crates/rskim/src/output/guardrail.rs — never-inflate guardrail: two-tier check (byte fast path, then token comparison); a direct consumer of counting and the model for constraint 8.
• crates/rskim/src/cascade.rs — token-budget cascade (progressively more aggressive modes until output fits a budget); a downstream consumer whose budgets become provider-accurate via this API.
• Workspace dependency tiktoken-rs = "0.7" (root Cargo.toml) — already provides both cl100k_base and o200k_base; no new tokenizer dependency required.
• Additive-crate precedent: rskim-search depends on rskim-core without changing rskim-core's public API — the pattern to weigh in the crate-placement open question.

Acceptance criteria

1. An external crate (compile-tested example or integration test in a separate test crate) can call the public API under default cargo features to count tokens for each of: cl100k, o200k, Anthropic offline approximation, and the fallback heuristic.
2. For each provider, a closure adapter from this API compiles against and executes through rskim_core::truncate_to_token_budget (integration test: truncate the same source under each provider's counter; output respects the budget per that counter).
3. cl100k counts from the new API are identical to current tokens.rs counts over a golden corpus (no regression in existing stats). The corpus must include special-token strings (e.g., <|endoftext|>) so a divergence from the encode_with_special_tokens behavior of constraint 13 is actually caught.
4. o200k counts match golden vectors checked into the repo, generated once with the upstream Python tiktoken reference implementation (pinned version recorded in the golden-file header) over a fixed corpus drawn from tests/fixtures/ plus special-token and multilingual samples, with a documented regeneration script so the vectors are reproducible.
5. The Anthropic offline counter is deterministic, performs zero network I/O (verified under a network-denied test), meets the constraint-14 bar (per-document count ≥ cl100k over the golden corpus, via a documented uplift), and its approximation basis and accuracy limits are documented in rustdoc.
6. The fallback heuristic is deterministic, never returns an error, and — per document over a fixed mixed corpus that must include prose, code, JSON, logs, CJK text, and emoji/symbol-heavy samples — produces a count ≥ max(cl100k, o200k) for that document (the pinned semantics of constraint 9).
7. The model-ID → encoding mapping resolves, at minimum: gpt-3.5-turbo, gpt-4, gpt-4-turbo → cl100k; gpt-4o, gpt-4o-mini, o1, o3, gpt-4.1 → o200k; current claude-* model IDs (including claude-sonnet-4-5, claude-opus-4-5, claude-haiku-4-5) → Anthropic offline approximation. A test asserts the public API consumes this table as its single source (no second mapping anywhere in the workspace) and that the public lookup function is exposed.
8. Unknown-ID resolution follows the two-tier rule and produces a count without error or panic: an unknown model ID in the OpenAI family resolves to o200k_base; an unknown Anthropic-family ID resolves to the offline approximation; an unknown provider resolves to the conservative heuristic (the PRISM #552 lesson, adopted up front).
9. The network counter can only be obtained through an explicitly-named opt-in constructor behind the non-default cargo feature; it enforces a fixed timeout and bounded retry cap; on any failure (missing ANTHROPIC_API_KEY, network error, non-success response) it returns Err without side effects. Tests prove no default constructor or existing CLI path can reach it, and a CI check proves the default-features build pulls no HTTP/TLS dependency (constraint 15).
10. Tokenizer initialization failure surfaces as Result::Err, not a panic; no unwrap/expect in non-test code (clippy/CI gate).
11. Determinism and thread-safety: counting the same input concurrently from multiple threads yields identical results; counter types are statically asserted Send + Sync.
12. Performance: a criterion benchmark with a tracked baseline covers counting a 100 KB input after initialization (target <5 ms locally), and a hard test asserts <25 ms in CI (the generous-margin bound of constraint 11).
13. Negative: no output, exit code, or analytics value of any existing skim command changes — the full existing test suite passes unmodified, and --show-stats output is unchanged against an exact golden run. Current --show-stats tests are pattern-based, so the exact golden output must be captured from current main and checked in before the migration begins.
14. Negative: no default code path opens a network socket (network-denied CI job covering the library's default surface).
15. Initialization-failure handling is verified at unit level: every Result-returning constructor's Err path is exercised by a test, and code review confirms each CLI call site of the counting API handles Err by reporting the failure (stderr) without altering or suppressing the command's primary transformed output. No new CLI injection seam is required for this criterion — cl100k_base() over compile-time-embedded vocabulary has no realistic runtime failure trigger, so an end-to-end CLI failure injection is explicitly not demanded.
16. Zero compiler/clippy warnings; every public item documented.

Open questions for the implementation planner

1. Crate placement: extend rskim-core vs a new additive crate (e.g., rskim-tokens). rskim-core is a published, pure, no-I/O transform crate; embedded BPE vocabularies add real binary weight to it. rskim-search proves the additive pattern. Note truncate_to_token_budget only needs a closure, so core need not depend on the tokenizer crate in either design. (If a new crate: the publish-pipeline updates are in scope per the In list.)
2. API shape: enum-selected free function (count_tokens(text, provider)) vs constructed counter objects yielding Fn(&str) -> usize closures — and how Result-at-construction reconciles with an infallible counting closure at the truncate_to_token_budget boundary.
3. Anthropic approximation method and calibration basis: scaled cl100k? byte-ratio model? What uplift factor honestly satisfies constraint 14 without the deferred benchmarking campaign? Should a small golden set captured once via the opt-in network counter (checked in with provenance: capture date, model ID, endpoint) back the directional bar with a documented tolerance? No captured reference data exists in the repo today — capturing it requires a network step the planner must schedule explicitly. Should PRISM #645's published data (~2026-06-17) be a planned later calibration input?
4. Binary size: AC1 requires all four counters under default features, so feature-gating may only add (the network feature) — it cannot remove cl100k/o200k vocabularies from the default build. Is the added o200k vocabulary weight acceptable across the 7 release targets, and if not, what in-build mitigation (e.g., compressed embedded vocab with one-time decompression at init) is worth its complexity?
5. Model-ID → encoding mapping breadth and update policy: the reuse contract is fixed (the table lives in this ticket's library behind a public lookup function — see In); the open parts are table breadth beyond the AC7 minimum, the matching strategy (exact IDs vs family-prefix rules vs both), and the policy for keeping the table current across skim releases as new model IDs appear.
6. Network counter surface details (placement and dependency policy are decided — constraint 15): the cargo feature name, the constructor name, whether an additional runtime env-var guard is warranted on top of explicit construction, and the HTTP client crate choice (must keep a minimal dependency tree and stay entirely out of the default build).
7. TokenStats / format_number: stay binary-private (display concerns) or move with the API?

Supplementary reading

Local-only, git-ignored artifacts (not available to external readers; the ticket above is self-contained without them):

• .devflow/docs/competitive/adoption-decisions.md — authoritative scope filter and evidence index for the Layer-3 slate.
• .devflow/docs/competitive/prism-synthesis.md — §F8 (multi-provider tokenization feature analysis).
• .devflow/docs/competitive/prism-capability-matrix.md — group (d) Multi-Provider & Tokenization.
• .devflow/docs/competitive/raw/skim-surface.md — skim internal capability audit (tokenizer section), commit e3dec42.