Multilingual agentic coding benchmark grounded in language, region, and culture.
Terminal-Bench-LILT is a suite of 300 authentic terminal-based coding tasks in ten languages — Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese (30 per language). Each task targets a problem specific to non-English software development with no direct English equivalent: internationalization, encoding, text normalization, script rendering, and regional or cultural conventions. Every task was authored from scratch in its native language by a native-speaker programmer and passed a four-layer quality-control pipeline (deterministic checks, LLM judges, initial human review, final human review).
The difficulty comes from the locale-specific knowledge the instructions leave implicit — the things a native developer takes for granted — not from general engineering complexity.
This repository contains 10 sample tasks from the benchmark. The full set is held out.
| Task | Lang | Category | Difficulty | Challenge |
|---|---|---|---|---|
cs-implicit-alphabet-frequency |
cs | Cultural & Other | Medium | Count occurrences of each Czech alphabet letter without being told what the Czech alphabet is — all 42 letters, including accented characters and the ch digraph. |
cs-nbsp-justify |
cs | Cultural & Other | Hard | Split Czech sentences into xelatex-justified lines with no Overfull hbox, honoring Czech orthotypographic non-breaking-space and soft-hyphen rules. |
cs-rozuctovani-tepla |
cs | Cultural & Other | Medium | Fix a script that audits a Czech SVJ heating-cost allocation against vyhláška 269/2015 Sb. |
de-bgb-citation-resolver |
de | Cultural & Other | Medium | Resolve German civil-code citations (§ 433 Abs. 1 Satz 2) against a BGB XML file, counting Sätze per Absatz while respecting period-terminated legal abbreviations. |
de-umsatzsteuer-klassifikation |
de | Cultural & Other | Hard | Classify German products by statutory VAT rate (7% vs 19%) under §12 UStG and Anlage 2. Many names are compounds where the Grundwort sets the rate while the Bestimmungswort misleads. |
es-locale-search-analyzer |
es | Text Handling | Easy | Build a Spanish search analyzer with locale-correct collation that emits snippets preserving the original surface form. |
ja-terminal-width |
ja | Internationalization | Medium | Wrap NOC-log lines to 40 display columns: strip ANSI escapes, fold halfwidth katakana (merging ﾞﾟ), count CJK/emoji as 2 and variation-selector clusters as one 2-wide unit, never split a wide char, keep bytes otherwise verbatim. |
sr-cadastral-parcel-ledger |
sr | Text Conversion | Easy | Normalize Serbian cadastral parcel requests across registry aliases, script variants, area units, ownership shares, and Serbian collation into a JSON ledger. |
sr-locale-sort |
sr | Cultural & Other | Easy | Fix a buggy Serbian Latin sorter so the digraphs dž, lj, and nj are treated as single letters in any word position, not only word-initially. |
tr-username-dedup-casefold |
tr | Text Handling | Hard | Detect duplicate registrations in a Turkish CRM using Turkish-locale username canonicalization (NFC + I-pair pre-fold + casefold), which str.lower() and str.casefold() do not implement. |
Each task runs under Harbor in an isolated container with deterministic verification:
tasks/<task-name>/
├── instruction.md # instruction, written natively in the target language
├── instruction_en.md # English translation
├── task.toml # metadata: language, category, difficulty, descriptions
├── README.md # design rationale, realism, and the linguistic/technical challenge
├── environment/ # Dockerfile + native-language data assets
├── solution/ # oracle solution accepted by the verifier
└── tests/ # deterministic verifier (sample and unseen probes)
Install Harbor:
uv tool install harborRun the oracle (reference) solution, which must always pass:
harbor run -p tasks/ja-terminal-width --agent oracleThese tasks carry harbor-canary GUIDs. Please do not train on this data, and exclude documents
containing these strings from training corpora.
© LILT, Inc. All rights reserved. You are welcome to clone these tasks and run them with Harbor for evaluation or research, and to quote or write about them. Redistribution or commercial use requires permission — please get in touch.