OPENNLP-1931: Regex removal (3a/10): Read opennlp-dl vocab and config JSON with a strict scanner - #1277
OPENNLP-1931: Regex removal (3a/10): Read opennlp-dl vocab and config JSON with a strict scanner#1277krickert wants to merge 20 commits into
Conversation
9ba4095 to
944ab39
Compare
rzo1
left a comment
rzo1
left a comment
There was a problem hiding this comment.
Little time, so here is a GPT 5.6-sol review instead for now
No blocking findings. No additional API or parsing regression found. The old and new JSON implementations agreed across 60,000 generated cases.
Validation across the combined stack: 1,856 targeted tests, zero failures, one skipped.
944ab39 to
fed91ba
Compare
|
Here are some additional comments. Def. needs an eval build ( Blocking
Minor
Verified:
|
1 similar comment
|
Here are some additional comments. Def. needs an eval build ( Blocking
Minor
Verified:
|
|
Follow-up: the pre-existing BOM problem in |
krickert
changed the title
8699305 to
a4ed75a
Compare
|
CI is red. |
rzo1
changed the title
rzo1
changed the title
886b1e8 to
f632f55
Compare
The deep-learning module reads two HuggingFace-shaped JSON files, the vocabulary and the model configuration, with regular expressions that look for a string literal, a colon with optional ASCII whitespace, and then either a digit run or another string literal. JsonScan collects the offset helpers those two readers need: the closing quote of a literal honoring backslash escapes, the closing quote of a literal that must stay on one line, a colon surrounded by whitespace, a whitespace run, and a digit run. The class is public so the doccat subpackage can reach it and is marked Internal. The tests cover accept and reject sides of every helper, including non-ASCII spaces and digits, the five line terminators, and supplementary-plane characters.
AbstractDL.loadJsonVocab used a find() loop over a pattern matching a string literal with backslash escapes, optional ASCII whitespace around a colon, and a run of ASCII digits, anywhere in the text. The loop now walks the text with JsonScan: from each quote it finds the closing quote, the colon, and the digit run; on success it records the entry and resumes after the digits, otherwise it resumes at the character after the quote exactly like the matcher did, so a quote inside a skipped literal can open the next candidate. The method is now package-private so tests feed it text directly. The added parameterized tests pin the odd inputs: a value that is not an integer is skipped, a fractional value keeps its integer prefix, escaped and unicode-escaped keys, keys spanning a line, a backslash before a line terminator, an empty key, whitespace and newlines around the colon, a non-ASCII space after the colon, and a later entry overwriting an earlier one.
DocumentCategorizerConfig.fromJson used a DOTALL pattern to cut the text between the brace after "id2label" and the first closing brace, and a second pattern to pull "key": "value" pairs out of that text with the key running to the next quote and the value, lazily, to the next quote on the same line. Both are now cursor scans over JsonScan: the key literal is located with indexOf and retried at the next occurrence when no colon and brace follow it, the content ends at the first closing brace even when that brace belongs to a nested value, and the entry loop resumes after a matched value or at the character after a quote that opened no entry. The parameterized tests pin the nested brace cut, a brace inside a value, a missing or non-object id2label, whitespace and newlines around colons, an escaped quote inside a value, a value spanning a line, an empty key, a numeric value, an overlapping key literal, and supplementary-plane keys and values.
Rebase onto ccea670: skipWhitespace now takes only space, tab, LF, and CR (vertical tab and form feed stop the run, as JSON requires); endOfDigits and the line-terminator check delegate to the shared StringUtil versions. The affected unit cases now pin the four-character set. Verified: dl suite 129 run, 0 failures.
…g the old patterns The scans that replaced the regular expressions had kept their gaps: a decimal value yielded an integer prefix, a negative id was skipped, the id2label object was cut at the first closing brace inside a label, a key could span a line while a label could not, and a "key": digits entry was taken as a token wherever it appeared in the file. JsonScan is now a small cursor-based JSON parser that follows RFC 8259 for structure, whitespace and string escapes, reports the offset of a malformed document, and returns an object as a list of members. It uses the shared StringUtil.isAsciiDigit and endOfAsciiDigits. - A JSON token file is one object that maps tokens to non-negative integer ids; any other value, a missing comma or trailing text is an IllegalArgumentException that names the token or the offset. - id2label is taken from the top-level object of the configuration; labels are decoded from their escapes, a brace inside a label does not end the object, and a value that is not a string is rejected. - A control character inside a string is kept as content, so a label wrapped over 2 lines still parses. - fromJson(null) throws IllegalArgumentException like the other parsers. The manual describes both token file layouts and the configuration rules in the document categorizer and name finder chapters. Tests cover the accept and reject side of each parser, escapes and surrogate pairs, nested values, duplicate keys, and the RFC 8259 whitespace set.
…vocab (red) The old pattern took a tokenizer.json file as well, with the added_tokens ids and vocab_size landing in the map. The strict parser rejects the file. The test fixes the intended result: the entries of model.vocab, and no other member, form the token map.
…reen) A top-level object without integer members that has a model object with a vocab object is the tokenizer.json layout; the tokens of model.vocab form the token map under the same strict rules. Any other layout keeps the current error. The manual names the layout in the document categorizer chapter.
A vocab.json file that starts with U+FEFF is read as plain text with the full content as one token, and a config.json with the mark is rejected as malformed at offset 0. RFC 8259 section 8.1 lets a parser ignore the mark, so JsonScan.document and the vocab file reader must skip it, and reject it at any later offset.
…(green) JsonScan.document skips U+FEFF as the first character, per RFC 8259 section 8.1, and reports it as malformed at any later offset. The vocab file reader drops the mark before it looks for the opening brace, so a vocab.json written with a mark is read as JSON, and a plain vocab.txt with a mark gives the first token without it. The file is read once and split into lines from memory.
…layouts JsonScan: the offset and reason in the message for text cut off after each token of a member, for content after the object, for separators and non-JSON whitespace, and for bad escapes; values nested 500 levels deep are skipped in full and rejected at the end when cut off; CR and CRLF outside strings are whitespace and inside strings are content; keys written with escapes decode. Vocabulary: tokenizer.json with CRLF or no whitespace, added_tokens ids that collide with vocab ids before or after the model, an empty vocab object, vocab objects at other depths, a later model, vocab, or token winning, member names written as escapes, ids that do not fit into an int naming the token, leading zeros, each proper prefix of a tokenizer.json, content after it, and files with CRLF in both formats. Configuration: CRLF layouts, id2label written with an escape, nested members around it, labels with line breaks, each proper prefix of a config, and content after it, with the offset in the message.
The manual states that a byte order mark at the start of a vocab or configuration file is skipped in both layouts, as RFC 8259 allows.
U+FEFF appeared in the test sources as a raw character, which no editor shows. It is now written as a Unicode escape.
…uration (red) A configuration text that is only the mark, or the mark followed by whitespace, is rejected as malformed at offset 1. Since the mark is skipped as content, such text must read as blank: a configuration without labels. A mark at any later offset is still malformed.
DocumentCategorizerConfig.fromJson drops one leading U+FEFF before it tests for blank text, so a file with the mark and whitespace is a configuration without labels, the same as an empty file. JsonScan reports a mark at any later offset as before.
…s, and file errors (red) Deeply nested values must be skipped without a stack overflow, a bad escape inside a nested string or an array must be rejected, NaN and the infinities that Python writes must be skipped in members that are not read, offsets and members outside the text must be rejected with an IllegalArgumentException, a malformed vocabulary file must surface as InvalidFormatException, a Unigram tokenizer.json must be named as an unsupported layout, and a lone control or space character is not blank configuration text.
…rrow the JSON API (green) JsonScan skips objects and arrays with an explicit stack, so a deeply nested or truncated file is rejected with an IllegalArgumentException that names the offset instead of a StackOverflowError. Each string is checked for bad escapes when it is passed, in keys and values at any depth. NaN, Infinity, and -Infinity are accepted where a value is skipped, as Python's json module writes them, and rejected where a value is read. The class exposes one reader, stringObject, which DocumentCategorizerConfig uses for id2label; the other members are package-private, arguments are validated, and Member checks the range it is given. Blank configuration text is JSON whitespace after an optional byte order mark, so it no longer depends on the whitespace mode. A malformed token or configuration file is reported as InvalidFormatException from loadVocabFile and readCategories, so the constructors that declare IOException keep that contract. A Unigram tokenizer.json, in which model.vocab is a list, is rejected with a message that names the layout. HexFormat decodes the \u escapes, the keyword array and the key names are constants, and unescape returns the text unchanged when there is no backslash in it.
The vocabulary and configuration paragraphs list the accepted layouts, the integer id rule, the byte order mark, the non-finite values, the Unigram exclusion, and the InvalidFormatException, without parser internals. The name finder chapter is back to the base text.
f632f55 to
ed9dd7e
Compare
|
Thanks for the update. CI is green now, and the description is refreshed. I have three points on the code. 1. Please move the Doing it properly needs more than this PR should carry. For this PR, please:
Nothing is released between the two PRs, since 2. Mark
3. Reference OPENNLP-1953 The byte order mark fix in The eval build ( |
Defer tokenizer.json support to OPENNLP-1988 and reject its layouts with an error describing the expected vocabulary format. Mark JsonScan internal and bound each escape search to the current string. Retain the OPENNLP-1953 leading BOM fix for JSON and plain-text vocabularies. Validation: the four new tokenizer rejection cases failed before the change and pass afterward. All 593 DL tests and dependency suites pass. Javadoc and manual XML checks pass. DocumentCategorizerDLEval passes eight enabled tests; its disabled GPU test is skipped. The 100,000-entry scanner probe improves from about 1,468 ms to 4 ms with the bounded search.
krickert
changed the title
2 participants
Replaces regex parsing in
AbstractDL.loadJsonVocabandDocumentCategorizerConfig.fromJsonwith a shared JSON scanner.tokenizer.json, are rejected with an error describing the expected format.id2labelobject. Escaped keys and labels are decoded, including quotes, Unicode escapes, and braces within strings.InvalidFormatException. Fractional, negative, and overflowing vocabulary IDs are rejected.JsonScanis marked@Internal(since = "3.0.0").The scanner accepts raw control characters inside strings and Python's
NaN,Infinity, and-Infinityin skipped values. Configuration whitespace handling is independent ofopennlp.whitespace.mode.tokenizer.jsonsupport is deferred to OPENNLP-1988 and a separate PR. The document categorizer manual describes the supported formats and error behavior.Local validation: 593 DL tests passed with no failures or skips; dependency suites passed. Javadoc generation and manual XML checks passed, with Javadoc warnings in unchanged files.
DocumentCategorizerDLEvalpassed all eight enabled tests with the nlptown BERT sentiment model, including automatic labels and concurrent inference. The GPU test is disabled and was skipped.OPENNLP-1931