OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil by krickert · Pull Request #1275 · apache/opennlp

krickert · 2026-09-07T05:19:28Z

Replaces the regular expressions of sixteen small sites with character scans and adds the helpers they share to StringUtil. Most sites give the same output as before; the ones that do not are listed below with the old and the new result. Where the two differ, the scan follows what the code was meant to do, and a test pins each corrected case.

Behavior changes

EmojiCharSequenceNormalizer (deprecated, part of the default language detector chain). Only supplementary-plane code points, U+10000 and above, are replaced by a space now. The old pattern was parsed as the range U+D83C to U+10FC00 plus a literal hyphen, so it also blanked the hyphen, the BMP characters from U+D83C to U+FFFF (fullwidth and halfwidth forms, CJK compatibility ideographs, Arabic presentation forms, the private use area, U+FE0F, and unpaired surrogates from U+D83C to U+DFFF), and it kept U+10FC01 to U+10FFFF. Output differs for U+002D, U+D83C to U+FFFF, and U+10FC01 to U+10FFFF. A run of supplementary code points still becomes one space, so CJK Extension B ideographs are still removed. Text without a supplementary code point is returned as it is. readResolve returns the singleton. The serialVersionUID equals the serialver default before and after the change, so it stays. The factory Javadoc and the manual row say what changed and that models trained with an earlier release were built with the old output. Those models keep working without retraining under a new process-wide switch, -Dopennlp.compat.mode=LEGACY (opennlp.tools.util.CompatibilityMode in opennlp-api, built like WhitespaceMode and independent of it): under LEGACY the normalizer produces the 1.x/2.x output, verified against the old pattern on every code point; CURRENT is the default. The introduction chapter documents the property; #1281 uses the same switch for its surrogate rule.

TokenSampleStream (English). Tokens are split on runs of Unicode White_Space with a fixed definition (StringUtil.splitOnUnicodeWhitespace), independent of the whitespace mode; before, split("\\s+") split on ASCII whitespace only. A leading space no longer yields an empty first token with the span (0,0). A line without a token resets the open-quote state. A token holding a letter or digit of any script is a word; before, [A-Za-z0-9] made non-ASCII words attach to the previous token as punctuation. All four are tested.

FeatureGeneratorUtil. The cp class needs one capital and one period and no other character. A.\n was cp and is ic now; the trailing terminator is only reachable through U+0085 in legacy mode. No other token changes class (all code points compared), so no POS or lemmatizer eval build is needed.

ParserTool. Brackets that follow each other are each separated: a))b gives a ) ) b, before it gave a ) )b. A line without a bracket is returned unchanged. Whitespace follows the toolkit whitespace mode, as the tokenizer that runs next does.

ConlluStream. A multiword token id must be two runs of ASCII digits without a leading zero, joined by one hyphen, with the end not smaller than the start. Any other id fails the read with an InvalidFormatException naming the id. Before, 3- threw ArrayIndexOutOfBoundsException, -4 and 3--4 threw NumberFormatException, and 4-3 and 1-2-3 were read without an error.

DownloadUtil. A blank checksum file is an IOException saying so; before, the model was rejected with "Expected: null". The hash is the first field of a Unicode whitespace split.

AD readers (ADNameSampleStream, ADPOSSampleStream, ADSentenceStream, ADSentenceSampleStream). A leading _ in a multiword name no longer yields an empty token; tags, contractions, and NER markup are separated by any whitespace of the toolkit whitespace mode, not only ASCII space; a line of non-ASCII letters is not a punctuation leaf. The training samples change for corpora with such lines. Moving these changes into #1276, which rewrites the same code, is an open decision (see below).

Rewrites without a change in output

BioCodec.extractNameType (the old regex and the scan agree on every input tried), DownloadUtil index page links, BrownCluster lexicon lines, the text_xx language code of ConlluStream, and the capital and digit features of the POS tagger and lemmatizer context generators, which stay ASCII-only because trained models depend on them.

CompatibilityMode

New in opennlp-api: CompatibilityMode with LEGACY and CURRENT, resolved from opennlp.compat.mode at startup, overridable with setActive and reset, one warning per process under LEGACY, scheduled for removal in 4.0 like the whitespace switch.

StringUtil

New helpers: isAsciiLetter, isAsciiLowerCase, isAsciiDigit, endOfAsciiDigits, isLineTerminator, indexOfLineTerminator (returns -1 when there is none). isLineTerminator is the set java.util.regex.Pattern uses; it is narrower than UnicodeWhitespace.lineBreakCodePointSet, which also includes vertical tab and form feed. isLineTerminator is used by the glob matcher of #1278 as well.

Manual

The emoji row of the language detector chapter, the bracket paragraph of the parser tool section, a CoNLL-U section in the corpora chapter, and the AD reader paragraph.

Before merge

Language detector eval build (the emoji normalizer is in the default chain).
ArvoresDeitadasEval (AD readers, tokenizer training input).
The tokenizer eval build that reads through TokenSampleStream.

Open decisions for the reviewers

Whether the AD changes move to OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276 and the emoji normalizer gets its own JIRA, as requested; the branch is prepared either way.
Rebase onto main (8 commits behind) and squash on merge.

Verification

opennlp-runtime 2718, opennlp-formats 520, opennlp-cli 60, opennlp-tools 624 tests with checkstyle, offline, -Dopennlp.forkCount=1. No new java.util.regex import, no JDK whitespace predicate.

mawiesne · 2026-09-11T07:58:58Z

@krickert If you think this PR is ready for review, please put it out of draft. I'll review once ready. Think we can make some progress, as M6 vote seems to pass and will be processed soon, so we can add on top of that.

krickert · 2026-09-11T09:55:22Z

Yes, I do think they're ready for review. Thank you! Let me know about anything you'd like to see changed.

rzo1

Little time, so here is a GPT 5.6-sol review instead for now

Request changes. Three behavior changes need preservation or an explicit compatibility decision.

Validation across the combined stack: 1,856 targeted tests, zero failures, one skipped.

rzo1 · 2026-09-15T09:36:15Z

-    if (matcher.matches()) {
-      return matcher.group(1);
+    int separator = outcome.lastIndexOf('-');
+    if (separator > 0 && isWordChars(outcome, separator + 1)) {


The previous regex rejected line terminators in the type prefix. person\u2028-start now produces a type instead of null. Please preserve that rejection.

rzo1 · 2026-09-15T09:36:15Z

+   * @return {@code true} for exactly that shape.
+   */
+  private static boolean isCapPeriod(String token) {
+    return token.length() == 2 && isCapPeriodStart(token.charAt(0)) && token.charAt(1) == '.';


A.\n previously produced cp; this now produces ic. The original $ anchor accepts a final line terminator. Please preserve the feature behavior.

rzo1 · 2026-09-15T09:36:15Z

+    int i = 0;
+    while (i < text.length()) {
+      int cp = Character.codePointAt(text, i);
+      if (cp >= LOWER_CODE_POINT && cp <= UPPER_CODE_POINT) {


The original regex also matches literal hyphens. a-b previously normalized to a b, but now remains a-b. Please preserve this behavior or explicitly document and test the compatibility change.

krickert · 2026-09-15T10:35:44Z

So some of the "corrections" are preserving regex, not doing the intent. They would create an intentional bug in the code.

The purpose of the ticket is to "de-regex" because regex overmatches and does unpredictable behavior / is slow (haha holding off on doing a regex rant here). If the regex is bugged, we're going to cover it in this ticket. We won't preserve bugs :)

I'm going to be methodical about this one, and I am in favor of writing a ton of unit tests to ensure the behavior of our code does what it is intended to do, but not to preserve the behavior if it's buggy. This is a great opportunity to harden edge cases that have gone unnoticed. I'm going to take advantage of that.

Please feel free to point out any edge cases I may want to try and they will be covered. I'm looking forward to this because we'll have deterministic behavior. I've even seen differences on regex matching between JDK versions. It hurts my eyes :)

rzo1 · 2026-09-15T10:38:24Z

Go for adding tests .

krickert · 2026-09-15T11:28:54Z

This is soooo regex -

Tests are finding more bugs - that existed in both regex and our code. Fixing all of the regex domain ones and improving the test coverage.

krickert · 2026-09-15T12:18:46Z

Poor sol - in one of the other branches your sol review suggested a regex fallback - I'm not going to do that. Fallbacks are weaksauce and by definition cause issues that go against the purpose of these tickets (and it's also a nasty habit most LLMs do to make their code pass where it should fail loudly). Instead I'll back up the decision with heavier testing. If regex is needed for that feature, we should be explicit and make it a separate feature and call out that it's a regex feature. I found one instance where we do accept patterns, I'll include a one-sentence brief as to how it is limited to the JDK char sets.

krickert · 2026-09-15T14:20:47Z

The emoji change touches default langdetect features. I'll trigger the eval build before we merge. Still working on it though - almost ready to merge.

.

rzo1 · 2026-09-15T17:23:26Z

Here are some additional comments. Def. needs an eval build for the compat changes (langdetect via EmojiCharSequenceNormalizer, AD readers / ArvoresDeitadasEval, TokenSampleStream) before this can be merged.

Blocking

Scope. This PR is not "trivial". It changes behavior in 7 components, and the AD part is rewritten again by OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276 (e.g. ADSentenceSampleStream.parseTextAndParagraph, added here, is moved to ADMetadata there). Move the AD changes into OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276 and give the emoji normalizer change its own JIRA. What stays here is the StringUtil helpers and the sites that behave exactly as before.
EmojiCharSequenceNormalizer.java:55-70. This changes the output of a class in the default langdetect chain. LanguageDetectorFactory.java:43-44 says it "stays in this chain because existing models were trained with it". Hyphens (well-known), fullwidth/halfwidth forms, CJK compatibility ideographs, Arabic presentation forms, PUA and U+FE0F now go into the n-grams. Checked on every code point: output differs for U+002D, U+D83C-U+FFFF and U+10FC01-U+10FFFF. Either keep the old output bug-for-bug in this deprecated class, or keep the fix, run the langdetect eval build before merge, and update the factory Javadoc.
EmojiCharSequenceNormalizer.java:38. The implementation changed and the static EMOJI_REGEX field was removed, but serialVersionUID is unchanged. Recompute it. Also add readResolve() returning INSTANCE, as ParagraphPreservingWhitespaceCharSequenceNormalizer does.
TokenSampleStream.java:57, :123. This is not an equivalent rewrite, and the description doesn't mention it. It changes output in four ways:
- split("\\s+") (ASCII) became WhitespaceTokenizer, so the result now depends on WhitespaceMode.
- A leading space no longer yields an empty first token with Span(0,0).
- An empty line now resets evenq.
- [A-Za-z0-9] became Character.isLetterOrDigit, so non-ASCII words no longer attach as punctuation.
Keep the old semantics, or document all four and add a test for the empty-line/evenq case.
AD readers (ADPOSSampleStream.java:165, ADSentenceStream.java:231, ADNameSampleStream.java:248/517). The training samples change: a leading _ no longer yields an empty token, NBSP/U+0085/U+2028 now separate POS tags, contractions and NER tags, and a line of non-ASCII letters is no longer a punctuation leaf. The output also depends on the global opennlp.whitespace.mode; "PR 3S" is joined under UNICODE and not under LEGACY. Run the ArvoresDeitadasEval eval build, document the mode dependency, and add LEGACY-mode tests.
Only part of the AD code is converted.
- ADSentenceStream.java:67-72: NODE/LEAF/BIZARRE_LEAF patterns still use ASCII \s, so corpora.xml:443 ("tabs and no-break spaces are handled like ordinary spaces") is false. getElement with an NBSP before F S) falls through to the bizarre-leaf branch.
- ADNameSampleStream.java:559-565: getTextID still compiles, per paragraph, the same ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* regex this PR removes from ADSentenceSampleStream, and :541-547 holds dead commented-out copies.
Either drop it or align all.
StringUtil.java:252, :267. This is new frozen public API in opennlp-api without a real need. isLineTerminator has no caller outside StringUtil and adds a second line-break definition next to UnicodeWhitespace.lineBreakCodePointSet(), which also includes VT and FF. indexOfLineTerminator has a single caller (BioCodec, from = 0, only needs a yes/no) and returns text.length() instead of -1 when nothing is found, unlike every JDK indexOf. Make both private or move them to BioCodec. If they stay public, return -1 and link the java.util.regex.Pattern line-terminator definition.
ConlluStream.
- corpora.xml:403-406 and the PR description say malformed ids were skipped before. That is wrong: on main 3- threw ArrayIndexOutOfBoundsException, -4/3--4 threw NumberFormatException, and 4-3/1-2-3 were silently accepted. Fix both texts.
- ConlluStream.java:145: postProcessContractions now throws InvalidFormatException. Add Javadoc with @throws.
- ConlluStream.java:179/187: a range whose word line is missing (1-2 with no line 2) still NPEs when index.get(id) is unboxed. That was already the case on main, but the doc now claims a damaged treebank "is reported". Throw InvalidFormatException, or drop the claim.
PR description.
- It contains chat residue and truncated sentences ("whitespacision", "row in t bracket handling", "default `Langu the eval build", and the whole "One row of the table also needs a touch…" paragraph).
- The BioCodec bullet reads like a behavior change, but there is none. Old regex vs new scan on 50M inputs: 0 differences.
- DownloadUtil "treats a blank checksum file as missing" is wrong: it fails closed with "Expected: null", as it did before.
- Emoji "also blanked unpaired surrogates" is only true for U+D83C-U+DFFF.
Rewrite it, and list the eval builds (langdetect, AD) as done before merge, not "will run".

Minor

StringUtil / DownloadUtil

StringUtil.java:156-157, :178-179: Javadoc of a generic utility names its callers ("POS tagger and lemmatizer context generators … trained models expect it"), and :198, :210 add usage advice. State only the contract.
StringUtil.java:169: containsAsciiUpperCase inlines 'A'..'Z' while containsAsciiDigit calls isAsciiDigit. Add a private isAsciiUpperCase and use it.
StringUtil.java:234, :271: the from range check is copied, and "must not be null" is repeated 6 times. Use a private helper and a constant.
StringUtilTest.java:194: one test checks null for two methods. Split it or parameterize it.
DownloadUtil.java:256-273: an ASCII hex checksum does not need the mode-dependent StringUtil.isWhitespace. Use StringUtil.splitOnUnicodeWhitespace(content) and return the first token or null; this also drops the double scan (isBlank + loop).
DownloadUtil.java:264/281: blank checksum file. Throw an explicit IOException("checksum file … is blank") and add a test in DownloadUtilCacheIntegrityTest that a blank sidecar rejects the model.
DownloadUtil.java:367-369: the "\">" literal and magic + 2. Declare ANCHOR_VALUE_END next to ANCHOR_START and use .length().
DownloadUtil.java:376: when there is no ">/</a> after an anchor, from++ rescans, which is quadratic. break instead (40k unclosed anchors: about 3 s). Output matches the old LINK_PATTERN (2M fuzz inputs, 0 differences).
DownloadUtil.java:389-403: the hand-written indexOfIgnoreCase with continue outer. Use regionMatches(true, …).

AD readers

ADNameSampleStream.java:429, :433, :441: the removed regexes are kept as comments. Remove them.
ADNameSampleStream.java:271: lexemes[lexemes.length - 1] throws AIOOBE for "" and "_" (old "".split("[_]+") gave [""]). Guard it; the manual now promises underscore-only lexemes produce no words.
ADNameSampleStream.java:377: allocates a list and an array for every lexeme. Return early when there is no _.
ADNameSampleStream.java:427, :499: matchHyphenatedToken and tagContent now accept line terminators that the old . refused (U+0085 is what byte 0x85 decodes to in ISO-8859-1 input). Reject them or pin that with tests.
ADNameSampleStream.java:503: magic 5 and a "NER:" literal. Declare NER_PREFIX and use 1 + NER_PREFIX.length(). Please also declare "-" (:345) as a constant.
ADNameSampleStream.java:377/403/427/460/480/499, ADPOSSampleStream.java:160, ADSentenceSampleStream.java:172, ADSentenceStream.java:256: no need to be static. Make them private instance methods (package-private only where tests need it).
ADNameSampleStream.java:480: isAllLetters is lettersEnd(tok, 0) == len - 1. Drop it.
ADSentenceStream.java:208-229: replacement is always '»' + punct and the text is scanned twice. Drop the parameter, add a guillemet constant, and do one pass.
ADSentenceStream.java:273-275, :323: the level is put into a String[] and read back with Integer.parseInt. Return the level as an int, or use a private record.
corpora.xml:448: "the first run of digits" is wrong; the text id must follow the leading ASCII letters and hyphens directly (see your own reject tests). :443 should say "the toolkit whitespace mode", not "Unicode whitespace".
ADNameSampleStreamTest.java:172-183: the {null,null,null} sentinel puts a reject case into the accept provider. Move "é-a" to the reject test.
ADSentenceStreamTest.java:37: testPunctuationLeaf repeats cases from the provider at :69. :104: make testFixPunctuation a @ParameterizedTest. :120: the comment "ASCII whitespace" is outdated.
ADSentenceSampleStreamTest.java:109/116: use ObjectStreamUtils.createObjectStream(lines), and assert the message rather than a bare RuntimeException. Add stream-level tests for the dropped leading _ token and for NBSP in a tag.

ConlluStream / ParserTool

ConlluStream.java:253: no need to be static.
ConlluStream.java:269: "Multiword token id runs backwards: " should be a constant like INVALID_MULTIWORD_ID.
ConlluStream.java:157 vs :254: the hyphen appears as both "-" and '-'. Declare one MULTIWORD_SEPARATOR constant.
ConlluStream.java:274-281: the Javadoc is hard to parse. Say "Returns the 2-3 lowercase ASCII letters after text_, or an empty string."
ConlluStream.java:287-291: magic 2/3. Name them. The scan matches the old text_([a-z]{2,3}) (500k fuzz inputs, 0 differences).
ConlluStream.java:239: the message still formats lang, which is always "" here. Put firstPart in it.
ConlluStreamTest.java:136: "01-02, 1, 2" pins leading zeros as valid, but the expansion then looks up "1"/"2", so ids 01/02 NPE. Remove the case or reject leading zeros.
ConlluStreamTest.java: no test covers empty-node ids (8.1). Add a read() test next to a multiword range.
ConlluStreamTest.java:126-131: use Set.of(...).contains(...). :152/166/182: extract the ByteArrayInputStream factory.
ParserTool.java:87: a StringBuilder and copy are allocated per line even without brackets, while the old replaceAll returned the input unchanged. Scan for a bracket first. Also magic + 8 and a repeated ' ' literal.
parser.xml:74-82: "any Unicode whitespace" only holds in the default WhitespaceMode. Reword it and add "Since 3.0.0". The bracket fix itself is justified: old a))b gave a ) )b.

Runtime / featuregen / emoji

NameFinderME.java:326-328: now a one-line delegate. NameFinderSequenceValidator:48-49 and BilouNameFinderSequenceValidator:48-49 still go through it, while BilouCodec already calls BioCodec.extractNameType. Point them at BioCodec, drop the delegate and NameFinderMETest:329-334.
BioCodec.java:49: the rewritten method has no Javadoc. Add @param/@return (null when there is no type), and drop the inline comment at :51.
BioCodecTest.java:270: "whatever it holds" is wrong, since line terminators are rejected. Add rows with unpaired surrogates.
TokenSampleStream.java:115: second blank line. :117-118: the Javadoc claim "attaches to the token before it" is wrong for `, --, $, (, &, #. TokenSampleStreamTest.java:99: the "ASCII" comment contradicts the code.
DefaultPOSContextGeneratorTest.java:105-120: not parameterized, and it duplicates the lemmatizer test. Use @CsvSource(token, capital, digit) and add boundary rows (@, [, /, :, Ａ, ０, U+1D400, unpaired surrogate). The features themselves are identical: all 1.1M code points, 0 differences, so no eval build is needed for POS or lemmatizer.
TokenPatternFeatureGenerator.java:102-110: containsNonLetter re-implements StringUtil.isAsciiLetter, and the name is misleading. Use the helper and rename to containsNonAsciiLetter. TokenPatternFeatureGeneratorTest.java:76: parameterize it and add café-bar and abc-123.
FeatureGeneratorUtil.java:117-119: the only divergence is a trailing line terminator (A.\n), which is only reachable in LEGACY mode with U+0085, so no eval build is needed. Add cp to the tokenFeature Javadoc list. FeatureGeneratorUtilTest.java:85-88: assert the exact class and drop the "tokens do not contain line breaks" comment.
BrownCluster.java:98-104: thin Javadoc. BrownClusterTest.java:47-62: add a leading tab and an empty middle field. Output matches the old tab split (2M fuzz inputs, 0 differences).
EmojiCharSequenceNormalizer.java:19-30: the 11-line class Javadoc says "Emoji live in that range", but BMP emoji (U+2600-27BF, U+2B50) pass through. Cut it to 2-3 lines. langdetect.xml:51-57: same problem. Condense to one sentence, mention that CJK Ext B+ is blanked, and note older models.
EmojiCharSequenceNormalizer.java:55-60: allocates even when nothing is replaced, calls codePointAt twice, and line 60 is 105 chars.
EmojiCharSequenceNormalizerTest.java:62: add U+20BB7 (CJK Ext B) so the blanking of zh/ja name characters is pinned on purpose.

Process

Rename the title to what users see, e.g. "OPENNLP-1928: Stop EmojiCharSequenceNormalizer from blanking hyphens and BMP characters; reject malformed CoNLL-U multiword ids".
The branch is 8 commits behind main (3.0.0-M6, OPENNLP-1885). Rebase, and squash the 18 commits on merge.
Verified: no new java.util.regex import, no Character.isWhitespace/isSpaceChar, no data files (no LICENSE/NOTICE impact).

Sibling PRs (#1276–#1282, all draft)

They all carry the OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil #1275 commits. Only OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276, OPENNLP-1931: Regex removal (3a/10): Read opennlp-dl vocab and config JSON with a strict scanner #1277 and OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 use the new StringUtil helpers; OPENNLP-1932: Regex removal (4/10): Match model finder wildcards with a glob matcher #1278, OPENNLP-1929: Regex removal (7/10): Split BasicContextGenerator features on a literal separator #1280 and OPENNLP-1934: Add explicit tokenizer character policies #1281 can be rebased straight onto main and made independent. OPENNLP-1935: Regex removal (10/10): Forbid java.util.regex in production code via forbiddenapis #1282 is draft correctly.
OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279's description refers to StringUtil.splitOnAsciiWhitespace, which doesn't exist; the diff uses WhitespaceTokenizer.
Please apply these to all open PRs likewise:
- version-history Javadoc and docs: OPENNLP-1929: Regex removal (7/10): Split BasicContextGenerator features on a literal separator #1280 BasicContextGenerator and machine-learning.xml "In releases before 3.0.0…"
- inline ASCII ranges: OPENNLP-1931: Regex removal (3a/10): Read opennlp-dl vocab and config JSON with a strict scanner #1277 JsonScan hex check
- arrays used as tuples: OPENNLP-1933: Regex removal (5/10): Replace per-call String.split/matches/replaceAll patterns #1279 MascIdentifiers.parseIds/splitColumns, OPENNLP-1932: Regex removal (4/10): Match model finder wildcards with a glob matcher #1278 splitClassPath, OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276 parseTextAndParagraph
- eval builds: OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276 (AD) and OPENNLP-1934: Add explicit tokenizer character policies #1281 (tokenizer)

rzo1 · 2026-09-15T18:12:32Z

Follow-up: the pre-existing issues found during review are now filed.

OPENNLP-1947: TokenizerME mutates the shared WhitespaceTokenizer.INSTANCE. That affects the new INSTANCE callers here (TokenSampleStream, AD readers).
OPENNLP-1952: ConlluStream NPE for a multiword range without its word lines (the corpora.xml "a damaged treebank is reported" point).

…in DefaultLemmatizerContextGenerator

…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.

…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.

Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.

… quirks Scans now do what the code meant instead of what the regular expression would accept: - EmojiCharSequenceNormalizer replaces only supplementary-plane code points. BMP characters, hyphens and unpaired surrogates are kept. - ParserTool separates brackets on Unicode whitespace in one pass, so brackets that follow each other are all spaced and no space is doubled. - The FeatureGeneratorUtil capital-period feature needs one capital and one period; a trailing line break no longer qualifies. - BioCodec rejects a line terminator at any position of an outcome type. - DownloadUtil accepts a checksum file with leading Unicode whitespace and treats a blank file as missing. - The AD streams split on the toolkit whitespace definition, drop empty underscore parts and judge punctuation lines by code point. - ConlluStream fails a malformed multiword token id with an InvalidFormatException instead of skipping it. StringUtil adds isAsciiLetter, isAsciiDigit, endOfAsciiDigits, isLineTerminator and indexOfLineTerminator, shared by BioCodec, ADSentenceSampleStream and ConlluStream here and needed by the related OPENNLP-1929 to OPENNLP-1935 branches. isAsciiWhitespace and splitOnAsciiWhitespace are removed; callers use the Unicode-aware StringUtil.isWhitespace and WhitespaceTokenizer. The manual documents the bracket handling of the parser tool, the CoNLL-U multiword id and language code rules, the AD reader rules and the emoji normalizer in the language detector chapter. Tests cover the accept and reject side of each scan, the Unicode plane boundaries and Unicode whitespace separators.

…cans Index pages with CRLF line endings and uppercase anchors, href values with entities or percent escapes returned as written, and checksum files with CR or CRLF line endings, an uppercase hash, or a path after the hash. Outcomes with a line terminator at either end for the BIO codec, multiword token ids with digits from other scripts, trailing whitespace or an overflowing right side for CoNLL-U, paragraph keys followed by digits from other scripts for the AD reader, CR and CRLF lexicon lines for Brown clusters, and unpaired surrogates at either end plus combining marks after a supplementary run for the emoji normalizer.

StringUtil: contract-only Javadoc, a shared null message and offset check, isAsciiLowerCase next to isAsciiLetter, and indexOfLineTerminator returns -1 when there is no terminator, with the Pattern definition and the wider UnicodeWhitespace line break set named. EmojiCharSequenceNormalizer: readResolve, a shorter class Javadoc, no allocation when the text has no supplementary code point, one codePointAt per step, and a CJK Extension B case in the test. The serialver default equals the written value on main and here, so the UID is unchanged. The English TokenSampleStream splits on Unicode White_Space with a fixed definition instead of the shared WhitespaceTokenizer, and the Javadoc and tests state the changed behavior, quote state reset included. DownloadUtil: the hash is the first field of a Unicode whitespace split, a blank checksum file is an IOException that names the file, the anchor scan ends at an unclosed anchor, and case is ignored via regionMatches. ConlluStream: a MultiwordRange record, constants for the separator and messages, leading zeros rejected, an empty node id test, and the manual states what earlier releases did with malformed ids. ParserTool returns a line without brackets unchanged. The name finder validators call BioCodec directly. Feature generator tests are parameterized with boundary cases, and the manuals for the parser tool and the language detector are corrected.

…nder LEGACY CompatibilityMode in opennlp-api selects, for the classes that corrected their output in 3.0.0, between the corrected output and the output of the 1.x/2.x releases, built like WhitespaceMode and independent of it. Under LEGACY the EmojiCharSequenceNormalizer replaces hyphens, code points from U+D83C to U+10FC00 and unpaired surrogates in that range, as the old pattern did, so language detector models trained with an earlier release keep their n-grams until they are retrained. A test compares the legacy output with the old pattern on all code points. The manual describes the property in a new section.

# Conflicts: # opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluStream.java # opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluStreamTest.java

# Conflicts: # opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/ad/ADNameSampleStreamTest.java

mawiesne · 2026-09-17T09:35:15Z

@krickert and team:
Please rebase against main, update the title of this PR and the related description to cover only what is left now.

I am confused here.

rzo1 · 2026-09-18T10:11:07Z

Closed in favor of a split, as discussed in review. The commits are regrouped (authorship kept) into:

OPENNLP-1928: Regex removal (1/10): Replace trivial patterns with character scans #1311 OPENNLP-1928, part 1: rewrites with unchanged output and the StringUtil helpers
OPENNLP-1973: Regex removal (prep): Add a CompatibilityMode switch for the earlier output #1312 OPENNLP-1973, prep: CompatibilityMode
OPENNLP-1974: Regex removal (8/10): Stop EmojiCharSequenceNormalizer from blanking hyphens and BMP characters #1313 OPENNLP-1974, part 8: EmojiCharSequenceNormalizer fix, stacked on OPENNLP-1973: Regex removal (prep): Add a CompatibilityMode switch for the earlier output #1312
OPENNLP-1975: Regex removal (9/10): Correct the output of TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil #1314 OPENNLP-1975, part 9: output corrections (TokenSampleStream, ParserTool, ConlluStream, DownloadUtil, FeatureGeneratorUtil), stacked on OPENNLP-1928: Regex removal (1/10): Replace trivial patterns with character scans #1311
AD reader changes: OPENNLP-1930: Regex removal (2/10): Parse Arvores Deitadas markup with cursor scans #1276

The other parts (#1276–#1282) should be rebased onto #1311.

Replaces the patterns in the POS tagger and lemmatizer context generators, BioCodec, BrownCluster and TokenPatternFeatureGenerator with character scans and adds the shared ASCII and line terminator helpers to StringUtil. The output of every site is unchanged. The type extraction of NameFinderME moves to BioCodec. Split out of apache#1275; the changes that alter output follow in separate parts.

The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.

Replaces the patterns in the POS tagger and lemmatizer context generators, BioCodec, BrownCluster and TokenPatternFeatureGenerator with character scans and adds the shared ASCII and line terminator helpers to StringUtil. The output of every site is unchanged. The type extraction of NameFinderME moves to BioCodec. Split out of apache#1275; the changes that alter output follow in separate parts.

The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.

…racter scans (#1311) * OPENNLP-1928: Replace trivial regular expressions with character scans Replaces the patterns in the POS tagger and lemmatizer context generators, BioCodec, BrownCluster and TokenPatternFeatureGenerator with character scans and adds the shared ASCII and line terminator helpers to StringUtil. The output of every site is unchanged. The type extraction of NameFinderME moves to BioCodec. Split out of #1275; the changes that alter output follow in separate parts. * OPENNLP-1928: Split Brown cluster lines with StringUtil.split * OPENNLP-1928: Check StringUtil arguments with requireNonNullArg instead of a message constant --------- Co-authored-by: Kristian Rickert <kristian@apache.org>

The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.

The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto current main.

The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. The corpora chapter states the reader contract. On the real corpora the sentence samples are unchanged: the verifyTrainingData hashes of ArvoresDeitadasEval stay the same for Bosque_CF_8.0 and FlorestaVirgem_CF_3.0. The tree, name, POS and chunk streams differ in 2 Bosque and 86 Floresta samples, all from the reader bug fixes above: tag fragments such as .vp and <ARGOpp no longer appear as lexemes, and the first character or the whole lexeme of a tag-less tree line is no longer lost (P: reads DP, CJT: reads CJT, and the =H==( and =H===( lines contribute ( again). Squashed from the review rounds of apache#1276 for the rebase onto current main.

…h cursor scans (#1276) * OPENNLP-1930: Parse Arvores Deitadas markup with cursor scans The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. The corpora chapter states the reader contract. On the real corpora the sentence samples are unchanged: the verifyTrainingData hashes of ArvoresDeitadasEval stay the same for Bosque_CF_8.0 and FlorestaVirgem_CF_3.0. The tree, name, POS and chunk streams differ in 2 Bosque and 86 Floresta samples, all from the reader bug fixes above: tag fragments such as .vp and <ARGOpp no longer appear as lexemes, and the first character or the whole lexeme of a tag-less tree line is no longer lost (P: reads DP, CJT: reads CJT, and the =H==( and =H===( lines contribute ( again). Squashed from the review rounds of #1276 for the rebase onto current main. * OPENNLP-1930: Close the detokenizer dictionary stream after loading DetokenizerSampleStreamFactory.createDetokenizer left the dictionary FileInputStream open, so JUnit TempDir cleanup failed on Windows for ADHyphenatedTokenTest: 'detokenizer.xml: The process cannot access the file because it is being used by another process' (27 errors, Windows jobs of run 35343354338). The dictionary reads the stream fully in its constructor, so close it there. * OPENNLP-1930: Restore the metadata reject rows and pin U+2029 The review asked to put back the dropped reject cases 'CF_1001 p=2' and 'CF 1001 p=2' (the text id follows the leading letters and hyphens directly). Also pin U+2029, which the corpora chapter documents as read like U+2028. * OPENNLP-1930: End a quoted lemma at the first matching quote at a field boundary The leaf scans searched backward from the end of the line, so a lemma extended to the last quote after which the rest still parsed, the greedy behavior of the replaced patterns: =H:n("x" M S)<tab>a') b produced lemma 'x" M S)<tab>a', no morphology, and lexeme 'b'. Now the first matching quote followed by whitespace, a secondary tag, or a closing bracket closes the field, giving lemma 'x', morphology 'M S', and lexeme 'a') b'. Later quotes are not tried, so a line that does not parse after that quote becomes a fallback leaf. Quotes inside a word remain part of the lemma, and the square bracket after some Floresta time and measurement lemmas is accepted. Red: 7 assertion failures over 247 parser cases. Passing: 500 focused AD cases. No parsed field changed for the 920,203 quoted tree-line candidates in Bosque and FlorestaVirgem. The manual documents the lemma boundary rule. * OPENNLP-1930: Update the Bosque chunker perceptron score for the corrected reader Eval build 81 of eval-tests-configurable failed one assertion on this branch: ArvoresDeitadasEval.evalPortugueseChunkerPerceptron expected 0.9631066789979492 and got 0.963736197429554. The other 14 Arvores Deitadas runs, among them the GIS, QN and naive Bayes chunker runs, passed at their expected values. The change comes from 2 Bosque node lines with tags joined by a stray separator, =====P.vp and ========N<ARGOpp. The old reader lost the first character of a tag-less tree line and put .vp and <ARGOpp into 2 sentences as words of an NP chunk. The reader on this branch reads them as tag-less lines with markup and adds no token. A dump of every Bosque chunk sample from main and from this branch (4212 samples each) shows no other difference than these 3 tokens. ADChunkSampleStreamTest.testNodeLineWithStraySeparatorAddsNoToken records that reading. Red on main: expected [que, teria, de, .] but was [que, .vp, teria, <ARGOpp, de, .]. Passes on this branch. The perceptron value is updated to the one the build got. Local runs with the same Bosque file (md5 b69a3b82c385741cd876d166878b52e8) reproduce 0.9631066789979492 on main and 0.963736197429554 on this branch.

mawiesne assigned krickert Sep 11, 2026

mawiesne added java Pull requests that update Java code tests Pull requests that add or update test code labels Sep 11, 2026

mawiesne requested review from jzonthemtn, mawiesne and rzo1 September 11, 2026 07:59

krickert marked this pull request as ready for review September 11, 2026 09:54

rzo1 previously requested changes Sep 15, 2026

View reviewed changes

krickert changed the title ~~OPENNLP-1928: Replace trivial regular expressions with explicit scans~~ OPENNLP-1928: Stop EmojiCharSequenceNormalizer from blanking hyphens and BMP characters; reject malformed CoNLL-U multiword ids Sep 16, 2026

krickert marked this pull request as draft September 16, 2026 04:02

krickert force-pushed the OPENNLP-1928-regex-removal-trivial branch from 694d650 to fece12a Compare September 16, 2026 06:01

krickert added 2 commits September 16, 2026 02:05

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks …

8af0509

…in DefaultLemmatizerContextGenerator

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks …

5a92fb7

…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.

krickert added 9 commits September 16, 2026 02:05

OPENNLP-1928: Document the scan helpers that replace the patterns

ee6a264

OPENNLP-1928: Reword three Javadoc sentences

6328d80

OPENNLP-1928: Document compatibility mode configuration and scope

1ecab10

krickert force-pushed the OPENNLP-1928-regex-removal-trivial branch from fece12a to 1ecab10 Compare September 16, 2026 06:09

krickert added 2 commits September 17, 2026 02:52

Merge Apache main into OPENNLP-1928-regex-removal-trivial

8f4d4ac

# Conflicts: # opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluStream.java # opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluStreamTest.java

Merge Apache main into OPENNLP-1928-regex-removal-trivial

28a6a64

# Conflicts: # opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/ad/ADNameSampleStreamTest.java

rzo1 changed the title ~~OPENNLP-1928: Stop EmojiCharSequenceNormalizer from blanking hyphens and BMP characters; reject malformed CoNLL-U multiword ids~~ OPENNLP-1928: Regex removal (1/8): Replace trivial patterns with character scans in StringUtil Sep 18, 2026

rzo1 closed this Sep 18, 2026

Conversation

krickert commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Behavior changes

Rewrites without a change in output

CompatibilityMode

StringUtil

Manual

Before merge

Open decisions for the reviewers

Verification

Uh oh!

mawiesne commented Sep 11, 2026

Uh oh!

krickert commented Sep 11, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

krickert commented Sep 15, 2026

Uh oh!

rzo1 commented Sep 15, 2026

Uh oh!

krickert commented Sep 15, 2026

Uh oh!

krickert commented Sep 15, 2026

Uh oh!

krickert commented Sep 15, 2026

Uh oh!

rzo1 commented Sep 15, 2026

Blocking

Minor

Sibling PRs (#1276–#1282, all draft)

Uh oh!

rzo1 commented Sep 15, 2026

Uh oh!

mawiesne commented Sep 17, 2026

Uh oh!

rzo1 commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

krickert commented Sep 7, 2026 •

edited

Loading