Conversation
…vent parsing Share real-valued event parsing and fixed field delimiters. Preserve Unicode feature names, keep outcome-only events in the stream and validate builder records. Validation: 25 Unicode regressions failed with the broad splitter before the fix; affected-module verify, LEGACY-mode tests, Javadoc and manual builds pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces regex-based event parsing with shared parsing that follows the text format's delimiters and preserves Unicode feature names.
RealValueFileEventStreamandRealBasicEventStreamuse the same parser, with space, tab, carriage return, line feed, and form feed separating fields, matchingFileEventStreamindependently of tokenizer settings. Other characters, including no-break and ideographic spaces, remain in feature names. Leading and repeated separators produce no empty features, outcome-only records remain valid events without ending the stream, and blank records report a format error.SimpleEventStreamBuilderuses the same context delimiters, preserves slashes after the outcome separator, and validates valued contexts. Tests and the manual cover these contracts and Unicode edge cases.JIRA: OPENNLP-1989. Part 5b/10 of the regex-removal work; targets
mainindependently.Validation on Linux with JDK 25:
opennlp.whitespace.mode=LEGACY; the default mode passes in the full verification.-Dopennlp.forkCount=1 -Drat.skip=true; existing unrelated skips and Javadoc warnings remain. Hosted CI has not run for this split.