Description
DocumentSplitter passes its separator to Series.str.split() without regex=False, so pandas treats multi-character separators as regular expressions:
DocumentSplitter(separator="..") on text a..b..c produces segments ['', '', '', 'c'] (regex .. matches any two chars) instead of ['a', 'b', 'c']
separator="||" splits on empty string alternation — wrong boundaries
separator="++" crashes with re.error: nothing to repeat at position 0
- the documented round-trip with
DocumentJoiner is broken, because the joiner always joins with the literal separator
Steps to reproduce
import pandas as pd
from nemo_curator.stages.text.modules import DocumentSplitter
# crash case
DocumentSplitter(separator="++")
# re.error: nothing to repeat at position 0 (raised per batch, at str.split)
# silent corruption case
pd.DataFrame({"text": ["a..b..c"]})["text"].str.split("..")
# [['', '', '', 'c']]
Expected behavior
The separator is documented as a plain string ("The separator to split the documents on") and DocumentJoiner treats it as literal — splitting should too.
Actual behavior
df[self.text_field].str.split(self.separator) — pandas only treats single-char string patterns as literal; multi-char patterns go through re.
Environment
nemo-curator main (93b48525), Python 3.12
Description
DocumentSplitterpasses its separator toSeries.str.split()withoutregex=False, so pandas treats multi-character separators as regular expressions:DocumentSplitter(separator="..")on texta..b..cproduces segments['', '', '', 'c'](regex..matches any two chars) instead of['a', 'b', 'c']separator="||"splits on empty string alternation — wrong boundariesseparator="++"crashes withre.error: nothing to repeat at position 0DocumentJoineris broken, because the joiner always joins with the literal separatorSteps to reproduce
Expected behavior
The separator is documented as a plain string ("The separator to split the documents on") and
DocumentJoinertreats it as literal — splitting should too.Actual behavior
df[self.text_field].str.split(self.separator)— pandas only treats single-char string patterns as literal; multi-char patterns go throughre.Environment
nemo-curator main (
93b48525), Python 3.12