Skip to content

DocumentSplitter treats multi-character separators as regex (wrong splits, re.error crash on "++") #2374

Description

@ManoharPaturi

Description

DocumentSplitter passes its separator to Series.str.split() without regex=False, so pandas treats multi-character separators as regular expressions:

  • DocumentSplitter(separator="..") on text a..b..c produces segments ['', '', '', 'c'] (regex .. matches any two chars) instead of ['a', 'b', 'c']
  • separator="||" splits on empty string alternation — wrong boundaries
  • separator="++" crashes with re.error: nothing to repeat at position 0
  • the documented round-trip with DocumentJoiner is broken, because the joiner always joins with the literal separator

Steps to reproduce

import pandas as pd
from nemo_curator.stages.text.modules import DocumentSplitter

# crash case
DocumentSplitter(separator="++")
# re.error: nothing to repeat at position 0  (raised per batch, at str.split)

# silent corruption case
pd.DataFrame({"text": ["a..b..c"]})["text"].str.split("..")
# [['', '', '', 'c']]

Expected behavior

The separator is documented as a plain string ("The separator to split the documents on") and DocumentJoiner treats it as literal — splitting should too.

Actual behavior

df[self.text_field].str.split(self.separator) — pandas only treats single-char string patterns as literal; multi-char patterns go through re.

Environment

nemo-curator main (93b48525), Python 3.12

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions