Skip to content

[Bug] Incorrect results with some numbers #5

@OlehOnyshchak

Description

@OlehOnyshchak

Steps

from redditscore.tokenizer import CrazyTokenizer

tokenizer = CrazyTokenizer(hashtags='split')
tokenizer.tokenize("#20yearsago")

Actual Result

['2', '0', 'y', 'e', 'a', 'r', 's', 'a', 'g', 'o']

Expected Result

['20', 'years', 'ago']

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions