Skip to content

Is it possible to have the model ignore certain tokens? #9339

Discussion options

You must be logged in to vote

I found a work around solution by saving formatting as a custom token extension https://spacy.io/usage/processing-pipelines#description and then removing the unwanted formatting tokens in the same custom pipeline.

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by Onyoursix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat / tokenizer Feature: Tokenizer
1 participant