Which Transformer Model can be used in sapcy with token size more than 512 to train long sentences for NER + RELATION problem?

[karndeepsingh]

Hi,
I have been training the NER+ RELATION Model with Roberta and its token size is 512. I have sentences which has token size more than 512. How I can do the training of NER + RELATION model together with token size more than 512? Do I need to change the Model? If so then, what could be those models other than RoBerta for training long sentence that has more than 512 tokens and compatible with Spacy.

Thanks

[adrianeboyd]

By default the transformer component uses overlapping strided spans (see: https://spacy.io/api/transformer#span_getters) so you can train and predict on longer texts without issues on transformer models that have a fixed max length.

[karndeepsingh]

Awesome ! Thanks for your answer that was helpful! Just wanted to know one more thing that, in what proportion or in what ratio I should change Window size and Stride? Any guidance would be helpful. Default window size is 64 and stride is 48. Any help on should I vary these values.

Thanks

[adrianeboyd]

The defaults should be 128 and 96, I think? The main thing is that stride needs to be smaller than window_size so that there's some overlap between the spans so all tokens have a left and right context for the transformer part.

If the window size (which is in spacy tokens) is smaller than necessary to get the transformer input spans under 512 transformer tokens, it will be unnecessarily slow.