Sentinal tokens leaking into lemmatizer output

[Jemoka]

Describe the bug
In the Finnish TDT lemmatizer, it seems like the SOS sentinal token is leaking into the model output. Very likely the padding/truncation procedure needs to be rebuilt again for Finnish as well.

In [15]: import stanza

In [16]: nlp = stanza.Pipeline(
    ...:     lang="fi",
    ...:     processors="tokenize,pos,lemma,depparse,mwt",
    ...:     tokenize_no_ssplit=True
    ...: )
2026-04-15 09:21:23 INFO: Checking for updates to resources.json in case models have been updated.  Note: this behavior can be turned off with download_method=None or download_method=DownloadMethod.REUSE_RESOURCES
Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.11.0.json: 439kB [00:00, 49.6MB/s]
2026-04-15 09:21:23 INFO: Downloaded file to /Users/houjun/Library/Caches/stanza/1.11.0/resources/resources.json
2026-04-15 09:21:24 INFO: Loading these models for language: fi (Finnish):
============================
| Processor | Package      |
----------------------------
| tokenize  | tdt          |
| mwt       | tdt          |
| pos       | tdt_charlm   |
| lemma     | tdt_nocharlm |
| depparse  | tdt_charlm   |
============================

2026-04-15 09:21:24 INFO: Using device: cpu
2026-04-15 09:21:24 INFO: Loading: tokenize
2026-04-15 09:21:24 INFO: Loading: mwt
2026-04-15 09:21:24 INFO: Loading: pos
2026-04-15 09:21:25 INFO: Loading: lemma
2026-04-15 09:21:25 INFO: Loading: depparse
2026-04-15 09:21:25 INFO: Done loading processors!

In [17]: nlp(INPUT_TEXT)
Out[17]:
[
  [
    {
      "id": 1,
      "text": "a",
      "lemma": "a",
      "upos": "NOUN",
      "xpos": "N",
      "feats": "Abbr=Yes|Case=Nom|Number=Sing",
      "head": 3,
      "deprel": "obl",
      "start_char": 0,
      "end_char": 1
    },
    {
      "id": [
        2,
        3
      ],
      "text": "tollei",
      "start_char": 2,
      "end_char": 8
    },
    {
      "id": 2,
      "text": "<SOS>tos",
      "lemma": "<SOS>tos",
      "upos": "SYM",
      "xpos": "Symb",
      "head": 1,
      "deprel": "flat:name"
    },
    {
      "id": 3,
      "text": "ei",
      "lemma": "ei",
      "upos": "VERB",
      "xpos": "V",
      "feats": "Number=Sing|Person=3|Polarity=Neg|VerbForm=Fin|Voice=Act",
      "head": 0,
      "deprel": "root"
    },
    {
      "id": 4,
      "text": "b",
      "lemma": "b",
      "upos": "NOUN",
      "xpos": "N",
      "feats": "Abbr=Yes|Case=Nom|Number=Sing",
      "head": 3,
      "deprel": "obj",
      "start_char": 9,
      "end_char": 10,
      "misc": "SpaceAfter=No"
    }
  ]
]

In [18]: INPUT_TEXT
Out[18]: 'a tollei b'

In [19]:

[AngledLuffa]

I think this is earlier than the lemmatizer. Rather it's in the tokenizer or the MWT. Otherwise the text of the word wouldn't have been "<SOS>tos" as well

Any idea if this is a proper Finnish word split? (Aside from the of course)

[AngledLuffa]

PS I can reproduce this with a pipeline that has only tokenize,mwt

>>> pipe("a tollei b")
[
  [
    {
      "id": 1,
      "text": "a",
      "start_char": 0,
      "end_char": 1
    },
    {
      "id": [
        2,
        3
      ],
      "text": "tollei",
      "start_char": 2,
      "end_char": 8
    },
    {
      "id": 2,
      "text": "<SOS>tos"
    },
    {
      "id": 3,
      "text": "ei"
    },
    {
      "id": 4,
      "text": "b",
      "start_char": 9,
      "end_char": 10,
      "misc": "SpaceAfter=No"
    }
  ]
]

[AngledLuffa]

ahahaha that's Start Of Sentence, not ...---...

[AngledLuffa]

so if i look in the treebank, there are no words starting with "t" that have MWT, but basically any longer word that ends with "ei" is an MWT. so the model's getting kinda screwed on both ends here. it thinks it should split something that ends with "ei" but then has no idea how to split a word that starts with "t"

[Jemoka]

that makes sense, and is the usual padding token? I wonder if we want to add an augmentation thaht does like for some stuff and then default to blank, or is the best move to just not produce the here?

[AngledLuffa]

looking at the code, it is basically the same problem we've seen with other processors - they can produce values in the internal control range of the vocabulary, which obviously has undesired effects when output later on. and as i mentioned, t at the start never happens, so it's producing the unexpected output

there's actually already a mechanism to turn UNK into -inf, which makes it impossible to predict. so one would think, maybe do that here and it'll work better. however, the result goes haywire

[
  [
    {
      "id": 1,
      "text": "a",
      "start_char": 0,
      "end_char": 1
    },
    {
      "id": [
        2,
        4
      ],
      "text": "tollei",
      "start_char": 2,
      "end_char": 8
    },
    {
      "id": 2,
      "text": "ttä"
    },
    {
      "id": 3,
      "text": "eiätätätä"
    },
    {
      "id": 4,
      "text": "e"
    },
    {
      "id": 5,
      "text": "b",
      "start_char": 9,
      "end_char": 10,
      "misc": "SpaceAfter=No"
    }
  ]
]

If I let it do its thing but drop output that falls in the control range, the split is still bizarre (well, I don't know Finnish) but at least it's not producing utter garbage

[
  [
    {
      "id": 1,
      "text": "a",
      "start_char": 0,
      "end_char": 1
    },
    {
      "id": [
        2,
        3
      ],
      "text": "tollei",
      "start_char": 2,
      "end_char": 8
    },
    {
      "id": 2,
      "text": "tos"
    },
    {
      "id": 3,
      "text": "ei"
    },
    {
      "id": 4,
      "text": "b",
      "start_char": 9,
      "end_char": 10,
      "misc": "SpaceAfter=No"
    }
  ]
]

I can check that in to avoid the junk output. If this is a real Finnish word that either needs splitting or needs no splitting, the easiest way to get better results will of course be to add a few sentences to the training data

[AngledLuffa]

is the usual padding token?

No, actually, it's just literally the SOS encoded from the input but left as a possible output state in the decoder. In other words, total crap when it shows up in the output. The ideal case scenario would be to retrain all the lemmatizers so that they don't have those output channels as possibilities. Two quick & dirty hacks I tried were:

• set score[SOS] = -inf, result is even more garbagey for this one test case
• ignore output states of SOS, which at least produces reasonable but likely incorrect output (again, for this one test case)

I can even try the retraining, just to see what happens...

[Jemoka]

ignore output states of <SOS> seems good. in this case does the model put a lot of softmax weight on <SOS> (thus setting it to -inf would break?

[AngledLuffa]

did the update in the dev branch help? i was thinking about it some more and thought maybe a bit of dropout on the first character might help fix the hallucinations

also, is this word even supposed to be split in the first place?

[AngledLuffa]

alright, i've picked up "vibe annotating" as a thing. ChatGPT assures me this word "tollei" is not supposed to split here. It suggested a small handful of sentences of other words which don't split:

Älä tee tollei.
Mä en tarkoittanut sitä tollei.
Ei sitä kannata tehdä tollei.
Tällei tämä onnistuu paremmin.
Ei sitä tollei tehdä.
Älä sano sillei.
Miksi teet sillei?
Ei sitä tehdä sillei.
Älä tee tolleen.
Se näyttää tolleen paremmalta.
Ei tarvitse puhua tolleen.

It assures me that none of the words in these sample sentences are supposed to be split, and there are a few important negative examples (tollei, sillei, etc).

having said all that, ChatGPT of course sometimes makes up random crap, so if you happen to know anyone who knows enough Finnish to verify these sentences, that would be great.

---

Your MWT inventory is very clarifying. It looks like the dataset splits mostly forms where ei is genuinely fused with a preceding subordinator/pronoun/clitic-ish host:

etteivät = että + eivät
ellei = elle + ei
jollei = jo/...? + ei <----- oops, a hallucination? not particularly confidence boosting
muttei = mutta + ei
vaikken = vaikka + en
miksei = miksi + ei

and colloquial clitic clusters like:

Oonksmä
Voisiksä

But tollei, sillei, tällei, tolleen are not in that class. They are lexicalized colloquial manner/deictic adverbs, not “host + negative auxiliary.”

[Jemoka]

Sadly I don't know enough Finnish to help for annotation, but perhaps @macw would? Prof. MacWhinney—would you know if tollei (along with other forms in the post from John above) is supposed to be split into two forms or kept as one?

[macw]

I know Hungarian, but not Finnish. Some of the patterns are similar. However, Google Translate wants tollee instead of tollei. Moreover, it appears that even this form is not standard Finnish, but a shortening, so really just one word. As AngledLuffa posted and as another AI query states: The Finnish word tollee is not a standard Finnish word but rather a colloquial, spoken-language (puhekieli) abbreviation for the phrase: Tuolla lailla So I guess some unique MWT — Brian MacWhinney Teresa Heinz Professor of Cognitive Psychology, Language Technologies and Modern Languages, CMU

…

On May 10, 2026, at 7:10 AM, Houjun Liu ***@***.***> wrote: Jemoka left a comment (stanfordnlp/stanza#1562) <#1562 (comment)> Sadly I don't know enough Finnish to help for annotation, but perhaps @macw <https://github.com/macw> would? Prof. MacWhinney—would you know if tollei (along with other forms in the post from John above) is supposed to be split into two forms or kept as one? — Reply to this email directly, view it on GitHub <#1562 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7HOJUC5UIBTCZNSZ5CQL4Z63HXAVCNFSM6AAAAACX2PBHQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMJTHEYTQNZTGA>. You are receiving this because you were mentioned.