German Entity Recognition incorrect star_char/end_char

[lsmith77]

How to reproduce the behaviour

model = spacy.load("de_core_news_lg")
# "Freundliche Grüße Nadia" (start_char: 0, end_char: 22, label_: "PER")
tokens = model("Freundliche Grüße Nadia")

# "Freundliche Grüße" (start_char: 0, end_char: 16, label_: "PER")
# "Nadia" (start_char: 32, end_char: 36, label_: MISC)
tokens = model("Freundliche Grüße meine liebste Nadia") 

# "Hallo Herr Müller" (start_char: 0, end_char: 16, label_: "PER")
tokens = model("Hallo Herr Müller.")

# no entities recognized
tokens = model("Hallo Herr Müller")

In the comments above we are examining tokens.ents.

Note for English we were in a position to reproduce the documented behavior, where it would accurately identify the entity.

That being said, in English we noticed some issues with typos, however using en_core_web_trf solved those.

Unfortunately de_dep_news_trf does not support entities.

Your Environment

• Operating System: OSX
• Python Version Used: 3.11
• spaCy Version Used: 3.5.1 (de-core-news-lg)
• Environment Information:

[lsmith77]

@DuyguA maybe this ticket interests you?

[DuyguA]

@DuyguA maybe this ticket interests you?

I had a quick look, it seems to me that the issue is greetings doesn't exist in NER datasets usually. As a result model is clueless in such lines. Also, having rest of the paragraph would help, this expressions are not full sentences anyway just a greeting and a name.

I see 2 options here:

• Make a small corpus of such cases with greeting + person names, update the NER component so make a custom model.
• Second option is to play some tricks. In the second example, I put a comma after Hallo and a period at the sentence like this: Hallo, Herr Mueller. , it worked. In the first sentence, I cut the greeting totally and the expressions is only Nadia, then recognized correctly by the model.
  This option should be quite easy, there are only a countable set of greetings is German language ... so you can implement a little preprocessor to cut the greetings maybe?
• Third option is to use the EntityRuler component, if you see a greeting, and if the next words are in capitals; most probably they're parts of a person name.

[svlandeg]

Hi all,

as this is not an actual bug but rather relates to incorrect predictions of the pretrained models (and what to do about that), I'll move this to discussions.

[lsmith77]

@DuyguA thx you for your advice

it looks like we will now most likely adopt https://github.com/flairNLP/flair

[42elenz]

This actually worked for me and german texts really well. Don't know if there are even better models out now.

[nikhilcms]

Hello,
Is it not possible to train transformer based German model for name entity recognition also ?, I can see for other languages it's supported ner but it's missing for German transformer model

[adrianeboyd]

Hi, in general it's better to start a new thread for questions like this.

We currently only provide an ner component in the trained trf pipelines for languages where we can train tagger+parser+ner on the same dataset, since otherwise ner would need to be trained separately with a second transformer component, which would make the pipeline a lot larger and slower.

If you need a German NER trf pipeline, you can create a config using the ner + GPU options from the training quickstart or spacy init config. The training dataset we use for German sm/md/lg is called WikiNER and is available under the "Source" links in the model descriptions: https://spacy.io/models/de

If you extract the first and last columns splitting on whitespace, you can convert it to the format recognized by spacy convert -c ner. If it's helpful, some example NER formats are shown in ner_example_data.