Error message W036 entity_ruler - Train custom NER model in combination with entity ruler

[alfred-ls]

Hi! I have created a gold standard dataset with a custom model in prodigy (ner.correct PERSON/EMAIL) and I would now like to train/optimise the PERSONs and evaluate the EMAIL recognition with spaCy. In parallel, I would like to log and optimise via W&B.
I get the following error message when starting the training:

C:\Miniconda3\lib\site-packages\spacy\pipeline\entityruler.py:335: UserWarning: [W036] The component 'entity_ruler' does not have any patterns defined. warnings.warn(Warnings.W036.format(name=self.name))

The training with prodigy and the evaluation of the entity ruler with spaCy works perfectly. I don't get any error messages, but unfortunately I can't log and sweep with W&B, so I would like to build up the training process with spaCy.

I have therefore pepared the config.cfg according to the documentation and I hope I have done it correctly. I think it has something to do with the initialisation. But according to the documentation it is done automatically when I start the training. The spaCy train log also does not contain the initialisation of the entity ruler:

[2021-09-20 16:35:22,650] [INFO] Finished initializing nlp object [2021-09-20 16:35:23,245] [INFO] Initialized pipeline components: ['tok2vec', 'ner'] ✔ Initialized pipeline

What am I doing wrong?

Thanks a lot for your support!

Custom model:

nlp = spacy.load("en_core_web_md", exclude=["tagger","parser","lemmatizer","textcat","attribute_ruler"])
entity_ruler = nlp.add_pipe("entity_ruler")
patterns = [{"label": "EMAIL", "pattern": [{'LIKE_EMAIL':True}]}]
entity_ruler.add_patterns(patterns)

nlp.to_disk('custom_model_md')

---

config.cfg:

[paths]
train = "corpus/train.spacy"
dev = "corpus/dev.spacy"
vectors = "training/custom_model_md"
raw = null
init_tok2vec = null
vocab_data = null

[variables]
wandb_project_name = "NER-prodigy-logWandB"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","senter","ner","entity_ruler"]
disabled = ["senter"]
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 256
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.entity_ruler]
factory = "entity_ruler"
ent_id_sep = "||"
overwrite_ents = false
phrase_matcher_attr = null
validate = false

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.ner.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.ner.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.senter]
factory = "senter"

[components.senter.model]
@architectures = "spacy.Tagger.v1"
nO = null

[components.senter.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.senter.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 16
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [1000,500,500,500]
include_static_vectors = true

[components.senter.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 16
depth = 2
window_size = 1
maxout_pieces = 2

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode:width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
limit = 0
max_length = 0
path = ${paths:dev}
gold_preproc = false
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths:train}
max_length = 5000
gold_preproc = false
limit = 0
augmenter = null


[training]
train_corpus = "corpora.train"
dev_corpus = "corpora.dev"
seed = ${system:seed}
gpu_allocator = ${system:gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 5000
max_epochs = 0
max_steps = 0
eval_frequency = 1000
frozen_components = ["entity_ruler"]
before_to_disk = null
annotating_components = []

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.WandbLogger.v2"
project_name = ${variables.wandb_project_name}
remove_config_values = []
model_log_interval = null
log_dataset_dir = null

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 0.33
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vocab_data = ${paths.vocab_data}
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
before_init = null
after_init = null

[initialize.components]

[initialize.components.entity_ruler]

[initialize.components.entity_ruler.patterns]
@readers = "srsly.read_jsonl.v1"
path = "training/custom_model_md/entity_ruler/patterns.jsonl"


[initialize.components.ner]

[initialize.components.ner.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/ner.json"
require = false

[initialize.components.parser]

[initialize.components.parser.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/parser.json"
require = false

[initialize.components.tagger]

[initialize.components.tagger.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/tagger.json"
require = false

[initialize.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = ${nlp.lang}
tables = ["lexeme_norm"]

[initialize.tokenizer]

---

Prodigy train:
=============================== train_prodigy ===============================
Running command: 'C:\Miniconda3\python.exe' -m prodigy train training/ --ner correct_UC01_train,eval:correct_UC01_eval --config training/custom_model_md/config.cfg --base-model training/custom_model_md --verbose --gpu-id -1
ℹ Using CPU

========================= Generating Prodigy config =========================
ℹ Using config from base model
✔ Generated training config

=========================== Initializing pipeline ===========================
[2021-09-20 16:25:34,072] [INFO] Set up nlp object from config
Components: ner
Merging training and evaluation data for 1 components

• [ner] Training: 513 | Evaluation: 118 (from datasets)
  Training: 502 | Evaluation: 113
  Labels: ner (2)
• [ner] EMAIL, PERSON
  [2021-09-20 16:25:34,153] [INFO] Pipeline: ['tok2vec', 'ner', 'entity_ruler']
  [2021-09-20 16:25:34,153] [INFO] Resuming training for: ['ner', 'tok2vec']
  [2021-09-20 16:25:34,172] [DEBUG] Loading lookups from spacy-lookups-data: ['lexeme_norm']
  [2021-09-20 16:25:34,345] [INFO] Added vocab lookups: lexeme_norm
  [2021-09-20 16:25:34,345] [INFO] Created vocabulary
  [2021-09-20 16:25:35,744] [INFO] Added vectors: training/custom_model_md
  [2021-09-20 16:25:35,883] [INFO] Finished initializing nlp object
  [2021-09-20 16:25:35,883] [INFO] Initialized pipeline components: []
  ✔ Initialized pipeline

============================= Training pipeline =============================
Components: ner
Merging training and evaluation data for 1 components

• [ner] Training: 513 | Evaluation: 118 (from datasets)
  Training: 502 | Evaluation: 113
  Labels: ner (2)
• [ner] EMAIL, PERSON
  [2021-09-20 16:25:36,079] [DEBUG] Removed existing output directory: training\model-best
  [2021-09-20 16:25:36,095] [DEBUG] Removed existing output directory: training\model-last
  ℹ Pipeline: ['tok2vec', 'ner', 'entity_ruler']
  ℹ Frozen components: ['entity_ruler']
  ℹ Initial learn rate: 0.001
  E # LOSS TOK2VEC LOSS NER ENTS_F ENTS_P ENTS_R SENTS_F SENTS_P SENTS_R SCORE

---

0 0 0.00 12.23 65.52 54.29 82.61 0.00 0.00 0.00 0.16
16 1000 0.00 366.10 82.61 82.61 82.61 0.00 0.00 0.00 0.21

---

spaCy train:
================================ train_spacy ================================
Running command: 'C:\Miniconda3\python.exe' -m spacy train training/custom_model_md/config.cfg --output training/ --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --verbose --gpu-id -1
[2021-09-20 16:35:20,191] [DEBUG] Config overrides from CLI: ['paths.train', 'paths.dev']
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2021-09-20 16:35:20,378] [INFO] Set up nlp object from config
[2021-09-20 16:35:20,863] [DEBUG] Loading corpus from path: corpus\dev.spacy
[2021-09-20 16:35:20,863] [DEBUG] Loading corpus from path: corpus\train.spacy
[2021-09-20 16:35:20,863] [INFO] Pipeline: ['tok2vec', 'ner', 'entity_ruler']
[2021-09-20 16:35:20,863] [DEBUG] Loading lookups from spacy-lookups-data: ['lexeme_norm']
[2021-09-20 16:35:20,886] [INFO] Added vocab lookups: lexeme_norm
[2021-09-20 16:35:20,886] [INFO] Created vocabulary
[2021-09-20 16:35:22,516] [INFO] Added vectors: training/custom_model_md
[2021-09-20 16:35:22,650] [INFO] Finished initializing nlp object
[2021-09-20 16:35:23,245] [INFO] Initialized pipeline components: ['tok2vec', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
[2021-09-20 16:35:23,268] [DEBUG] Loading corpus from path: corpus\dev.spacy
[2021-09-20 16:35:23,269] [DEBUG] Loading corpus from path: corpus\train.spacy
[2021-09-20 16:35:23,278] [DEBUG] Removed existing output directory: training\model-best
[2021-09-20 16:35:23,287] [DEBUG] Removed existing output directory: training\model-last
ℹ Pipeline: ['tok2vec', 'ner', 'entity_ruler']
ℹ Frozen components: ['entity_ruler']
ℹ Initial learn rate: 0.001

wandb: Tracking run with wandb version 0.12.2
wandb: Syncing run splendid-frog-7
wandb: Run data is saved locally in C:\Documents\02_DEV\01_Code_Thesis\NER-prodigy-logWandB\wandb\run-20210920_163523-1pho61wk
wandb: Run wandb offline to turn off syncing.

E # LOSS TOK2VEC LOSS NER SENTS_F SENTS_P SENTS_R ENTS_F ENTS_P ENTS_R SCORE

---

C:\Miniconda3\lib\site-packages\spacy\pipeline\entityruler.py:335: UserWarning: [W036] The component 'entity_ruler' does not have any patterns defined.
warnings.warn(Warnings.W036.format(name=self.name))

C:\Miniconda3\lib\site-packages\spacy\pipeline\entityruler.py:335: UserWarning: [W036] The component 'entity_ruler' does not have any patterns defined.
warnings.warn(Warnings.W036.format(name=self.name))

[adrianeboyd]

The entity ruler in your config is being defined from scratch and it isn't reading anything from the saved custom model.

You can source the entity ruler from the saved custom model (probably the easiest change here) or you can initialize it in the [initialize] block: https://spacy.io/api/entityruler#init

[adrianeboyd]

frozen_components are not initialized, so if you remove it from frozen_components, it should work in spacy train. It shouldn't cause any problems to have it non-frozen during training because it can't be updated anyway and the training steps should skip over it.

[alfred-ls]

Thank you 👍! That was the problem.

But in general I think what I want to do doesn't work in one pipeline. If I have annotated training data (Prodigy ner correct) with different entities/labels, I cannot decide whether an entity is trained or not. To evaluate the entity ruler without creating a model for EMAIL detection, I need to build a separate pipeline like Vincent did -> https://github.com/koaning/proglang-project

My training data set looks like this:

This means that every annotated entity in the training dataset is also trained, even though an entity ruler is in the pipeline that is supposed to recognise the entity via pattern matching.
Is this statement correct?

Sorry for annoying you. I just want to make sure I've got it right, because I have quite a few use cases with exactly this requirement.

[alfred-ls]

I found a pretty good explanation about the interaction between EntityRuler and EntityRecognizer -> https://spacy.io/usage/rule-based-matching#entityruler
It helped a lot. But could you help me please with the interpretation of the training?

Due to the integrated entity_ruler with pattern matching for EMAILs I start with 90% f/p/r out of the box. Then further training cycles with annotated training data follow, sometimes better sometimes worse. How does the interaction between the two components work in this situation?

[adrianeboyd]

The basics are:

• the entity ruler has an overwrite setting that determines whether existing entity spans or new entity spans are preferred if they overlap
• the entity recognizer (ner) always preserves existing entities, meaning that it will only add entities where there is no existing entity annotation (unset or O)

The final annotation depends on the settings and the order of the components in the pipeline. You can see an example project explaining the interactions between two ner components here: https://github.com/explosion/projects/tree/v3/tutorials/ner_double

The evaluation always evaluates the final Doc at the end of the pipeline, not any intermediate steps, so you would need separate evaluations to analyze each component's performance.

Unless you use [training.annotating_components], the ner component will not see the entity ruler annotations during training.

[alfred-ls]

Super cool! Thank you for the great support, much appreciated ❤️