ValueError("[E024] Could not find an optimal move to supervise the parser.

[michelemarzollo]

Hello,
I am trying to train a model on some data that was annotated in prodigy. When following the normal procedure, I got a strange error caused in particular by two documents that you can find attached in spacy format. Do you have any idea of what the issue could be?

Setup:
I initialized the configuration in the standard way:

spacy init config lg_ner.cfg --lang fr --pipeline ner

Then I passed the two documents as training data, and i give some other documents as development data (these do not have any influence). In this case I am only providing these two documents, but I was getting the same error with many more, and removing these documents I didn't get the error anymore.

I run

spacy train lg_ner.cfg --output bad_model --paths.train bad_doc.spacy --paths.dev dev_data.spacy 

and the output is the following:

ℹ Using CPU

=========================== Initializing pipeline ===========================
[2021-08-13 16:06:48,191] [INFO] Set up nlp object from config
[2021-08-13 16:06:48,199] [INFO] Pipeline: ['tok2vec', 'ner']
[2021-08-13 16:06:48,202] [INFO] Created vocabulary
[2021-08-13 16:06:48,202] [INFO] Finished initializing nlp object
[2021-08-13 16:06:49,210] [INFO] Initialized pipeline components: ['tok2vec', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  ------------  --------  ------  ------  ------  ------
⚠ Aborting and saving the final best model. Encountered exception:
ValueError("[E024] Could not find an optimal move to supervise the parser.
Usually, this means that the model can't be updated in a way that's valid and
satisfies the correct annotations specified in the GoldParse. For example, are
all labels added to the model? If you're training a named entity recognizer,
also make sure that none of your annotated entity spans have leading or trailing
whitespace or punctuation. You can also use the `debug data` command to validate
your JSON-formatted training data. For details, run:\npython -m spacy debug data
--help")
Traceback (most recent call last):
  File "/home/francelabs/spacy_v31/venv/bin/spacy", line 8, in <module>
    sys.exit(setup_cli())
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/cli/train.py", line 59, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/training/loop.py", line 203, in train_while_improving
    nlp.update(
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/language.py", line 1123, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy/pipeline/transition_parser.pyx", line 387, in spacy.pipeline.transition_parser.Parser.update
  File "spacy/pipeline/transition_parser.pyx", line 643, in spacy.pipeline.transition_parser.Parser._init_gold_batch
  File "spacy/pipeline/_parser_internals/transition_system.pyx", line 132, in spacy.pipeline._parser_internals.transition_system.TransitionSystem.get_oracle_sequence_from_state
ValueError: [E024] Could not find an optimal move to supervise the parser. Usually, this means that the model can't be updated in a way that's valid and satisfies the correct annotations specified in the GoldParse. For example, are all labels added to the model? If you're training a named entity recognizer, also make sure that none of your annotated entity spans have leading or trailing whitespace or punctuation. You can also use the `debug data` command to validate your JSON-formatted training data. For details, run:
python -m spacy debug data --help

These are the documents (in one spacy file): bad_docs.tar.gz

Thank you for your attention!

[adrianeboyd]

What does spacy debug data show for these docs with your config/setup?

[michelemarzollo]

It shows

Traceback (most recent call last):
  File "/home/francelabs/spacy_v31/venv/bin/spacy", line 8, in <module>
    sys.exit(setup_cli())
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/cli/debug_data.py", line 65, in debug_data_cli
    debug_data(
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/cli/debug_data.py", line 89, in debug_data
    cfg = util.load_config(config_path, overrides=config_overrides)
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/spacy/util.py", line 549, in load_config
    return config.from_disk(
  File "/home/francelabs/spacy_v31/venv/lib/python3.8/site-packages/thinc/config.py", line 454, in from_disk
    text = file_.read()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte

But the same error is obtained also with the data that works, so I couldn't get any relevant information from it. By the way, I forgot to mention it in the post.

[polm]

Getting this error even with working data suggests something is wrong even with the "good" data, and you might want to take another look at that.

Looking at the bad docs, what is the first character on the line with the text below supposed to be?

Formulaire non obligatoire disponible

The first character on that line in particular looks broken to me (I get the Unicode Replacement Character). The list that includes this item also has some strange looking bullet points (maybe emoji?):

Nom commercial et dénomination sociale

What are those characters supposed to be? I would try replacing or removing them since it doesn't look like they're significant to the meaning of your text.

As another note, your documents are kind of long - when you have problems like this, it's a good idea to split your documents into smaller chunks, like one line each, to figure out where the problem is exactly.

[adrianeboyd]

I think the config issue must be separate.

I'm pretty sure the problem with the docs is one of the issues mentioned in the original error message: you have multiple entity spans that start with a non-breaking space. The NER component includes some custom behavior related to whitespace and it doesn't work with spans that start or end with whitespace.

[gun-hash]

thanks. that helped.