Tuning scoring in a config file, and want to specify cats_f_per_type

[Lolologist]

Hello all,

I have a two-category classifier that I want to tune to care more about positive than negative examples.
Part of my config file, as an example (I'm also doing NER, we can ignore that for now.)

[training.score_weights]
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = null
cats_macro_p = 0.30
cats_macro_r = 0.15
cats_macro_f = null
cats_macro_auc = null
cats_f_per_type = null
cats_macro_auc_per_type = null
cats_score = null
ents_f = 0.0
ents_p = 0.40
ents_r = 0.15
ents_per_type = null

Here's my results in meta.json in the model-best weights:
"cats_f_per_type":{ "positive":{ "p":1.0, "r":0.0294117647, "f":0.0571428571 }, "negative":{ "p":0.8765432099, "r":0.940397351, "f":0.9073482428 } }

As you can imagine, this seems overly tuned and not ideal.

Is there a way to specify scoring weights per type?

[adrianeboyd]

The weights here are only used for the eval display and early stopping, not within the component itself while training.

On the component config side of things, you can try adjusting threshold in the config.

You can also provide a positive_label in your config (in the [initialize.components.textcat] block) to have the overall cats_score that's used for early stopping be the positive F rather than the average of positive and negative F. You would need to add a positive weight to cats_score in the score weights above. See: https://spacy.io/api/textcategorizer#pipe (the anchor in the link looks odd, but look for the example next to TextCategorizer.initialize)

And from these results I would guess that your dataset is imbalanced? There are number of techniques you can try on the training data, like oversampling. There's nothing directly in the spacy library to support this, but you can look for general advice about dealing with imbalanced datasets.

[Lolologist]

Thank you for the speedy reply! I had imagined that the training.score_weights was a way to tune how important a given feature was, not just in how it's reported. I'll try some variations on the threshold as well as the positive_label suggestion.