Amharic - training model

[yosiasz]

Greetings,

for training a model docu says the following for English

python -m spacy download en_core_web_sm

What do I need to do to generate or find an Amharic version of en_core_web_sm or how can I generate one for Amharic?

In other words: How do I generate a small Amharic pipeline trained on written web text (blogs, news, comments), that includes vocabulary, syntax and entities.

What is the minimum number of components (term ? - listed below) are required for a trained model to be acceptable for spaCy?

tagger, a lemmatizer, a parser and an entity recognizer

Thank!

@svlandeg You have guided me well in the past (4 years ago since my last commit to spaCy for Amharic and Tigrinya, how time flies!!) so if you could please provide some much needed guidance on this?

[yosiasz]

The spaCy documentation is amazing but for new language entries one gets easily lost each click. There should be some very simple step by step docu for a pedestrian user like me to at least guide one to get up and running very quickly even just to boost confidence to move forward.

1. create language support if one does not exist
   Add Amharic አማርኛ Language support #6583

2. Find or create training data (extension .conll or .conllu)
   Found one here https://github.com/UniversalDependencies/UD_Amharic-ATT
   used spacy data utilities to convert it
   python -m spacy ./am_att-ud-test.conllu ./corpus
   Here is a sample conll file
   Amharic conllu file

3. Convert conllu file to .spacy (this will create 2 spacy files to be used for training)
   python -m spacy convert ..\data\am_att-ud-train.conllu ../corpus
python -m spacy convert ..\data\am_att-ud-dev.conllu ../corpus

4. create a training config file
   spaCy training configuration
   key sections in your config file
   [paths]
   train = "../corpus/am_att-ud-train.spacy"
   dev = "../corpus/am_att-ud-dev.spacy"
   vectors = null
   init_tok2vec = null

   [system]
   gpu_allocator = null
   seed = 0

   [nlp]
   lang = "am"
   pipeline = ["tok2vec","tagger","morphologizer","trainable_lemmatizer","parser","ner"]

5. Start training
   python -W ignore -m spacy train .\config.cfg