spaCy convert on a directory instead of a file?

[demongolem-biz]

I just wanted to know if it is possible to use the convert tool directly on a directory of conll files, instead of just a single file? I have a whole mess of them and it sure seems like working directly on a single file is not too useful for me. Input Directory -> Output Directory would be my flow.

[danieldk]

Thank you for your question! spacy convert can convert a directory of corpus files, as long as they have the same type. For example:

$ ls conllu_dir
nl_lassysmall-ud-dev.conllu
nl_lassysmall-ud-test.conllu
nl_lassysmall-ud-train.conllu
$ mkdir output_dir
$ spacy convert conllu_dir output_dir
ℹ Grouping every 1 sentences into a document.
⚠ To generate better training data, you may want to group sentences
into documents with `-n 10`.
✔ Generated output file (676 documents):
output_dir/nl_lassysmall-ud-dev.spacy
ℹ Grouping every 1 sentences into a document.
⚠ To generate better training data, you may want to group sentences
into documents with `-n 10`.
✔ Generated output file (876 documents):
output_dir/nl_lassysmall-ud-test.spacy
ℹ Grouping every 1 sentences into a document.
⚠ To generate better training data, you may want to group sentences
into documents with `-n 10`.
✔ Generated output file (5789 documents):
output_dir/nl_lassysmall-ud-train.spacy
$ ls output_dir
nl_lassysmall-ud-dev.spacy
nl_lassysmall-ud-test.spacy
nl_lassysmall-ud-train.spacy