Saving a Custom Tokenizer to Disk

[wjbmattingly]

Hello all,

I could use a bit of advice. I am trying to add a pipe that does preprocessing on texts so that the tokenization separates parentheses and brackets appropriately when they are attached to a token incorrectly.

Here is my solution after a bit of searching the web:

@Language.component("ReTokenizer")
class ReTokenizer():
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
    def __call__(self, string):
        return self.tokenizer(string.replace("[", " [").replace("(", " ("))

When I run this, everything works great. When I try and save it to disk, I receive the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-5906c05ebe38> in <module>
      5 nlp.meta["description"] = "This is a pipeline of heuristics to find scientific names for plants."
      6 nlp.meta['url'] = "www.wjbmattingly.com"
----> 7 nlp.to_disk("plants_models")

c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages\spacy\language.py in to_disk(self, path, exclude)
   1910             serializers[name] = lambda p, proc=proc: proc.to_disk(p, exclude=["vocab"])
   1911         serializers["vocab"] = lambda p: self.vocab.to_disk(p, exclude=exclude)
-> 1912         util.to_disk(path, serializers, exclude)
   1913 
   1914     def from_disk(

c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages\spacy\util.py in to_disk(path, writers, exclude)
   1210         # Split to support file names like meta.json
   1211         if key.split(".")[0] not in exclude:
-> 1212             writer(path / key)
   1213     return path
   1214 

c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages\spacy\language.py in <lambda>(p)
   1898         path = util.ensure_path(path)
   1899         serializers = {}
-> 1900         serializers["tokenizer"] = lambda p: self.tokenizer.to_disk(
   1901             p, exclude=["vocab"]
   1902         )

AttributeError: 'ReTokenizer' object has no attribute 'to_disk'

Is there an easy solution here that I am just missing?

Thanks for any help you all can provide,

William

[polm]

I'm confused as to what you're doing here. Components take and return Docs, but you seem to be taking a string. That's normal for the tokenizer, but the tokenizer isn't decorated with Language.component and isn't put in the pipeline. How are you actually using this?

As far as serializing your component, you should look at the section on serializing components in the docs. Since your component is actually stateless, I think if you decorate a function that instantiates it rather than a class it should work.

Also, backing up a little, I don't think you need a component here at all. You should just be able to use tokenizer exceptions to handle brackets the way you want.

[wjbmattingly]

Thanks for the prompt reply, polm! As always, your advice is great. I was doing this after working on different things for 12 hours and was not thinking clearly. You were absolutely right to point out the oddity of using the component decorator. I just wrote in the tokenizer exceptions and viola! It worked. Packaged successfully.

Thanks again!!

William