Added JS bindings for tokenizers library

[Conarnar]

Summary

Added JavaScript bindings for the tokenizer library so that we can use them to run LLMs in a web browser.

Test plan

I will add end to end tests later.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13566

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 9a8993c with merge base bbc281f ():

NEW FAILURE - The following job has failed:

• Build documentation / build (buck2) / Build doc (gh)
  At least one of the pre-conditions you specified did not hold

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[JacobSzwejbka]

I wonder if we have too many cmake options lmao.

[JacobSzwejbka]

I know you've tested this in the demo repo and the tokenizer unit tests are a bit of a pain to set up because they are a submodule.

[JacobSzwejbka]

Can you measure the codesize of this lib. I'm curious how bad it is having to individually bind each template instantiation

[Conarnar]

102K js, 1.4M wasm
Changing JsTokenizer to not be a template and only need to bind one class reduces the js to 101K.
Removing everything except SentencePiece leaves it with 99K js, 807K wasm so it seems like most of the codesize comes from including each of the different types of tokenizer.

[mergennachin]

What are the pros-and-cons between putting here vs https://github.com/meta-pytorch/tokenizers library directly?

[Conarnar]

What are the pros-and-cons between putting here vs https://github.com/meta-pytorch/tokenizers library directly?

Pros:

• Puts all the Wasm modules together.

Cons:

• Separated from the rest of the Tokenizers libraries, tests, resources, etc.

[JacobSzwejbka]

What are the pros-and-cons between putting here vs https://github.com/meta-pytorch/tokenizers library directly?

I have strong preference to have all the wasm code colocated right now since its in a more exploratory phase of development.