Add Byte Pair Encoding (BPE) class for subword tokenization

[Cydral]

Description:

This PR introduces a new bpe_tokenizer class to Dlib, implementing the Byte Pair Encoding (BPE) algorithm for subword tokenization. The BPE tokenizer is a widely used technique in natural language processing (NLP) for handling out-of-vocabulary words and reducing vocabulary size while maintaining text representation capabilities.

Key Features:

• BPE Algorithm: Implements the BPE algorithm as described in Sennrich et al., 2016.
• Special Tokens: Supports predefined special tokens (e.g., <text>, <url>, <image>) for marking specific elements in the text.
• Training and Encoding: Provides methods for training the tokenizer on a text corpus and encoding/decoding text into subword tokens.
• Serialization: Supports saving and loading the tokenizer model and vocabulary for reuse.
• Thread-Safe: Utilizes multi-threading for efficient frequency statistics computation during training.

Usage:

dlib::bpe_tokenizer tokenizer;
tokenizer.train(corpus_text, target_vocab_size, true); // Train on a text corpus
std::vector<int> tokens = tokenizer.encode("Sample text to tokenize."); // Encode text
std::string decoded_text = tokenizer.decode(tokens); // Decode tokens back to text

[davisking]

This is a good thing to test. But use the std::ostringstream and std::istringstream so the test doesn't end up leaving files around. With the stringstream it's all just in memory. And you don't need to mess with .close() so it's simpler too.

[davisking]

Do DLIB_TEST(text == decoded) right? Don't cout anything. Need to DLIB_TEST() something for the test to do anything.

[davisking]

This is cool. Add a bpe_tokenizer_abstract.h file and put the docs in there so it's like all the other docs in dlib. I'll eventually link that into the dlib.net web page and docs and whatnot too. And use the same comment/doc style as the other parts of the library (outlined in https://dlib.net/intro.html#notation but there are tons of examples in the library)