Skip to content

Added FFI bindings for rust lib

Choose a tag to compare

View all tags
@yethee yethee released this 31 Mar 11:10
· 18 commits to master since this release
0.9.0
6866101
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Added an alternative implementation of the encoder using tiktoken-rs library (#27).

Added

  • Added Yethee\Tiktoken\Exception\IOError class
  • Added Yethee\Tiktoken\Exception\LibError class
  • Added Yethee\Tiktoken\Encoder\LibEncoder class
  • Added Yethee\Tiktoken\Encoder\NativeEncoder class
  • Added Yethee\Tiktoken\Vocab\VocabLoader::loadFile() method

Changed

  • Yethee\Tiktoken\Encoder is now an interface
  • Added argument $useLib to Yethee\Tiktoken\EncoderProvider constructor 
    - public function __construct()
+ public function __construct(private bool $useLib = false)
  • [BC break] You can no longer disable cache for vocabs.
    Changes signature of EncoderProvider::setVocabCache() method: 
    - public function setVocabCache(string|null $cacheDir): void
+ public function setVocabCache(string $cacheDir): void
Assets 2
Loading