Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)#239

l0rinc wants to merge 2 commits intoopenai:mainopenai/tiktoken:mainfrom

l0rinc:paplorinc/add-linearithmic-byte-pair-mergel0rinc/tiktoken:paplorinc/add-linearithmic-byte-pair-mergeCopy head branch name to clipboard