chat template is broken when input contains emoji

[queue-min]

Hello! 👋
I have a chat template and tokenization problem when input text contains emoji.

I decoded the tokenized input text after tokenizer.applyChatTemplate and the result is as follows.

<|eot_id|><|start_header_id|>user<|end_header_id|>

🥳🥳🥳<|e<|eot_id|><|start_header_id|>istant<|e<|end_header_id|>

But it has to be

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

🥳🥳🥳<|eot_id|><|start_header_id|>assistant<|end_header_id|>

All the other emojis occur same problem after it encodes.
I got same problem when I use different model (Qwen3-0.6B)

Any comments or help would be greatly appreciated :)

[FL33TW00D]

Thanks for raising the issue!

Wrote some tests to confirm:

    func testIssue210() async throws {
        let tokenizer = try await AutoTokenizer.from(pretrained: "Qwen/Qwen3-0.6B")

        let testMessages: [[String: String]] = [
            [
                "role": "user",
                "content": "🥳🥳🥳",
            ],
        ]

        let encoded = try tokenizer.applyChatTemplate(messages: testMessages)
        let decoded = tokenizer.decode(tokens: encoded)
        print("Encoded tokens: \(encoded)")
        print("Decoded output: \(decoded)")
    }

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

messages = [
    {"role": "user", "content": "🥳🥳🥳"}
]

tokenized = tokenizer.apply_chat_template(messages)
decoded = tokenizer.decode(tokenized)

print(tokenized)
print(decoded)

Produced by Swift:

[151644, 872, 198, 145863, 145863, 145863, 27, 91, 72, 151645, 72, 151644, 11202, 198]
<|im_start|>user
🥳🥳🥳<|i<|im_end|>i<|im_start|>istant

Produced by Python:

[151644, 872, 198, 145863, 145863, 145863, 151645, 198]
<|im_start|>user
🥳🥳🥳<|im_end|>

[FL33TW00D]

This was some unicode pain.
🥳 is a UTF-16 "surrogate pair", meaning it's encoded as 2 UTF-16 code points: https://www.compart.com/en/unicode/U+1F973
0xD83E 0xDD73.

The bug is on Line 313 here.

Match range: {23, 10}
Start index: 0[any]
Limited by End index: 62[utf8]
Safe prefix end: 32[utf8]

Result so far: ["<|im_start|>", "user\n🥳🥳🥳<|i", "<|im_end|>"]

We are using index which uses graphemes to get the ending index. We are treating 🥳 as 1 position, when it should be 2 positions! This is why we go "over".

Fix comes from this SO post. Basically we need to convert from an NSRange to a native String.Range.

[queue-min]

@FL33TW00D

Thank you for the swift progress 👍

We’d like to share another case we’ve found where the chat template appears to be affected.
It seems that when \r\n appears twice in the body, the chat template is also disrupted—similar to the behavior observed with emojis.

It would be great if this case could also be taken into consideration.

We truly appreciate your support.

[pcuenca]

Hi @queue-min, do you have a reproducible snippet? Thanks!

[queue-min]

Please make a txt file with following code

text = "This is the first line.\r\nThis is the second line.\r\nThis is the third line."

with open("example_text.txt", "w", newline="") as f:
    f.write(text)

Then copy and paste the texts in the txt file and test it!
It occurs same error in my case.

[FL33TW00D]

Python version:

[151644, 872, 198, 1986, 374, 279, 1156, 1555, 3224, 1986, 374, 279, 2086, 1555, 3224, 1986, 374, 279, 4843, 1555, 13, 151645, 198, 151644, 77091, 198]
<|im_start|>user
This is the first line.
This is the second line.
This is the third line.<|im_end|>
<|im_start|>assistant

Swift version on main:

[151644, 872, 198, 1986, 374, 279, 1156, 1555, 13, 319, 1986, 374, 279, 2086, 1555, 13, 319, 1986, 374, 279, 4843, 1555, 15757, 91, 151645, 91, 151644, 82, 11202, 198]
<|im_start|>user
This is the first line.
This is the second line.
This is the third line.<|<|im_end|>|<|im_start|>sistant

Swift version on fix/210:

[151644, 872, 198, 1986, 374, 279, 1156, 1555, 13, 319, 1986, 374, 279, 2086, 1555, 13, 319, 1986, 374, 279, 4843, 1555, 13, 151645, 198, 151644, 77091, 198]
<|im_start|>user
This is the first line.
This is the second line.
This is the third line.<|im_end|>
<|im_start|>assistant

so we can confirm that the latest PR fixes both issues.

[FL33TW00D]

as #211 was merged I'll close this.

Please reraise if any further issues appear @queue-min!