Acute accent in placeholder name breaks tokenization

[lutejka]

Reporting a bug?

When having an acute accent within a placeholder name, the tokenizer will fail

[unplugin-vue-i18n] 7 (error code: 7) in /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
  target message: test {tést} test
  target message path: test

  1  |  test {tést} test
   |          ^

file: /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
error during build:
Error: 7 (error code: 7) in /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
  target message: test {tést} test
  target message path: test

  1  |  test {tést} test
   |          ^

    at onError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/unplugin-vue-i18n/lib/index.mjs:688:13)
    at newOptions.onError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:156:14)
    at emitError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:231:20)
    at readTokenInPlaceholder (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:718:30)
    at readToken (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:813:20)
    at nextToken (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:860:16)
    at parseNamed (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:947:19)
    at parseMessage (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1085:37)
    at parseResource (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1133:37)
    at parse (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1148:21)
    at baseCompile (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1594:24)
    at generateMessageFunction (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:160:56)
    at enterNode (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:414:46)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:45:13)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverseNodes (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:55:13)
    at _generate$2 (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:373:24)
    at generate$2 (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:346:31)
    at transform (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/unplugin-vue-i18n/lib/index.mjs:470:56)
    at plugin.transform (file:///home/projects/vitejs-vite-cekhrigp/node_modules/unplugin/dist/index.mjs:1275:25)
    at <anonymous> (file:///home/projects/vitejs-vite-cekhrigp/node_modules/rollup/dist/es/shared/node-entry.js:25543:40)
    at <anonymous> (https://vitejsvitecekhrigp-4yzy.w-corp-staticblitz.com/blitz.96435430.js:31:29386)

Expected behavior

Having an acute accent within placeholder names is not an issue.

Reproduction

Run npm run build

https://stackblitz.com/edit/acute-accent-issue?file=src%2Flocales%2Fen.json

System Info

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 20.19.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.8.2 - /usr/local/bin/npm
    pnpm: 8.15.6 - /usr/local/bin/pnpm
  npmPackages:
    @intlify/unplugin-vue-i18n: ^1.0.1 => 1.0.1 
    @vitejs/plugin-vue: ^4.2.3 => 4.3.4 
    vite: ^4.4.9 => 4.4.9 
    vue: ^3.3.4 => 3.3.4 
    vue-i18n: ^9.3.0 => 9.3.0 
    vue-tsc: ^1.8.8 => 1.8.10

Screenshot

No response

Additional context

No response

Validations

• Read the Contributing Guidelines
• Read the Documentation
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussions

[lutejka]

No I think this is not an issue with only acute accent, but its caused by any composite character. I also tested with an emoji in the placeholder name, and got a very similar error

[unplugin-vue-i18n] 2 (error code: 2) in /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
  target message: test {🫶🏼} test
  target message path: test

  1  |  test {🫶🏼} test
   |         ^^^^

file: /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
error during build:
Error: 2 (error code: 2) in /home/projects/vitejs-vite-cekhrigp/src/locales/en.json
  target message: test {🫶🏼} test
  target message path: test

  1  |  test {🫶🏼} test
   |         ^^^^

    at onError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/unplugin-vue-i18n/lib/index.mjs:688:13)
    at newOptions.onError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:156:14)
    at emitError (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:231:20)
    at readTokenInPlaceholder (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:740:30)
    at readToken (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:813:20)
    at nextToken (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:860:16)
    at parseMessage (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1066:50)
    at parseResource (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1133:37)
    at parse (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1148:21)
    at baseCompile (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/message-compiler/dist/message-compiler.node.mjs:1594:24)
    at generateMessageFunction (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:160:56)
    at enterNode (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:414:46)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:45:13)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverse (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:49:21)
    at traverseNodes (/home/projects/vitejs-vite-cekhrigp/node_modules/jsonc-eslint-parser/lib/parser/traverse.js:55:13)
    at _generate$2 (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:373:24)
    at generate$2 (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/bundle-utils/lib/index.mjs:346:31)
    at transform (file:///home/projects/vitejs-vite-cekhrigp/node_modules/@intlify/unplugin-vue-i18n/lib/index.mjs:470:56)
    at plugin.transform (file:///home/projects/vitejs-vite-cekhrigp/node_modules/unplugin/dist/index.mjs:1275:25)
    at <anonymous> (file:///home/projects/vitejs-vite-cekhrigp/node_modules/rollup/dist/es/shared/node-entry.js:25543:40)
    at <anonymous> (https://vitejsvitecekhrigp-4yzy.w-corp-staticblitz.com/blitz.96435430.js:31:29386)

[babu-ch]

There seems to be a restriction on the characters that can be used.
The error code: 2 for emojis seems to be correct.
In this case, error code: 7 may need to be corrected to show 2 🤔

[lutejka]

Hmm I looked for any invalid chars for place holder names before creating this issue, but was not able to find them. Do you know if/where they are documented?

[lutejka]

I was looking at the tokenizer a bit more. This seems to be the conditions for a valid placeholder name

It starts with a letter or an underscore, followed by any combination of letters, digits, underscores (_), hyphens (-), or dollar signs ($).
OR
It can be digits, optionally preceded by a minus sign (-).
OR
It can be any text enclosed within single quotes (').

[babu-ch]

Thank you for your research!

I think the following measures are necessary.

1. Set error code to 2 when there is an acute accent.
2. Clearly indicate valid characters in the documentation.