[Jinja] Everything is an identifier

[xenova]

Turns out, ✨ anything ✨ can be an identifier, even supposed keywords. For example, the following jinja template is valid:

{% if if in in %}a{% endif %}{% set if = "a" %}{% set in = "abc" %}{% if if in in %}b{% endif %}

So, like other jinja implementations, we should rather treat all "keywords" as identifiers during lexing, and only assert semantics during parsing.

This is necessary to support the call functionality (see docs), which had conflicts with existing templates on the HF hub that happened to use call as a variable instead.

[julien-c]

🤯

[xenova]

I've made a bunch of improvements, and now @huggingface/jinja supports at least the top 100,000 valid template on the HF hub (ordered by downloads). 🥳

There were some strange cases to consider (people are very creative), but I think this is really impactful for the new chat template formatting feature (cc @mishig25).

Also cc @Rocketknight1 for viz, just highlighting the kinds of features that people are using in the wild.

[julien-c]

supports at least the top 100,000 valid template on the HF hub

this is insane 🤯

[mishig25]

lgtm !

So, like other jinja implementations, we should rather treat all "keywords" as identifiers during lexing, and only assert semantics during parsing.

do you have examples of other implementations?

[xenova]

I've now tested rendering chat templates on the top 100,000 templates, and I think I've got all the cases covered now! 🥳

Last thing: improve formatting to not require excessive bracketing (not the most important thing; might leave for a separate PR).

do you have examples of other implementations?

I'm mainly basing my implementation on the official library, and double-checking with minijinja.

[xenova]

Turns out, it was pretty simple ✅

Before:

{%- set reasoning_content = ((((((message.content).split("</think>"))[0]).rstrip("\n")).split("<think>"))[-1]).lstrip("\n") -%}

After:

{%- set reasoning_content = message.content.split("</think>")[0].rstrip("\n").split("<think>")[-1].lstrip("\n") %}

Also, I've tested that formatting doesn't affect any output for the top 100,000 templates.

[mishig25]

lgtm, amazing!

[xenova]

I think this is a good checkpoint for now ✅ Will merge and put out a new release (0.5.0).

Summary:

• Aligned usage of identifiers w/ official jinja implementation (anything can now be an identifier)

• We now support comment parsing and formatting (meaning this won't be preprocessed away when formatting). Rendering remains unchanged of course (ignore comments).

• Differentiate between Integer and Float types, which is something Python does, but JavaScript doesn't, so we needed to take special care here.

• Add support for new statement types:

  • filter: {% filter %}...{% endfilter %}
  • call: {% call %}...{% endcall %}
  • (custom) generation: {% generation %}{% endgeneration %}

• Add support for new expression types:

  • spread: {{ fn(*args) }}
  • ternary (previously we just used the if expression): {{ 1 if true else 2 }}

• Add support for new functions:

  • replace
  • strftime_now (get current time according to a narrow set of time templates, found in the wild)
  • and many others

• Reduce number of redundant brackets when formatting membership and property accesses.

• Improved binary operator precedence rules and general formatting rules.

• Fixed edge-case lexing issues

This now means we support (at least) parsing, formatting, and rendering of the top 100,000 transformers-compatible chat templates on the Hugging Face Hub 🥳