Chat Templates

[stbaione]

Discussion for this here.

Different servers have options to specify a chat template, which essentially wraps the user prompt is model specific tags, so that the LLM can process and respond better.

• ChatTemplates
• Reference Code

An example of how the tags can impact the accuracy of the model can be found here.

We temporarily updated our user docs to include the tags, but it's pretty garbage to ask users to add those tags themselves.

As can be seen by the link above, SGLang defaults their server to use a ChatTemplate, based on the model that was specified at startup. We should do the same.

If a user runs a llama model, we should default to use the llama-3 chat template, unless otherwise specified at startup. As we add more chat models, we should add more ChatTemplates