Error thrown by `chat_template` if multiple system messages (llama.cpp / Huggingface models)

[awijshoff]

Hi,

first of all, thank you for the awesome project — really cool!

Describe the bug
I'm currently experimenting with the new @blocknote/xl-ai package, using the Vercel OpenAI-compatible provider to connect with a local llama.cpp server. I'm running smaller models from HuggingFace e. g. to summarize personal notes.

These HF-models typically include a chat_template in the tokenizer_config.json, which is used when converting to .gguf format during quantization. The default chat_template often expects alternating user/assistant messages (see examples below). Using default settings, this causes the llama.cpp server to raise an error (HTTP 500), when the request has multiple system role messages.

Examples:

• Tokenizer from Mistral-7B
• GGUF from Gemma-4B

The error seems to originate from this section of the code:
https://github.com/TypeCellOS/BlockNote/blob/main/packages/xl-ai/src/api/LLMRequest.ts#L168-L180

The comments around the code suggest the issue is known already.

I'm currently fiddling with a workaround via a custom fetch-function in the OpenAI-compatible provider. It modifies the request by merging all system messages into a single system-message before sending it to the llama-server.

Another workaround would be to start the llama-server with the flags --jinja --chat-template chatml, which seems to work — but I noticed that breaks compatibility with the default llama.cpp webUI due to BOS/EOS token issues.

To Reproduce

1. Download a GGUF model that includes a strict chat_template, such as Gemma 3B.
2. Start the llama.cpp server with:

   ./llama-server.exe --model gemma-3b.gguf --port 8000 --jinja --cache-reuse 256 --ctx-size 8192

3. Use the @blocknote/xl-ai package with the OpenAI-compatible provider @ai-sdk/openai-compatible.
4. Observe the 500 error from the server due to chat_template validation.

Misc
I'm not sure if other LLM engines such as:

• vLLM
• SGLang

also run into this issue when using the default provided tokenizer_config.json or chat_template.

• Node version:
• Package manager:
• Browser:
• I'm a sponsor and would appreciate if you could look into this sooner than later 💖

[YousefED]

Thanks! Cool to hear - what are you building?

Imo this ideally should be implemented in the AI SDK Provider. For example, I think the Anthropic provider has something similar to merge adjacent messages: https://github.com/vercel/ai/pull/2518/files.

Alternatively (easier but a bit more hacky), you could supply a custom PromptBuilder which does some post-processing

[awijshoff]

Thanks for the quick response!

I'm just tinkering with a small note-taking app to help organize my ideas and tasks. I've added /-commands for quick insertion of today's and tomorrow's dates, ! to prioritize ToDos, and # to tag items by topic (like #friends, #groceries, etc.). I really like the ability to quickly add subtasks and move unfinished tasks of today to tomorrow’s checklist.

Regarding OpenAI compatibility: it appears that OpenAI accepts multiple system messages. So, it seems that it's actually the chat_template provided by model-creators that causes incompatibility, and it's up to the developer to either pass a custom chat_template to the LLM-engine or adjust it before calling the API.
Maybe a pointer somewhere might help for people stumbling upon this issue?

I looked into the default HTML PromptBuilder, and replacing the multiple system messages with just one concatenated version in a custom PromptBuilder probably will resolve the issue.
This also allows me to use my own translated instructions and test whether the model output tends to follow the target language, or whether that impacts tool invocation, especially in smaller local models - cool!

I’ll likely still want to intercept the request using the fetch API provided by Vercel OpenAI-compatible provider, but for responses. I’ve observed that with smaller models, I can drastically improve the success-rate by checking the referenceId for a $-character and just adding if it's not there as it will often respond without $. I think this has to do with the training-data in smaller models and I'll just leave this here for anyone struggling with similar issues.

args.operations.forEach((operation) => {
  if (
    operation.referenceId &&
    !operation.referenceId.endsWith("$")
  ) {
    operation.referenceId += "$";
  }
});

[alancmclean]

Thanks for the quick response!

I'm just tinkering with a small note-taking app to help organize my ideas and tasks. I've added /-commands for quick insertion of today's and tomorrow's dates, ! to prioritize ToDos, and # to tag items by topic (like #friends, #groceries, etc.). I really like the ability to quickly add subtasks and move unfinished tasks of today to tomorrow’s checklist.

Regarding OpenAI compatibility: it appears that OpenAI accepts multiple system messages. So, it seems that it's actually the chat_template provided by model-creators that causes incompatibility, and it's up to the developer to either pass a custom chat_template to the LLM-engine or adjust it before calling the API. Maybe a pointer somewhere might help for people stumbling upon this issue?

I looked into the default HTML PromptBuilder, and replacing the multiple system messages with just one concatenated version in a custom PromptBuilder probably will resolve the issue. This also allows me to use my own translated instructions and test whether the model output tends to follow the target language, or whether that impacts tool invocation, especially in smaller local models - cool!

I’ll likely still want to intercept the request using the fetch API provided by Vercel OpenAI-compatible provider, but for responses. I’ve observed that with smaller models, I can drastically improve the success-rate by checking the referenceId for a $-character and just adding if it's not there as it will often respond without $. I think this has to do with the training-data in smaller models and I'll just leave this here for anyone struggling with similar issues.

args.operations.forEach((operation) => {
if (
operation.referenceId &&
!operation.referenceId.endsWith("$")
) {
operation.referenceId += "$";
}
});

@awijshoff Can you say a little more on where you provided this function to add the $? I haven't figured out how to get the Ai extension to work due this error.

[awijshoff]

@alancmclean yeah sure. This is how I do it currently, I've set streaming to false in order to just check the response provided:

import { createOpenAICompatible } from "@ai-sdk/openai-compatible";

const customFetch = async (input, init) => {
  // get the response from the ai to modify before passing it back to blocknote
    const response = await fetch(input, init);

  try {
    const clone = response.clone();
    const text = await clone.text();
    const json = JSON.parse(text);

    json.choices?.forEach((choice) => {
      choice.message?.tool_calls?.forEach((toolCall) => {
        try {
          const args = JSON.parse(toolCall.function?.arguments ?? "{}");
          args.operations.forEach((operation) => {
            // Check and fix referenceId if it doesn't end with $
            if (operation.referenceId && !operation.referenceId.endsWith("$")) {
              operation.referenceId += "$";
            }

           // you can add more modifications if needed
          });
        } catch {
            console.error("Failed to parse function arguments:", toolCall.function?.arguments);
        }
      });
    });
  } catch (err) {
    console.error("Error processing response:", err);
  }

  return response;
};

export const model = createOpenAICompatible({
  baseURL: "<PLACEHOLDER_URL>", // Replace with the URL
  apiKey: "", // Not need with llama.cpp
  fetch: customFetch,
})("<PLACEHOLDER_MODEL>"); // Replace with the actual model, I am using Gemma.

Since Blocknote validates it somewhere internally, it might be good UX to add a check + healing somewhere (edit: here?) directly - but then again, strictly speaking - that should be the handled by the model itself.

I also noticed that small models also get confused by the blocks-Array with block within, often naming the blocks-array block.
Often LLM engines accept something like a schema in the request, see e. g. schema-constrained JSON in llama.cpp, which might improve the success-rate even further, but I haven't come around to testing that.

[YousefED]

Often LLM engines accept something like a schema in the request, see e. g. schema-constrained JSON in llama.cpp, which might improve the success-rate even further, but I haven't come around to testing that.

The schema of the tool calls should already be passed to the LLM. Could you validate if this is the case?

[awijshoff]

Yes the schema of the tool-call is passed to the LLM engine.

Here is an example of a failed request with the response:

The error I see is the following:

Error calling LLM Error: invalid operation: Invalid block: block must be a string
at preprocess.ts:67:13
at $e (filterValidOperations.ts:41:28)
at async Nt (preprocess.ts:72:10)
at async Object.start (stream.ts:10:26)

Request:

{
	"model": "gemma-3-4b-it-Q4_K_M.gguf",
	"temperature": 0,
	"messages": [{
		"role": "system",
		"content": "You're manipulating a text document using HTML blocks. \n\n      Make sure to follow the json schema provided. When referencing ids they MUST be EXACTLY the same (including the trailing $). \n\n      List items are 1 block with 1 list item each, so block content `<ul><li>item1</li></ul>` is valid, but `<ul><li>item1</li><li>item2</li></ul>` is invalid. We'll merge them automatically.\n\n      For code blocks, you can use the `data-language` attribute on a code block to specify the language.\n\n      This is the document as an array of html blocks (the cursor is BETWEEN two blocks as indicated by cursor: true): \n[{\"id\":\"1de0fd51-039a-481a-a69b-92f249141e8b$\",\"block\":\"<p>test.</p>\"},{\"id\":\"5cbbf9b1-d306-45c5-820e-95d5c1577f27$\",\"block\":\"<p></p>\"}]\n\n      First, determine what part of the document the user is talking about. You SHOULD probably take cursor info into account if needed.\n\n      EXAMPLE: if user says \"below\" (without pointing to a specific part of the document) he / she probably indicates the block(s) after the cursor. \n\n      EXAMPLE: If you want to insert content AT the cursor position (UNLESS indicated otherwise by the user), \nthen you need `referenceId` to point to the block before the cursor with position `after` (or block below and `before`).\n\n      Prefer updating existing blocks over removing and adding (but this also depends on the user's question).\n\n      The user asks you to do the following:\n"
	}, {
		"role": "user",
		"content": "Scheibe zwei kurze Paragraphen über Hunde und Katzen."
	}],
	"tool_choice": {
		"type": "function",
		"function": {
			"name": "json"
		}
	},
	"tools": [{
		"type": "function",
		"function": {
			"name": "json",
			"description": "Respond with a JSON object.",
			"parameters": {
				"type": "object",
				"properties": {
					"operations": {
						"type": "array",
						"items": {
							"anyOf": [{
								"type": "object",
								"description": "Update a block",
								"properties": {
									"type": {
										"type": "string",
										"enum": ["update"]
									},
									"id": {
										"type": "string",
										"description": "id of block to update"
									},
									"block": {
										"$ref": "#/$defs/block"
									}
								},
								"required": ["type", "id", "block"],
								"additionalProperties": false
							}, {
								"type": "object",
								"description": "Insert new blocks",
								"properties": {
									"type": {
										"type": "string",
										"enum": ["add"]
									},
									"referenceId": {
										"type": "string",
										"description": "MUST be an id of a block in the document"
									},
									"position": {
										"type": "string",
										"enum": ["before", "after"],
										"description": "`after` to add blocks AFTER (below) the block with `referenceId`, `before` to add the block BEFORE (above)"
									},
									"blocks": {
										"items": {
											"$ref": "#/$defs/block"
										},
										"type": "array"
									}
								},
								"required": ["type", "referenceId", "position", "blocks"],
								"additionalProperties": false
							}, {
								"type": "object",
								"description": "Delete a block",
								"properties": {
									"type": {
										"type": "string",
										"enum": ["delete"]
									},
									"id": {
										"type": "string",
										"description": "id of block to delete"
									}
								},
								"required": ["type", "id"],
								"additionalProperties": false
							}]
						}
					}
				},
				"additionalProperties": false,
				"required": ["operations"],
				"$defs": {
					"block": {
						"type": "string",
						"description": "html of block (MUST be a single HTML element)"
					}
				}
			}
		}
	}]
}

Response:

{
	"choices": [{
		"finish_reason": "tool_calls",
		"index": 0,
		"message": {
			"role": "assistant",
			"content": null,
			"tool_calls": [{
				"type": "function",
				"function": {
					"name": "json",
					"arguments": "{\"operations\":[{\"type\":\"add\",\"referenceId\":\"1de0fd51-039a-481a-a69b-92f249141e8b$\",\"position\":\"after\",\"blocks\":[{\"block\":\"<p>Hunde sind bekannt für ihre Loyalität und verspielte Natur. Sie sind oft sehr anhänglich und genießen es, Zeit mit ihren Besitzern zu verbringen. Viele Rassen haben unterschiedliche Temperamente und Fähigkeiten, von energiegeladenen Hunden, die gerne rennen, bis hin zu ruhigen Hunden, die gerne kuscheln.</p>\"},{\"block\":\"<p>Katzen hingegen sind oft unabhängiger und eigenwilliger. Sie sind bekannt für ihre Sauberkeit und ihr Jagdverhalten. Viele Katzen genießen es, zu schlafen und zu spielen, und sie können sehr liebevoll mit ihren Besitzern sein, wenn sie es möchten. Es gibt eine Vielzahl von Katzenrassen, jede mit ihren eigenen einzigartigen Eigenschaften.</p>\"}]}]}"
				},
				"id": "Olpxmpdli2BsyhmYE6FJzy49hjRIgxhq"
			}]
		}
	}],
	"created": 1751020853,
	"model": "gemma-3-4b-it-Q4_K_M.gguf",
	"system_fingerprint": "b5716-d27b3ca1",
	"object": "chat.completion",
	"usage": {
		"completion_tokens": 289,
		"prompt_tokens": 1209,
		"total_tokens": 1498
	},
	"id": "chatcmpl-uNANfqEarofApjWpUgfT2o3KYvKJ9rG5",
	"timings": {
		"prompt_n": 1016,
		"prompt_ms": 75256.893,
		"prompt_per_token_ms": 74.07174507874015,
		"prompt_per_second": 13.50042447274564,
		"predicted_n": 289,
		"predicted_ms": 78479.59,
		"predicted_per_token_ms": 271.5556747404844,
		"predicted_per_second": 3.6824861088086727
	}
}

I think the tool_choice and tools make sense and are conform with function calling and the open-ai tool_choice documentation.

As far as I understand, the response format is left to the receiver?

Formatting results
A result must be a string, but the format is up to you (JSON, error codes, plain text, etc.). The model will interpret that string as needed.

Therefore I was wondering, if it makes sense to add the response format and the expected schema as described in json-mode/structured outputs.

I am new to this, so I might be totally wrong.

[alancmclean]

@awijshoff Are you using a custom promptbuilder as well?

[awijshoff]

@alancmclean Sorry for the late response. Yes I do. As suggested by Yousef I copied the defaultHTMLPromptBuilder.ts where I merged the system prompt into one block (needs more testing on my side).

However, if you copy it, you'll need getDataForPromptNoNoSelection and getDataForPromptWithSelection-functions. They are not exported by the xl-ai-package, so I copied those over too and importet them into my customHTMLPromptbuilder.js.

@YousefED The issue with the schema seems to be that for the add-method the model really likes to output objects for the string-type:

[{block: "<p>Text</p>"}, {block: "<p>Second text</p>"}]

instead of:

["<p>Text</p>", "<p>Second text</p>"]

This can be healed after receiving the response in a similar fashion as adding the missing $-character, but it is a bit of a hassle Maybe there is a better way using json-schema in the tool-call?

[YousefED]

@awijshoff you could try supplying an example to the LLM, this might help?

The functions you mention are actually exported under llmFormats.html.promptHelpers

[YousefED]

@awijshoff the array inconsistency might be resolved in this PR: #1819.

The PR will include a link for a package in a bit (pr.new). Would you mind testing to see if it resolves your issue?