Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emits incorrect results when utf-8 codepoint is split between two chunks #23

Open
laverdet opened this issue Oct 4, 2023 · 0 comments

Comments

@laverdet
Copy link

laverdet commented Oct 4, 2023

Describe the bug

meros does not correctly decode utf-8 encoded data which is yielded in a different chunk.

You need to pass { stream: true } to TextDecoder#decode:
https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode#options

Furthermore, you may not share instances of the encoder between streams since this is a stateful object. If you share instances then multiple concurrent streams will impact each other.

To reproduce

const stream = async function*() {
	const smiley = Buffer.from("🤔");
	yield Buffer.from("\r\n---\r\n\r\n");
	yield smiley.subarray(0, 2);
	yield smiley.subarray(2);
	yield Buffer.from("\r\n-----\r\n");
}();

const chunks = await meros(new Response(stream, {
	headers: {
		"content-type": 'multipart/mixed; boundary="-"',
	},
}));

await Promise.all([
	async function() {
		for await (const chunk of chunks) {
			console.log(chunk);
		}
	}(),
]);

Output:

{ headers: {}, body: '���', json: false }

Expected behavior

It should decode correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant