Non UTF-8 encoding throws a 500 Internal Server Error

[shabani1]

We're not checking content encoding, which can lead to a 500. Here's the simplest way to reproduce.

from lexy_py import LexyClient

lx = LexyClient()
lx.add_documents([
    {'content': '\x00'}
])

On the client, this results in LexyAPIError: 500 Internal Server Error. On the server, it results in the following:

sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) 
<class 'asyncpg.exceptions.CharacterNotInRepertoireError'>: invalid byte sequence for encoding "UTF8": 0x00
(Background on this error at: https://sqlalche.me/e/14/dbapi)

[shabani1]

Turns out it's not an encoding issue, just an issue with the null byte, which Postgres won't allow you to store in a text field.

https://stackoverflow.com/questions/1347646/postgres-error-on-insert-error-invalid-byte-sequence-for-encoding-utf8-0x0