[feature] Watch fails when monitored JSON contains invalid Unicode surrogate escapes

[xingchenfei]

Description:

When a monitored webpage occasionally returns malformed JSON containing an invalid Unicode escape sequence, the watch fails with:

Plain Text

Invalid \uXXXX\uXXXX surrogate pair escape at line 1, column 546
For example:

JSON

{
"title": "Example \uD800"
}
The response comes from the monitored website. changedetection.io should handle this more gracefully, for example by replacing invalid surrogate escapes with U+FFFD (�) or falling back to plain-text processing.

Valid surrogate pairs, such as \uD83D\uDE00, should remain unaffected.

A possible fix would be to sanitize invalid surrogate escapes before passing the response to json.loads(), orjson.loads(), or other JSON parsers.

http://jgwalist.881881881.xyz:5008/surrogate-test.json it's a test url.

[dgtlmoon]

Can you do a curl request with -vvv to show all headers of http://jgwalist.881881881.xyz:5008/surrogate-test.json ?

[xingchenfei]

Can you do a curl request with -vvv to show all headers of http://jgwalist.881881881.xyz:5008/surrogate-test.json ?

16:48:35.193000 [0-x] * [READ] client_reset, clear readers
16:48:35.203000 [0-0] * [SETUP] added
16:48:35.264000 [0-0] * [SETUP] happy eyeballing to origin jgwalist.881881881.xyz:5008
16:48:35.512000 [0-0] * Host jgwalist.881881881.xyz:5008 was resolved.
16:48:35.520000 [0-0] * IPv6: (none)
16:48:35.524000 [0-0] * IPv4: 168.107.29.190
16:48:35.530000 [0-0] * HTTPS-RR: -
16:48:35.546000 [0-0] * Trying 168.107.29.190:5008...
16:48:35.634000 [0-0] * Established connection to jgwalist.881881881.xyz (168.107.29.190 port 5008) from 192.168.12.116 port 54612
16:48:35.649000 [0-0] * [SETUP] removing connected setup filter
16:48:35.656000 [0-0] * [SETUP] destroy
16:48:35.661000 [0-0] * using HTTP/1.x
16:48:35.666000 [0-0] => Send header, 110 bytes (0x6e)
0000: GET /surrogate-test.json HTTP/1.1
0023: Host: jgwalist.881881881.xyz:5008
0046: User-Agent: curl/8.21.0
005f: Accept: /
006c:
16:48:35.691000 [0-0] * Request completely sent off
16:48:35.746000 [0-0] <= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
16:48:35.756000 [0-0] * [WRITE] [OUT] wrote 17 header bytes -> 17
16:48:35.766000 [0-0] * [WRITE] [PAUSE] writing 17/17 bytes of type c -> 0
16:48:35.775000 [0-0] * [WRITE] download_write header(type=c, blen=17) -> 0
16:48:35.784000 [0-0] * [WRITE] client_write(type=c, len=17) -> 0
16:48:35.794000 [0-0] <= Recv header, 15 bytes (0xf)
0000: Server: nginx
16:48:35.803000 [0-0] * [WRITE] header_collect pushed(type=1, len=15) -> 0
16:48:35.814000 [0-0] * [WRITE] [OUT] wrote 15 header bytes -> 15
16:48:35.821000 [0-0] * [WRITE] [PAUSE] writing 15/15 bytes of type 4 -> 0
16:48:35.830000 [0-0] * [WRITE] download_write header(type=4, blen=15) -> 0
16:48:35.839000 [0-0] * [WRITE] client_write(type=4, len=15) -> 0
16:48:35.845000 [0-0] <= Recv header, 37 bytes (0x25)
0000: Date: Thu, 23 Jul 2026 08:48:35 GMT
16:48:35.858000 [0-0] * [WRITE] header_collect pushed(type=1, len=37) -> 0
16:48:35.865000 [0-0] * [WRITE] [OUT] wrote 37 header bytes -> 37
16:48:35.874000 [0-0] * [WRITE] [PAUSE] writing 37/37 bytes of type 4 -> 0
16:48:35.881000 [0-0] * [WRITE] download_write header(type=4, blen=37) -> 0
16:48:35.888000 [0-0] * [WRITE] client_write(type=4, len=37) -> 0
16:48:35.896000 [0-0] <= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
16:48:35.905000 [0-0] * [WRITE] header_collect pushed(type=1, len=32) -> 0
16:48:35.912000 [0-0] * [WRITE] [OUT] wrote 32 header bytes -> 32
16:48:35.920000 [0-0] * [WRITE] [PAUSE] writing 32/32 bytes of type 4 -> 0
16:48:35.927000 [0-0] * [WRITE] download_write header(type=4, blen=32) -> 0
16:48:35.935000 [0-0] * [WRITE] client_write(type=4, len=32) -> 0
16:48:35.943000 [0-0] <= Recv header, 21 bytes (0x15)
0000: Content-Length: 164
16:48:35.951000 [0-0] * [WRITE] header_collect pushed(type=1, len=21) -> 0
16:48:35.958000 [0-0] * [WRITE] [OUT] wrote 21 header bytes -> 21
16:48:35.965000 [0-0] * [WRITE] [PAUSE] writing 21/21 bytes of type 4 -> 0
16:48:35.973000 [0-0] * [WRITE] download_write header(type=4, blen=21) -> 0
16:48:35.982000 [0-0] * [WRITE] client_write(type=4, len=21) -> 0
16:48:35.989000 [0-0] <= Recv header, 46 bytes (0x2e)
0000: Last-Modified: Thu, 23 Jul 2026 07:49:21 GMT
16:48:36.000000 [0-0] * [WRITE] header_collect pushed(type=1, len=46) -> 0
16:48:36.009000 [0-0] * [WRITE] [OUT] wrote 46 header bytes -> 46
16:48:36.017000 [0-0] * [WRITE] [PAUSE] writing 46/46 bytes of type 4 -> 0
16:48:36.025000 [0-0] * [WRITE] download_write header(type=4, blen=46) -> 0
16:48:36.033000 [0-0] * [WRITE] client_write(type=4, len=46) -> 0
16:48:36.040000 [0-0] <= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
16:48:36.048000 [0-0] * [WRITE] header_collect pushed(type=1, len=24) -> 0
16:48:36.057000 [0-0] * [WRITE] [OUT] wrote 24 header bytes -> 24
16:48:36.064000 [0-0] * [WRITE] [PAUSE] writing 24/24 bytes of type 4 -> 0
16:48:36.073000 [0-0] * [WRITE] download_write header(type=4, blen=24) -> 0
16:48:36.082000 [0-0] * [WRITE] client_write(type=4, len=24) -> 0
16:48:36.090000 [0-0] <= Recv header, 21 bytes (0x15)
0000: ETag: "6a61c781-a4"
16:48:36.099000 [0-0] * [WRITE] header_collect pushed(type=1, len=21) -> 0
16:48:36.107000 [0-0] * [WRITE] [OUT] wrote 21 header bytes -> 21
16:48:36.116000 [0-0] * [WRITE] [PAUSE] writing 21/21 bytes of type 4 -> 0
16:48:36.124000 [0-0] * [WRITE] download_write header(type=4, blen=21) -> 0
16:48:36.134000 [0-0] * [WRITE] client_write(type=4, len=21) -> 0
16:48:36.140000 [0-0] <= Recv header, 22 bytes (0x16)
0000: Accept-Ranges: bytes
16:48:36.150000 [0-0] * [WRITE] header_collect pushed(type=1, len=22) -> 0
16:48:36.158000 [0-0] * [WRITE] [OUT] wrote 22 header bytes -> 22
16:48:36.166000 [0-0] * [WRITE] [PAUSE] writing 22/22 bytes of type 4 -> 0
16:48:36.173000 [0-0] * [WRITE] download_write header(type=4, blen=22) -> 0
16:48:36.183000 [0-0] * [WRITE] client_write(type=4, len=22) -> 0
16:48:36.189000 [0-0] <= Recv header, 2 bytes (0x2)
0000:
16:48:36.197000 [0-0] * [WRITE] header_collect pushed(type=1, len=2) -> 0
16:48:36.205000 [0-0] * [WRITE] [OUT] wrote 2 header bytes -> 2
16:48:36.212000 [0-0] * [WRITE] [PAUSE] writing 2/2 bytes of type 4 -> 0
16:48:36.219000 [0-0] * [WRITE] download_write header(type=4, blen=2) -> 0
16:48:36.229000 [0-0] * [WRITE] client_write(type=4, len=2) -> 0
16:48:36.235000 [0-0] <= Recv data, 164 bytes (0xa4)
0000: {. "id": 123,. "title": "Unicode surrogate pair test",. "desc
0040: ription": "This JSON contains an invalid Unicode escape: \uD800"
0080: ,. "valid_emoji": "\uD83D\uDE00".}.
{
"id": 123,
"title": "Unicode surrogate pair test",
"description": "This JSON contains an invalid Unicode escape: \uD800",
"valid_emoji": "\uD83D\uDE00"
}
16:48:36.262000 [0-0] * [WRITE] [OUT] wrote 164 body bytes -> 164
16:48:36.269000 [0-0] * [WRITE] [PAUSE] writing 164/164 bytes of type 1 -> 0
16:48:36.278000 [0-0] * [WRITE] download_write body(type=1, blen=164) -> 0
16:48:36.286000 [0-0] * [WRITE] client_write(type=1, len=164) -> 0
16:48:36.294000 [0-0] * [WRITE] xfer_write_resp(len=401, eos=0) -> 0
16:48:36.304000 [0-0] * [WRITE] [OUT] done
16:48:36.308000 [0-0] * [READ] client_reset, clear readers
16:48:36.313000 [0-0] * Connection #0 to host jgwalist.881881881.xyz:5008 left intact

[ebarkhordar]

The error text is the clue here: Invalid \uXXXX\uXXXX surrogate pair escape at line 1, column N is jq's parser talking, not Python's. So this only bites jq: and jqraw: filters, and it does not depend on the response headers you asked for.

Reproduced at HEAD ebbdef78 in a clean python:3.12-slim container, through the repo's own code rather than a live fetch:

from changedetectionio import html_tools

BS = chr(92)  # build the escapes at runtime so no quoting layer eats them
cases = {
    "lone high surrogate":  '{"title": "Example ' + BS + 'uD800", "other": "untouched"}',
    "valid escape pair":    '{"title": "smile ' + BS + 'uD83D' + BS + 'uDE00", "other": "untouched"}',
    "literal emoji char":   '{"title": "smile \U0001F600", "other": "untouched"}',
}
for name, raw in cases.items():
    for f in ["json:$.other", "jq:.other", "jqraw:.other"]:
        try:
            print(name, f, "-> OK", html_tools.extract_json_as_string(content=raw, json_filter=f))
        except Exception as e:
            print(name, f, "->", f"{type(e).__name__}: {e}")

lone high surrogate  json:$.other   -> OK "untouched"
lone high surrogate  jq:.other      -> ValueError: parse error: Invalid \uXXXX\uXXXX surrogate pair escape at line 1, column 26
lone high surrogate  jqraw:.other   -> ValueError: parse error: Invalid \uXXXX\uXXXX surrogate pair escape at line 1, column 26
valid escape pair    json:$.other   -> OK "untouched"
valid escape pair    jq:.other      -> OK "untouched"
valid escape pair    jqraw:.other   -> OK untouched
literal emoji char   json:$.other   -> OK "untouched"
literal emoji char   jq:.other      -> OK "untouched"
literal emoji char   jqraw:.other   -> OK untouched

So a valid \uD83D\uDE00 pair is unaffected on all three filter types, which is the behaviour you would want to keep.

The mechanism is two parsers of different strictness in sequence:

1. html_tools.py:536 parses the document with stdlib json.loads, which accepts a lone \uD800 and hands back the character '\ud800' inside a normal Python dict.
2. _parse_json (html_tools.py:418-437) then passes that object to jq.compile(expr).input(json_data). The jq binding serializes it back to JSON text for jq's own C parser, and that parser rejects lone surrogates.

One detail from the run that may affect how you want to fix it: the filter above is .other, a field nowhere near the bad value, and it still fails. jq re-parses the whole document, so a single malformed field takes down any jq filter on that page.

The lone-surrogate sanitizer at processors/base.py:299 does not cover this. Adding print(raw.encode("utf-8", errors="replace").decode("utf-8") != raw) to the script above prints False: at that point \uD800 is still six ASCII characters in the fetched text, so the encode round trip leaves it byte for byte identical. The surrogate only comes into existence later, at the json.loads in step 1.

On whether this was a known trade-off, the only commit in the repo's history that touches surrogates is da43a175 (PR #3954, for #3952). Its scope is decoded surrogate characters arriving from mixed-encoding responses and the UnicodeEncodeError they cause in checksums, diffs and file writes. Neither it nor the jq commits (#1001, #2430) mentions JSON escape sequences, so this reads as out of scope at the time rather than a deliberate exclusion.

If you do want it handled, the point where the surrogates are addressable is between step 1 and step 2, where the data is a Python object and jq has not seen it yet. I have not written a patch, so I cannot tell you what that would cost on the json: path.