"error: cannot allocate memory" and crash when defining infinite recursive function

[chinggg]

Describe the bug
When defining and using infinite recursive function, latest jq release will crash/abort after around 2s with a memory allocation error.

It seems #519 already discuss about disabling recursion, but I tested two examples there, (eg. jq -n 'def f: f; f') and found them only cause timeout. But my PoC cause the program to crash.

To Reproduce

I found the original PoC when testing the strftime function, which accidentally redefined strftime to be infinite recursion.

jq -n 'def strftime(x): strftime(x); 0 | strftime("")'

a minimal example would be

❯ jq -n 'def a(x): a(x); a(0)'
jq: error: cannot allocate memory
zsh: IOT instruction (core dumped)  jq -n "def a(x): a(x); a(0)"

Building with ASAN will trigger error below

=================================================================
==447421==ERROR: AddressSanitizer: requested allocation size 0xffffffff99248fe0 (0xffffffff99249fe0 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x7f5635b62c38 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x7f56359ab2ac in jv_mem_realloc ../src/jv_alloc.c:184

Expected behavior
The program gracefully exit.

Environment (please complete the following information):

• OS and Version: Ubunu 22.04 and Arch Linux
• jq version: jq 1.7 - jq 1.8.1 and latest master commit when submitting issue b33a763

[wader]

Hi, also in #519 it's demonstrated that disable recursion would not help and there are lots of other ways to make jq use lots of memory, build huge strings, objects, arrays etc. jq: error: cannot allocate memory is jq "gracefully" exiting with error exit code when it can't allocate more memory. What would be the alternative?

With ASAN is that just ASAN handling allocation failure before jq's own abort code would run?