Segfault in lj_debug_funcname

[kgs]

I am trying to use this module in high-load environment but unfortunately, from time to time I get segfaults :/

Every coredump I've catched segfaults here:

#0  0x00001490360ba710 in lj_debug_funcname (L=L@entry=0x147f369c6b48, frame=frame@entry=0x147f375efe40, name=name@entry=0x7ffefab28f78) at lj_debug.c:323
#1  0x00001490360bad34 in lj_debug_getinfo (L=L@entry=0x147f369c6b48, what=0x4c580b "nl", what@entry=0x4c580a "Snl", ar=ar@entry=0x7ffefab28f70, ext=ext@entry=0) at lj_debug.c:497
#2  0x00001490360bb027 in lua_getinfo (L=L@entry=0x147f369c6b48, what=what@entry=0x4c580a "Snl", ar=ar@entry=0x7ffefab28f70) at lj_debug.c:542
#3  0x0000000000494a03 in ngx_http_lua_thread_traceback (coctx=0x1490320160a8, co=0x147f369c6b48, L=0x1) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_util.c:3005
#4  ngx_http_lua_run_thread (L=L@entry=0x149036079380, r=r@entry=0x149031f99050, ctx=ctx@entry=0x149032016080, nrets=<optimized out>, nrets@entry=0)
    at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_util.c:1502
#5  0x000000000048bae2 in ngx_http_lua_read_body_resume (r=0x149031f99050) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_req_body.c:1135
#6  0x0000000000499bd8 in ngx_http_lua_access_handler (r=0x149031f99050) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_accessby.c:100
#7  0x000000000044fead in ngx_http_core_access_phase (r=0x149031f99050, ph=0x149032a9f750) at src/http/ngx_http_core_module.c:1083
#8  0x000000000044c045 in ngx_http_core_run_phases (r=0x149031f99050) at src/http/ngx_http_core_module.c:858
#9  0x000000000045cc67 in ngx_http_do_read_client_request_body (r=r@entry=0x149031f99050) at src/http/ngx_http_request_body.c:421
#10 0x000000000045d0cf in ngx_http_read_client_request_body_handler (r=0x149031f99050) at src/http/ngx_http_request_body.c:255
#11 ngx_http_read_client_request_body_handler (r=0x149031f99050) at src/http/ngx_http_request_body.c:245
#12 0x0000000000454eb4 in ngx_http_request_handler (ev=<optimized out>) at src/http/ngx_http_request.c:2248
#13 0x0000000000441b52 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#14 0x0000000000438b07 in ngx_process_events_and_timers (cycle=cycle@entry=0x149032a85410) at src/event/ngx_event.c:242
#15 0x000000000043fcd2 in ngx_worker_process_cycle (cycle=cycle@entry=0x149032a85410, data=data@entry=0x37) at src/os/unix/ngx_process_cycle.c:750
#16 0x000000000043e517 in ngx_spawn_process (cycle=cycle@entry=0x149032a85410, proc=proc@entry=0x43fc60 <ngx_worker_process_cycle>, data=data@entry=0x37, name=name@entry=0x4bd0ec "worker process", respawn=respawn@entry=-3)
    at src/os/unix/ngx_process.c:199
#17 0x000000000043f15b in ngx_start_worker_processes (cycle=cycle@entry=0x149032a85410, n=64, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:359
#18 0x000000000044057b in ngx_master_process_cycle (cycle=0x149032a85410) at src/os/unix/ngx_process_cycle.c:131
#19 0x00000000004195a6 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382

Any ideas? Help will be much appreciated :)

I am using:

• luajit2-2.1-20231117
• lua-nginx-module-0.10.20
• nginx 1.15.8

[kingluo]

Can you print the source line of the top frame?
I looked at the code based on the line number, but there is only one return statement there.
https://github.com/openresty/luajit2/blob/v2.1-20231117/src/lj_debug.c#L323

The ngx_http_lua_thread_traceback() happens after the lua_resume(), so it's expected to inspect the lua stack successfully (in fact, the lj_debug_getinfo() already confirms it's a valid stack via some assert()s).

Can you make sure there are no problems when you don't enable ngx-lua-request-time?

Anyway, I'll test it and try to reproduce the issue later. Thank you for your feedback.

[kgs]

Well, I see exact same return statement there:

return "metamethod";

Also, there are no problems when I do not enable LD_PRELOAD env variable.
Thanks for help!

[kingluo]

@kgs I tried to do a file uploading test with the versions you mentioned, but failed to reproduce the issue, which matches my expectations.

        location / {
            access_by_lua_file {
                ngx.req.read_body()
                print("recv body ok")
                assert(false)
            }
            log_by_lua_block {
                print("openresty_request_time_us: ", ngx.ctx.openresty_request_time_us)
            }
        }

What panics in your Lua code? Or, panics inside the FFI call? And what's the stack traceback in your log file?

And did you modify the source code of those repos?

[kgs]

@kingluo in my error.log I have only:

2024/03/27 14:40:08 [alert] 14239#0: worker process 14252 exited on signal 9

I've tried to compile LuaJIT without optimizations (XCFLAGS='-DLUAJIT_ENABLE_LUA52COMPAT' CCDEBUG=-g CCOPT=-O0) and this time I have stacktrace that makes a bit more sense:

#0  0x000014dbe82c5f75 in lj_debug_funcname (L=0x14caeaaae848, frame=0x14caeab0e058, name=0x7ffef5b5c0f8) at lj_debug.c:310
#1  0x000014dbe82c6a5a in lj_debug_getinfo (L=0x14caeaaae848, what=0x4c580b "nl", ar=0x7ffef5b5c0f0, ext=0) at lj_debug.c:497
#2  0x000014dbe82c6ed7 in lua_getinfo (L=0x14caeaaae848, what=0x4c580a "Snl", ar=0x7ffef5b5c0f0) at lj_debug.c:542
#3  0x0000000000494a03 in ngx_http_lua_thread_traceback (coctx=0x14caeafe30d8, co=0x14caeaaae848, L=0x1) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_util.c:3005
#4  ngx_http_lua_run_thread (L=L@entry=0x14dbe827f380, r=r@entry=0x14dbe4ce6050, ctx=ctx@entry=0x14caeafe30b0, nrets=<optimized out>, nrets@entry=0)
    at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_util.c:1502
#5  0x000000000048bae2 in ngx_http_lua_read_body_resume (r=0x14dbe4ce6050) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_req_body.c:1135
#6  0x0000000000499bd8 in ngx_http_lua_access_handler (r=0x14dbe4ce6050) at /github/workspace/xxx/build/modules/lua-nginx-module-0.10.20/src/ngx_http_lua_accessby.c:100
#7  0x000000000044fead in ngx_http_core_access_phase (r=0x14dbe4ce6050, ph=0x14dbe4c9f850) at src/http/ngx_http_core_module.c:1083
#8  0x000000000044c045 in ngx_http_core_run_phases (r=0x14dbe4ce6050) at src/http/ngx_http_core_module.c:858
#9  0x000000000045cc67 in ngx_http_do_read_client_request_body (r=r@entry=0x14dbe4ce6050) at src/http/ngx_http_request_body.c:421
#10 0x000000000045d0cf in ngx_http_read_client_request_body_handler (r=0x14dbe4ce6050) at src/http/ngx_http_request_body.c:255
#11 ngx_http_read_client_request_body_handler (r=0x14dbe4ce6050) at src/http/ngx_http_request_body.c:245
#12 0x0000000000454eb4 in ngx_http_request_handler (ev=<optimized out>) at src/http/ngx_http_request.c:2248
#13 0x0000000000441b52 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#14 0x0000000000438b07 in ngx_process_events_and_timers (cycle=cycle@entry=0x14dbe4c85690) at src/event/ngx_event.c:242
#15 0x000000000043fcd2 in ngx_worker_process_cycle (cycle=cycle@entry=0x14dbe4c85690, data=data@entry=0x22) at src/os/unix/ngx_process_cycle.c:750
#16 0x000000000043e517 in ngx_spawn_process (cycle=cycle@entry=0x14dbe4c85690, proc=proc@entry=0x43fc60 <ngx_worker_process_cycle>, data=data@entry=0x22, name=name@entry=0x4bd0ec "worker process", respawn=respawn@entry=-3)
    at src/os/unix/ngx_process.c:199
#17 0x000000000043f15b in ngx_start_worker_processes (cycle=cycle@entry=0x14dbe4c85690, n=64, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:359
#18 0x000000000044057b in ngx_master_process_cycle (cycle=0x14dbe4c85690) at src/os/unix/ngx_process_cycle.c:131
#19 0x00000000004195a6 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382

which is this line: https://github.com/openresty/luajit2/blob/v2.1-20231117/src/lj_debug.c#L310

Any thoughts?