OpenSIPS version you are running
version: opensips 3.5.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 331e24a2c
main.c compiled on 00:00:00 Jun 25 2025 with cc 11
Note: I verified modules/cgrates/cgrates_common.c is identical across the 3.5, 3.6 and master (4.0) branches, so although this was observed on 3.5.6, the affected code is the same on current 4.0.
Describe the bug
A CGRateS engine that pushes an unsolicited request to OpenSIPS over a reactor-registered cgrates connection (for example SessionSv1.DisconnectSession, sent when a prepaid account goes negative mid-call) can drive an OpenSIPS worker into a permanent 100% CPU spin. The worker stops processing SIP but still answers MI, so it appears alive. Over time multiple workers enter this state independently and the instance effectively dies.
The spin is in cgrc_async_read (modules/cgrates/cgrates_common.c). On the listen path (cgrates_async_resume_req calls it with f=NULL), after a pushed request has been fully read and processed, the next read() on the level-triggered fd returns EAGAIN, and this block busy-loops on it instead of yielding to the reactor:
try_again:
bytes_read = read(c->fd, buffer, CGR_BUFFER_SIZE);
if (bytes_read < 0) {
if (errno == EINTR || errno == EAGAIN)
goto try_again;
EINTR retrying is correct; EAGAIN means "nothing more to read right now" and should return control to epoll.
I diffed modules/cgrates/cgrates_common.c across the 3.5, 3.6 and master (4.0) branches. The file is byte-identical on all three except for a single cosmetic change (an ASYNC_SET_RESUME_F macro wrapping the same resume-function assignment). cgrc_async_read itself, including the try_again EAGAIN loop, is unchanged. So this bug is present and unfixed on 3.5, 3.6 and 4.0 alike, and upgrading does not resolve it.
To Reproduce
- OpenSIPS using the cgrates module for accounting (
cgrates_acc) with a persistent connection to the engine's SessionS on port 2014.
- Engine doing prepaid rating with
AllowNegative:false, so it issues SessionSv1.DisconnectSession back over the same connection when balance is exhausted.
- Run prepaid traffic where accounts reach zero balance mid-call.
- A worker begins spinning at 100% CPU and never recovers.
Expected behavior
When read() returns EAGAIN on the listen path, cgrc_async_read should return and let the reactor re-arm the fd, rather than looping on the syscall. The worker should continue processing SIP normally after handling an engine-initiated push.
Proposed fix (splits EINTR from EAGAIN):
if (bytes_read < 0) {
if (errno == EINTR)
goto try_again;
else if (errno == EAGAIN || errno == EWOULDBLOCK) {
async_status = ASYNC_DONE;
return final_ret;
}
else if (errno == ECONNRESET) {
...
Returning ASYNC_DONE lets cgrates_async_resume_req re-arm the fd normally. Partial replies are already handled earlier by the json_tokener_continue → ASYNC_CONTINUE branch before any second read, so the only path reaching a second read() is after a complete parse, where yielding on EAGAIN is correct.
Relevant System Logs
Backtrace of a spinning worker (captured live, reproduced on three separate occasions):
#0 read () from /lib64/libc.so.6
#1 cgrc_async_read () from cgrates.so
#2 cgrates_async_resume_req () from cgrates.so
#3 async_fd_resume ()
#4 handle_io.constprop ()
#5 io_wait_loop_epoll.constprop ()
#6 udp_start_processes ()
#7 main ()
strace -c on the worker (5-second sample): ~279000 read() calls, all returning EAGAIN, ~0.8s total in-syscall, no other syscalls. /proc/<pid>/syscall alternates between read on the engine fd and userspace. ss shows a few unread bytes stuck in the connection's Recv-Q.
OS/environment information
- Operating System: CentOS Stream 9
- OpenSIPS installation: manual packages (yum.opensips.org)
- other relevant information: cgrates module, JSON-RPC to engine SessionS on :2014; engine performs remote session disconnects on negative prepaid balance
OpenSIPS version you are running
Note: I verified
modules/cgrates/cgrates_common.cis identical across the3.5,3.6andmaster(4.0) branches, so although this was observed on 3.5.6, the affected code is the same on current 4.0.Describe the bug
A CGRateS engine that pushes an unsolicited request to OpenSIPS over a reactor-registered cgrates connection (for example
SessionSv1.DisconnectSession, sent when a prepaid account goes negative mid-call) can drive an OpenSIPS worker into a permanent 100% CPU spin. The worker stops processing SIP but still answers MI, so it appears alive. Over time multiple workers enter this state independently and the instance effectively dies.The spin is in
cgrc_async_read(modules/cgrates/cgrates_common.c). On the listen path (cgrates_async_resume_reqcalls it withf=NULL), after a pushed request has been fully read and processed, the nextread()on the level-triggered fd returns EAGAIN, and this block busy-loops on it instead of yielding to the reactor:EINTR retrying is correct; EAGAIN means "nothing more to read right now" and should return control to epoll.
I diffed
modules/cgrates/cgrates_common.cacross the3.5,3.6andmaster(4.0) branches. The file is byte-identical on all three except for a single cosmetic change (anASYNC_SET_RESUME_Fmacro wrapping the same resume-function assignment).cgrc_async_readitself, including thetry_againEAGAIN loop, is unchanged. So this bug is present and unfixed on 3.5, 3.6 and 4.0 alike, and upgrading does not resolve it.To Reproduce
cgrates_acc) with a persistent connection to the engine's SessionS on port 2014.AllowNegative:false, so it issuesSessionSv1.DisconnectSessionback over the same connection when balance is exhausted.Expected behavior
When
read()returns EAGAIN on the listen path,cgrc_async_readshould return and let the reactor re-arm the fd, rather than looping on the syscall. The worker should continue processing SIP normally after handling an engine-initiated push.Proposed fix (splits EINTR from EAGAIN):
Returning
ASYNC_DONEletscgrates_async_resume_reqre-arm the fd normally. Partial replies are already handled earlier by thejson_tokener_continue→ASYNC_CONTINUEbranch before any second read, so the only path reaching a secondread()is after a complete parse, where yielding on EAGAIN is correct.Relevant System Logs
Backtrace of a spinning worker (captured live, reproduced on three separate occasions):
strace -con the worker (5-second sample): ~279000read()calls, all returning EAGAIN, ~0.8s total in-syscall, no other syscalls./proc/<pid>/syscallalternates betweenreadon the engine fd and userspace.ssshows a few unread bytes stuck in the connection's Recv-Q.OS/environment information