Skip to content

[BUG] cgrates: cgrc_async_read spins at 100% CPU on unsolicited engine push (EAGAIN loop) #4249

Description

@abisaim

OpenSIPS version you are running

version: opensips 3.5.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 331e24a2c
main.c compiled on 00:00:00 Jun 25 2025 with cc 11

Note: I verified modules/cgrates/cgrates_common.c is identical across the 3.5, 3.6 and master (4.0) branches, so although this was observed on 3.5.6, the affected code is the same on current 4.0.

Describe the bug

A CGRateS engine that pushes an unsolicited request to OpenSIPS over a reactor-registered cgrates connection (for example SessionSv1.DisconnectSession, sent when a prepaid account goes negative mid-call) can drive an OpenSIPS worker into a permanent 100% CPU spin. The worker stops processing SIP but still answers MI, so it appears alive. Over time multiple workers enter this state independently and the instance effectively dies.

The spin is in cgrc_async_read (modules/cgrates/cgrates_common.c). On the listen path (cgrates_async_resume_req calls it with f=NULL), after a pushed request has been fully read and processed, the next read() on the level-triggered fd returns EAGAIN, and this block busy-loops on it instead of yielding to the reactor:

try_again:
	bytes_read = read(c->fd, buffer, CGR_BUFFER_SIZE);
	if (bytes_read < 0) {
		if (errno == EINTR || errno == EAGAIN)
			goto try_again;

EINTR retrying is correct; EAGAIN means "nothing more to read right now" and should return control to epoll.

I diffed modules/cgrates/cgrates_common.c across the 3.5, 3.6 and master (4.0) branches. The file is byte-identical on all three except for a single cosmetic change (an ASYNC_SET_RESUME_F macro wrapping the same resume-function assignment). cgrc_async_read itself, including the try_again EAGAIN loop, is unchanged. So this bug is present and unfixed on 3.5, 3.6 and 4.0 alike, and upgrading does not resolve it.

To Reproduce

  1. OpenSIPS using the cgrates module for accounting (cgrates_acc) with a persistent connection to the engine's SessionS on port 2014.
  2. Engine doing prepaid rating with AllowNegative:false, so it issues SessionSv1.DisconnectSession back over the same connection when balance is exhausted.
  3. Run prepaid traffic where accounts reach zero balance mid-call.
  4. A worker begins spinning at 100% CPU and never recovers.

Expected behavior

When read() returns EAGAIN on the listen path, cgrc_async_read should return and let the reactor re-arm the fd, rather than looping on the syscall. The worker should continue processing SIP normally after handling an engine-initiated push.

Proposed fix (splits EINTR from EAGAIN):

	if (bytes_read < 0) {
		if (errno == EINTR)
			goto try_again;
		else if (errno == EAGAIN || errno == EWOULDBLOCK) {
			async_status = ASYNC_DONE;
			return final_ret;
		}
		else if (errno == ECONNRESET) {
			...

Returning ASYNC_DONE lets cgrates_async_resume_req re-arm the fd normally. Partial replies are already handled earlier by the json_tokener_continueASYNC_CONTINUE branch before any second read, so the only path reaching a second read() is after a complete parse, where yielding on EAGAIN is correct.

Relevant System Logs

Backtrace of a spinning worker (captured live, reproduced on three separate occasions):

#0  read () from /lib64/libc.so.6
#1  cgrc_async_read () from cgrates.so
#2  cgrates_async_resume_req () from cgrates.so
#3  async_fd_resume ()
#4  handle_io.constprop ()
#5  io_wait_loop_epoll.constprop ()
#6  udp_start_processes ()
#7  main ()

strace -c on the worker (5-second sample): ~279000 read() calls, all returning EAGAIN, ~0.8s total in-syscall, no other syscalls. /proc/<pid>/syscall alternates between read on the engine fd and userspace. ss shows a few unread bytes stuck in the connection's Recv-Q.

OS/environment information

  • Operating System: CentOS Stream 9
  • OpenSIPS installation: manual packages (yum.opensips.org)
  • other relevant information: cgrates module, JSON-RPC to engine SessionS on :2014; engine performs remote session disconnects on negative prepaid balance

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions