rtpengine/bencode: heap corruption, double close, wrong-fd close and OOM crash fixes

[zbyslb]

Summary

Four clear-cut bug fixes for the rtpengine module and its bundled bencode helper, found while hardening a production OpenSIPS deployment. Each commit is self-contained; none of them changes any behaviour apart from removing the crash/corruption itself.

bencode: fully initialize the buffer before the first allocation

If __bencode_piece_new() fails (pkg memory exhausted), bencode_buffer_init() returns -1 having only assigned buf->pieces — free_list and error keep their previous (garbage) values. Callers, which cannot distinguish the failure stage, routinely run bencode_buffer_free() on the returned error, and the free routine starts by dereferencing buf->free_list: undefined behaviour (reading a garbage pointer and calling through it). Zeroing the struct up front makes the failure path a safely freeable empty object; the success path assigns all three fields explicitly, so nothing changes there.

rtpengine: fix heap corruption when caching delete statistics

rtpe_stats_free(ctx->stats);
pkg_free(&(ctx->stats->buf));

buf is an embedded bencode_buffer_t inside struct rtpe_stats — the second member, after dict — so &ctx->stats->buf is an interior pointer, not the address pkg_malloc() returned. Freeing it corrupts the pkg allocator's free lists. On top of that, ctx->stats itself was not NULLed afterwards, so the code right below kept writing through a dangling pointer (ctx->stats->buf = ...). The fix keeps rtpe_stats_free() (which releases the json string and the buffer's inner pieces, making the struct safely reusable once the three fields are overwritten just below) and drops the bogus pkg_free(); rtpe_ctx_free() eventually frees the struct using its real base address. Triggers on the second delete-with-stats reply processed on the same message context.

rtpengine: close the async reply fd exactly once, and not the node index

Two fd lifecycle bugs in resume_async_send_rtpe_command():

1. The unix branch closes the fd right after read(). But every return path sets async_status = ASYNC_DONE_CLOSE_FD, so the async framework (tm/async.c:180, async.c:242) closes the same fd again once the resume function returns — a double close that hits an unrelated descriptor whenever the fd number is reused in between.
2. The UDP error branch calls RTPE_IO_ERROR_CLOSE(param->node->idx). The macro expands to close(_fd) followed by (_fd) = -1, so on its EPIPE/EBADF branch it (a) closes whichever descriptor happens to carry the node's array index number — for the first nodes of a set that is one of the stdio descriptors — and (b) writes -1 into the shared-memory node->idx, turning every later rtpe_socks[node->idx] access into an out-of-bounds array access.

Both fixed by dropping the manual close attempts and leaving the fd exclusively to the framework, which already guarantees the close through ASYNC_DONE_CLOSE_FD on all paths.

rtpengine: check the pkg_malloc() result of the async buffer

rtpe_function_call_async() never checks the bencbuf allocation result: on pkg memory exhaustion, the very first touch of the buffer (bencode_buffer_init()) dereferences the NULL pointer and crashes the worker. Nothing else is allocated at that point, so an early return is sufficient.

Testing

Built and running on a production 3.6.x deployment (as a source overlay on the 3.6.9 tag); the heap-corruption path was identified through code audit of the stats caching flow, the fd and NULL-check paths through the same audit plus runtime behaviour on a ~10 node rtpengine farm.