mem: the pre-fork pkg arena must not be hugetlb - a child's COW there is a SIGBUS

Every forked child inherits the attendant's pkg arena copy-on-write and
keeps it for life (module state parsed pre-fork lives there). While that
arena sat on MAP_HUGETLB, a child's first write into any of its pages
needed a fresh huge page - no 4K fallback inside a hugetlb VMA, and a
forked child holds no reservation on its parent's mapping - so with the
pool momentarily empty the fault was a silent SIGBUS. Startup exposed
it deterministically: the last no-script child (TCP main) freed the
inherited route AST before swapping to its own arena, and died with
'failed to initialize child process N / cannot fork tcp main process'
the moment the pool ran short (08-07 slot 52 on x86, again on an ARM
box with a small pool). The 08-07 ordering fix made that path free
against the right arena; it could not make it pool-independent.

Two changes, either sufficient, both kept:
 - HG_INIT_INHERITED: hg_malloc_init() takes a flags word; the pre-fork
   pkg arena passes it and hg_mem_reserve() starts its ladder at THP.
   THP COW splits the huge PMD and copies one 4K page - it cannot SIGBUS,
   and it cuts every child's COW cost from 2 MB per touched page to 4 KB.
   Per-child arenas are never inherited and keep the full ladder.
 - internal_fork_child_setup() no longer walks the inherited route AST
   with pkg_free under HG pkg: the child abandons that arena wholesale,
   so the frees bought nothing and were the one pre-swap writer.

Proven fail-first on a 16-child rig (-m 32 -M 8): pool 40 and 60 -
unpatched dies at child 14 = TCP main, strace shows SIGBUS BUS_ADRERR
with si_addr inside the attendant's 8 MB MAP_PRIVATE|MAP_HUGETLB
mapping; patched starts all 17 processes on the same pools, and more
children land on hugetlb since the pool no longer feeds the parent's
arena or COW pages. Pool 0 starts either way, as the mechanism predicts.

Author: Lt-Flash

mem/hg_malloc.c

@@ -260,7 +260,8 @@ static void hg_exclude_from_core(void *base, unsigned long size)
  *                        between processes through the shared gpool.
  */
 static void *hg_mem_reserve(unsigned long size, unsigned long *cap,
-		enum hg_mem_tier *tier, unsigned long *locked_mb, int shared)
+		enum hg_mem_tier *tier, unsigned long *locked_mb, int shared,
+		int inherited)
 {
 	unsigned long asize = HG_HPS_ROUND(size);
 	unsigned long csize = HG_HPS_ROUND(*cap < size ? size : *cap);
@@ -298,9 +299,17 @@ static void *hg_mem_reserve(unsigned long size, unsigned long *cap,
 	 * trade than simply not being able to grow. A cap-less tier-1 arena is
 	 * what v2 shipped; losing the tier is a real regression.
 	 */
-	p = mmap(NULL, csize, PROT_READ|PROT_WRITE,
+	/*
+	 * ...except for the arena children inherit copy-on-write (the pre-fork
+	 * pkg arena, HG_INIT_INHERITED): a child's write into an inherited
+	 * hugetlb page needs a fresh huge page with no 4K fallback and no
+	 * reservation behind it - an empty pool at that instant is a SIGBUS.
+	 * That arena starts the ladder at THP, whose COW splits to 4K pages
+	 * instead. See the flag's comment in hg_malloc.h.
+	 */
+	p = inherited ? MAP_FAILED : mmap(NULL, csize, PROT_READ|PROT_WRITE,
 	         vis|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
-	if (p == MAP_FAILED && csize > asize) {
+	if (p == MAP_FAILED && !inherited && csize > asize) {
 		p = mmap(NULL, asize, PROT_READ|PROT_WRITE,
 		         vis|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
 		if (p != MAP_FAILED) {
@@ -816,7 +825,7 @@ static void hg_arena_reg_del(struct hg_block *hb)
  * (it cannot: HG_MALLOC IS what those macros dispatch to when selected).
  */
 struct hg_block *hg_malloc_init(unsigned long size, char *name, int shared,
-		const char *proc_desc)
+		const char *proc_desc, unsigned int flags)
 {
 	enum hg_mem_tier tier;
 	unsigned long locked_mb;
@@ -842,7 +851,8 @@ struct hg_block *hg_malloc_init(unsigned long size, char *name, int shared,
 		cap = hg_pkg_cap_bytes;
 	else
 		cap = 0;
-	base = hg_mem_reserve(size, &cap, &tier, &locked_mb, shared);
+	base = hg_mem_reserve(size, &cap, &tier, &locked_mb, shared,
+		(flags & HG_INIT_INHERITED) != 0);
 	if (!base) {
 		LM_ERR("failed to reserve %lu bytes for %s HG_MALLOC arena\n",
 			size, name);
@@ -931,8 +941,11 @@ struct hg_block *hg_malloc_init(unsigned long size, char *name, int shared,
 			name, proc_desc, size >> 20, hg_mem_tier_str(tier), locked_mb);
 	else
 		LM_NOTICE("%s " HG_MALLOC_NAME " arena: %lu MB on %s, %lu MB "
-			"pinned from swapping\n",
-			name, size >> 20, hg_mem_tier_str(tier), locked_mb);
+			"pinned from swapping%s\n",
+			name, size >> 20, hg_mem_tier_str(tier), locked_mb,
+			(flags & HG_INIT_INHERITED) ?
+			" (pre-fork arena, inherited copy-on-write by every child: "
+			"hugetlb deliberately skipped, its COW cannot fall back)" : "");
 	if (hb->hcap > hb->hsize)
 		LM_NOTICE("%s arena can grow to %lu MB (%lu MB headroom "
 			"reserved, uncommitted)\n", name, hb->hcap >> 20,

mem/hg_malloc.h

@@ -713,9 +713,24 @@ static inline unsigned long hg_leaf_of(const struct hg_block *hb, const void *p)
 /* @shared: 1 for shm/shm_dbg (MAP_SHARED - one arena for every worker),
  * 0 for pkg (MAP_PRIVATE - each forked worker gets its own copy-on-write
  * arena, lock and free pools). See hg_mem_reserve() in hg_malloc.c for why
- * getting this wrong for pkg is a correctness AND a performance bug. */
+ * getting this wrong for pkg is a correctness AND a performance bug.
+ *
+ * @flags: HG_INIT_INHERITED marks the ONE arena that outlives fork() as a
+ * copy-on-write inheritance: the pre-fork (attendant) pkg arena. Every
+ * child holds a private COW view of it for life - it reads module state
+ * the parent parsed into it, and any write it makes there is a COW fault.
+ * Such an arena must never be hugetlb-backed: a COW fault inside a
+ * hugetlb VMA can only be satisfied by a huge page (no 4K fallback), and
+ * a forked child holds no reservation on its parent's mapping, so with
+ * the pool momentarily empty the fault is a SIGBUS - silent, at fork
+ * time (measured: it is exactly how the last no-script child, TCP main,
+ * died at startup on a short pool). THP has the fallback: a COW fault
+ * splits the huge PMD and copies one 4K page. Per-child arenas are never
+ * inherited (children do not fork) and keep the full ladder. */
+#define HG_INIT_INHERITED  (1U << 0)
+
 struct hg_block *hg_malloc_init(unsigned long size, char *name, int shared,
-		const char *proc_desc);
+		const char *proc_desc, unsigned int flags);
 void hg_malloc_destroy(struct hg_block *hb);
 
 /*

mem/mem.c

@@ -89,10 +89,17 @@ int init_pkg_mallocs(void)
 	 * leak that block (never freed, never used) - across dozens of
 	 * worker processes that adds up, so the call is skipped for it
 	 * rather than allocated-and-ignored.
+	 *
+	 * HG_INIT_INHERITED: this is the ONE arena every forked child
+	 * inherits copy-on-write (pt.c hands each child a fresh arena of its
+	 * own, but the parent's stays mapped in the child, holding all the
+	 * module state parsed pre-fork). It must not sit on hugetlb - a
+	 * child's COW fault there has no 4K fallback and no reservation, so
+	 * an empty pool at fork is a silent SIGBUS. It starts on THP instead.
 	 */
 #ifdef INLINE_ALLOC
 #if defined HG_MALLOC
-	mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL);
+	mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL, HG_INIT_INHERITED);
 #else
 	mem_pool = malloc(pkg_mem_size);
 	if (!mem_pool) {
@@ -170,7 +177,7 @@ int init_pkg_mallocs(void)
 #endif
 #ifdef HG_MALLOC
 	case MM_HG_MALLOC:
-		mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL);
+		mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL, HG_INIT_INHERITED);
 		gen_pkg_malloc     = (osips_block_malloc_f)hg_malloc;
 		gen_pkg_realloc    = (osips_block_realloc_f)hg_realloc;
 		gen_pkg_free       = (osips_block_free_f)hg_free;
@@ -235,7 +242,7 @@ int init_pkg_mallocs(void)
 #endif
 #ifdef HG_MALLOC
 	case MM_HG_MALLOC_DBG:
-		mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL);
+		mem_block = hg_malloc_init(pkg_mem_size, "pkg", 0, NULL, HG_INIT_INHERITED);
 		gen_pkg_malloc    = (osips_block_malloc_f)hg_malloc_dbg;
 		gen_pkg_realloc   = (osips_block_realloc_f)hg_realloc_dbg;
 		gen_pkg_free      = (osips_block_free_f)hg_free_dbg;

mem/shm_mem.c

@@ -298,7 +298,7 @@ int shm_mem_init_mallocs(void* mempool, unsigned long pool_size,int idx)
 	/* ignores mempool/pool_size as an address - HG_MALLOC does its own
 	 * reservation, pool_size is reused only as the size to request, see
 	 * the shm_mem_init() caller-side skip of shm_getmem() for this case */
-	shm_block = hg_malloc_init(pool_size, "shm", 1, NULL);
+	shm_block = hg_malloc_init(pool_size, "shm", 1, NULL, 0);
 #endif
 #else
 
@@ -528,7 +528,7 @@ int shm_mem_init_mallocs(void* mempool, unsigned long pool_size,int idx)
 	case MM_HG_MALLOC:
 		/* ignores mempool - HG_MALLOC does its own reservation; see the
 		 * shm_mem_init() caller-side skip of shm_getmem() for this case */
-		shm_block = hg_malloc_init(pool_size, "shm", 1, NULL);
+		shm_block = hg_malloc_init(pool_size, "shm", 1, NULL, 0);
 		gen_shm_malloc         = (osips_block_malloc_f)hg_malloc;
 		gen_shm_malloc_unsafe  = (osips_block_malloc_f)hg_malloc;
 		gen_shm_realloc        = (osips_block_realloc_f)hg_realloc;
@@ -605,7 +605,7 @@ int shm_mem_init_mallocs(void* mempool, unsigned long pool_size,int idx)
 #endif
 #ifdef HG_MALLOC
 	case MM_HG_MALLOC_DBG:
-		shm_block = hg_malloc_init(pool_size, "shm", 1, NULL);
+		shm_block = hg_malloc_init(pool_size, "shm", 1, NULL, 0);
 		gen_shm_malloc         = (osips_block_malloc_f)hg_malloc_dbg;
 		gen_shm_malloc_unsafe  = (osips_block_malloc_f)hg_malloc_dbg;
 		gen_shm_realloc        = (osips_block_realloc_f)hg_realloc_dbg;
@@ -811,7 +811,7 @@ int shm_dbg_mem_init_mallocs(void* mempool, unsigned long pool_size)
 	case MM_HG_MALLOC:
 	case MM_HG_MALLOC_DBG:
 		/* ignores mempool, same reasoning as shm_mem_init_mallocs() */
-		shm_dbg_block = hg_malloc_init(pool_size, "shm_dbg", 1, NULL);
+		shm_dbg_block = hg_malloc_init(pool_size, "shm_dbg", 1, NULL, 0);
 		break;
 #endif
 	default:

pt.c

@@ -556,6 +556,29 @@ static int internal_fork_child_setup(const struct internal_fork_params *ifpp)
 
 	/* free the script if not needed */
 	if (!(ifpp->flags & OSS_PROC_NEEDS_SCRIPT) && sroutes) {
+#if defined(PKG_MALLOC) && defined(HG_MALLOC)
+		/*
+		 * Under HG_MALLOC this child is about to abandon the whole pkg
+		 * arena it inherited from the parent (internal_fork() swaps in
+		 * a fresh, private one right after this handler chain), so
+		 * freeing the route AST cell by cell buys nothing - and it was
+		 * the one thing writing into the inherited arena before the
+		 * swap: every such write is a copy-on-write fault against the
+		 * parent's mapping, and while that mapping was hugetlb-backed
+		 * an empty pool turned the very first one into a SIGBUS (how
+		 * TCP main, the last no-script child, died at startup on a
+		 * short pool). The parent's arena no longer sits on hugetlb
+		 * (HG_INIT_INHERITED), so this is not load-bearing for safety
+		 * any more - it just spares every no-script child a private
+		 * 4K copy of each AST page it would otherwise dirty. Dropping
+		 * the pointer is all the child needs.
+		 */
+		if (mem_allocator_pkg == MM_HG_MALLOC ||
+		    mem_allocator_pkg == MM_HG_MALLOC_DBG) {
+			sroutes = NULL;
+			return 0;
+		}
+#endif
 		free_route_lists(sroutes);
 		sroutes = NULL;
 	}
@@ -726,24 +749,24 @@ int internal_fork(const struct internal_fork_params *ifpp)
 			 * specific process - exactly the semantics wanted here, just
 			 * never previously invoked per child.
 			 *
-			 * MUST run after the post-fork handler chain below, not before:
-			 * internal_fork_child_setup() (the default handler) calls
-			 * free_route_lists(), which pkg_frees the route AST this child
-			 * inherited from the parent pre-fork. Those pointers only make
-			 * sense against the ORIGINAL (COW-inherited) arena. Swapping
-			 * mem_block to a fresh arena before that handler runs made it
-			 * free foreign pointers through the new arena's bookkeeping -
-			 * silent cross-arena corruption at best, and a SIGBUS at worst
-			 * (the COW page-fault into the abandoned old arena has nothing
-			 * left to fall back on once the hugetlb pool is tight, which is
-			 * exactly what exposed this). Running the swap here instead
-			 * lets free_route_lists() finish its cleanup against the still-
-			 * correct original arena first; only then is it abandoned - the
-			 * parent's inherited mapping is left unmapped-but-unreferenced,
-			 * and this child never touches it again.
+			 * Runs after the post-fork handler chain, so anything a handler
+			 * frees is still resolved against the arena it was allocated
+			 * from (the inherited one). Two guarantees back this up now,
+			 * neither of which depends on the hugetlb pool having a page
+			 * left at fork time:
+			 *   - internal_fork_child_setup() no longer walks the route AST
+			 *     with pkg_free under HG (see there): core makes NO writes
+			 *     into the inherited arena before the swap;
+			 *   - the inherited arena is not hugetlb-backed
+			 *     (HG_INIT_INHERITED in mem.c): should anything ever write
+			 *     into it - now or later in the child's life - the COW is
+			 *     an ordinary page fault with a 4K fallback, not a SIGBUS.
+			 * The parent's mapping stays in this child, unreferenced; the
+			 * child reads parent-parsed module state through it and never
+			 * allocates from it again.
 			 */
 			struct hg_block *child_pkg =
-				hg_malloc_init(pkg_mem_size, "pkg", 0, ifpp->proc_desc);
+				hg_malloc_init(pkg_mem_size, "pkg", 0, ifpp->proc_desc, 0);
 			if (!child_pkg) {
 				LM_CRIT("failed to init this child's own pkg memory "
 					"(%lu bytes)\n", pkg_mem_size);