Skip to content

Commit 600d40d

Browse files
committed
README: the pool no longer needs a fork-window COW margin
The pre-fork pkg arena is off hugetlb now, so a short pool degrades late children to THP instead of SIGBUSing them. Sizing text, rule 2 and the troubleshooting row updated accordingly.
1 parent fee8acc commit 600d40d

2 files changed

Lines changed: 36 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -675,8 +675,8 @@ shm_auto_scaling_profile = MEM_SHM
675675
# /etc/sysctl.d/60-opensips.conf — the pool must fit the CAPS:
676676
# shm cap: 1024 MB = 512 pages
677677
# pkg cap × 31 workers: 32 MB × 31 = 496 pages
678-
# fork-window COW margin (~12%) ≈ 120 pages
679-
vm.nr_hugepages=1128
678+
# rounding / restart headroom (~5%) ≈ 50 pages
679+
vm.nr_hugepages=1060
680680
```
681681

682682
```ini
@@ -685,11 +685,16 @@ vm.nr_hugepages=1128
685685
LimitMEMLOCK=infinity
686686
```
687687

688-
The COW margin is not optional: children briefly copy-on-write-touch
689-
the parent's pkg arena before building their own, and each touched page
690-
draws a **free** pool page. A pool with zero free pages at fork time
691-
SIGBUSes children — measured, and it is the same crash family as a real
692-
2026-08 production incident. If the pool cannot fit a cap:
688+
Only the caps draw from the pool. The pre-fork (attendant) pkg arena —
689+
the one every child inherits copy-on-write — is deliberately **not**
690+
hugetlb-backed: a child's COW fault inside a hugetlb mapping has no 4K
691+
fallback and no reservation, so an empty pool at fork time was a silent
692+
SIGBUS (measured; it is how TCP main, the last no-script child, died at
693+
startup on a short pool — twice). Its ladder starts at THP instead,
694+
whose COW splits to 4K pages, and no-script children no longer walk the
695+
inherited route AST with `pkg_free` before swapping to their own arena.
696+
Startup therefore cannot SIGBUS on pool state: a short pool only pushes
697+
late children down the tier ladder. If the pool cannot fit a cap:
693698

694699
```
695700
NOTICE: hugetlb pool cannot back a 1024 MB cap; reserving the 128 MB in use instead - the arena keeps huge pages but cannot grow. Raise vm.nr_hugepages to allow growth.
@@ -876,8 +881,11 @@ All at their exact severities; `%` values are illustrative.
876881
reserved from the hugetlb pool at map time, and every per-child pkg
877882
arena reserves its own `-M` cap the same way. Budget
878883
`shm_cap + pkg_cap × workers + ~12% margin` pages.
879-
2. **Leave free pages for the fork window** (the COW margin, 12.1).
880-
Zero free pages at fork time = SIGBUS in children. Measured, twice.
884+
2. **A short pool degrades, it no longer kills.** The one arena
885+
children inherit copy-on-write — the attendant's pkg arena — is kept
886+
off hugetlb (12.1), so a pool with zero free pages at fork time
887+
pushes late children to THP instead of SIGBUSing them. That SIGBUS
888+
was measured, twice, before this rule existed.
881889
3. **pkg caps multiply** — RAM and, on tier 1, pool reservations.
882890
4. **Tier-1 shrink returns pages to the pool, not to RAM.** Freeing
883891
host memory requires shrinking `vm.nr_hugepages` as well.
@@ -915,7 +923,7 @@ All at their exact severities; `%` values are illustrative.
915923
| never shrinks | top page busy, or inside the post-grow cool-off (10× down-cycles), or usage above the down-threshold, or below-floor/blocked safety hold | check `hg_stats`; prefer-low needs time to drain the top |
916924
| `DRY RUN - would grow` but nothing happens | `hg_autoscale_dry_run = 1` | that is the point; set 0 to act |
917925
| `grow_refused` climbing, gauge 0 | isolated refusals; hysteresis holding | by design — the gauge latches on *sustained* refusal |
918-
| children SIGBUS at startup on tier 1 | pool sized to caps with zero margin | rule 2 |
926+
| `failed to initialize child process N` / `cannot fork tcp main` at startup, no arena line for that child | a build predating `HG_INIT_INHERITED`: the last no-script child COW-faulted the parent's hugetlb pkg arena on an empty pool | upgrade; meanwhile leave free pages in the pool at fork time |
919927
| arena runs unpinned (`continuing unpinned`) | `RLIMIT_MEMLOCK` too low for a tier 2–4 arena | `LimitMEMLOCK=infinity` in the unit |
920928
| testing under `ulimit -l` shows no refusals | you are root — `CAP_IPC_LOCK` bypasses `RLIMIT_MEMLOCK` entirely | test the mlock leg as an unprivileged user (`setpriv`) |
921929
| pool numbers "prove" shrink is broken | rule 5 | use the exact object-residency formula |

‎mem/README.hg_malloc_v3.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -669,8 +669,8 @@ shm_auto_scaling_profile = MEM_SHM
669669
# /etc/sysctl.d/60-opensips.conf — the pool must fit the CAPS:
670670
# shm cap: 1024 MB = 512 pages
671671
# pkg cap × 31 workers: 32 MB × 31 = 496 pages
672-
# fork-window COW margin (~12%) ≈ 120 pages
673-
vm.nr_hugepages=1128
672+
# rounding / restart headroom (~5%) ≈ 50 pages
673+
vm.nr_hugepages=1060
674674
```
675675

676676
```ini
@@ -679,11 +679,16 @@ vm.nr_hugepages=1128
679679
LimitMEMLOCK=infinity
680680
```
681681

682-
The COW margin is not optional: children briefly copy-on-write-touch
683-
the parent's pkg arena before building their own, and each touched page
684-
draws a **free** pool page. A pool with zero free pages at fork time
685-
SIGBUSes children — measured, and it is the same crash family as a real
686-
2026-08 production incident. If the pool cannot fit a cap:
682+
Only the caps draw from the pool. The pre-fork (attendant) pkg arena —
683+
the one every child inherits copy-on-write — is deliberately **not**
684+
hugetlb-backed: a child's COW fault inside a hugetlb mapping has no 4K
685+
fallback and no reservation, so an empty pool at fork time was a silent
686+
SIGBUS (measured; it is how TCP main, the last no-script child, died at
687+
startup on a short pool — twice). Its ladder starts at THP instead,
688+
whose COW splits to 4K pages, and no-script children no longer walk the
689+
inherited route AST with `pkg_free` before swapping to their own arena.
690+
Startup therefore cannot SIGBUS on pool state: a short pool only pushes
691+
late children down the tier ladder. If the pool cannot fit a cap:
687692

688693
```
689694
NOTICE: hugetlb pool cannot back a 1024 MB cap; reserving the 128 MB in use instead - the arena keeps huge pages but cannot grow. Raise vm.nr_hugepages to allow growth.
@@ -870,8 +875,11 @@ All at their exact severities; `%` values are illustrative.
870875
reserved from the hugetlb pool at map time, and every per-child pkg
871876
arena reserves its own `-M` cap the same way. Budget
872877
`shm_cap + pkg_cap × workers + ~12% margin` pages.
873-
2. **Leave free pages for the fork window** (the COW margin, 12.1).
874-
Zero free pages at fork time = SIGBUS in children. Measured, twice.
878+
2. **A short pool degrades, it no longer kills.** The one arena
879+
children inherit copy-on-write — the attendant's pkg arena — is kept
880+
off hugetlb (12.1), so a pool with zero free pages at fork time
881+
pushes late children to THP instead of SIGBUSing them. That SIGBUS
882+
was measured, twice, before this rule existed.
875883
3. **pkg caps multiply** — RAM and, on tier 1, pool reservations.
876884
4. **Tier-1 shrink returns pages to the pool, not to RAM.** Freeing
877885
host memory requires shrinking `vm.nr_hugepages` as well.
@@ -909,7 +917,7 @@ All at their exact severities; `%` values are illustrative.
909917
| never shrinks | top page busy, or inside the post-grow cool-off (10× down-cycles), or usage above the down-threshold, or below-floor/blocked safety hold | check `hg_stats`; prefer-low needs time to drain the top |
910918
| `DRY RUN - would grow` but nothing happens | `hg_autoscale_dry_run = 1` | that is the point; set 0 to act |
911919
| `grow_refused` climbing, gauge 0 | isolated refusals; hysteresis holding | by design — the gauge latches on *sustained* refusal |
912-
| children SIGBUS at startup on tier 1 | pool sized to caps with zero margin | rule 2 |
920+
| `failed to initialize child process N` / `cannot fork tcp main` at startup, no arena line for that child | a build predating `HG_INIT_INHERITED`: the last no-script child COW-faulted the parent's hugetlb pkg arena on an empty pool | upgrade; meanwhile leave free pages in the pool at fork time |
913921
| arena runs unpinned (`continuing unpinned`) | `RLIMIT_MEMLOCK` too low for a tier 2–4 arena | `LimitMEMLOCK=infinity` in the unit |
914922
| testing under `ulimit -l` shows no refusals | you are root — `CAP_IPC_LOCK` bypasses `RLIMIT_MEMLOCK` entirely | test the mlock leg as an unprivileged user (`setpriv`) |
915923
| pool numbers "prove" shrink is broken | rule 5 | use the exact object-residency formula |

0 commit comments

Comments
 (0)