You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 600d40d
Browse filesBrowse the repository at this point in the historyBrowse files
README: the pool no longer needs a fork-window COW margin
The pre-fork pkg arena is off hugetlb now, so a short pool degrades
late children to THP instead of SIGBUSing them. Sizing text, rule 2 and
the troubleshooting row updated accordingly.
# /etc/sysctl.d/60-opensips.conf — the pool must fit the CAPS:
676
676
# shm cap: 1024 MB = 512 pages
677
677
# pkg cap × 31 workers: 32 MB × 31 = 496 pages
678
-
#fork-window COW margin (~12%) ≈ 120 pages
679
-
vm.nr_hugepages=1128
678
+
#rounding / restart headroom (~5%) ≈ 50 pages
679
+
vm.nr_hugepages=1060
680
680
```
681
681
682
682
```ini
@@ -685,11 +685,16 @@ vm.nr_hugepages=1128
685
685
LimitMEMLOCK=infinity
686
686
```
687
687
688
-
The COW margin is not optional: children briefly copy-on-write-touch
689
-
the parent's pkg arena before building their own, and each touched page
690
-
draws a **free** pool page. A pool with zero free pages at fork time
691
-
SIGBUSes children — measured, and it is the same crash family as a real
692
-
2026-08 production incident. If the pool cannot fit a cap:
688
+
Only the caps draw from the pool. The pre-fork (attendant) pkg arena —
689
+
the one every child inherits copy-on-write — is deliberately **not**
690
+
hugetlb-backed: a child's COW fault inside a hugetlb mapping has no 4K
691
+
fallback and no reservation, so an empty pool at fork time was a silent
692
+
SIGBUS (measured; it is how TCP main, the last no-script child, died at
693
+
startup on a short pool — twice). Its ladder starts at THP instead,
694
+
whose COW splits to 4K pages, and no-script children no longer walk the
695
+
inherited route AST with `pkg_free` before swapping to their own arena.
696
+
Startup therefore cannot SIGBUS on pool state: a short pool only pushes
697
+
late children down the tier ladder. If the pool cannot fit a cap:
693
698
694
699
```
695
700
NOTICE: hugetlb pool cannot back a 1024 MB cap; reserving the 128 MB in use instead - the arena keeps huge pages but cannot grow. Raise vm.nr_hugepages to allow growth.
@@ -876,8 +881,11 @@ All at their exact severities; `%` values are illustrative.
876
881
reserved from the hugetlb pool at map time, and every per-child pkg
877
882
arena reserves its own `-M` cap the same way. Budget
2.**Leave free pages for the fork window** (the COW margin, 12.1).
880
-
Zero free pages at fork time = SIGBUS in children. Measured, twice.
884
+
2.**A short pool degrades, it no longer kills.** The one arena
885
+
children inherit copy-on-write — the attendant's pkg arena — is kept
886
+
off hugetlb (12.1), so a pool with zero free pages at fork time
887
+
pushes late children to THP instead of SIGBUSing them. That SIGBUS
888
+
was measured, twice, before this rule existed.
881
889
3.**pkg caps multiply** — RAM and, on tier 1, pool reservations.
882
890
4.**Tier-1 shrink returns pages to the pool, not to RAM.** Freeing
883
891
host memory requires shrinking `vm.nr_hugepages` as well.
@@ -915,7 +923,7 @@ All at their exact severities; `%` values are illustrative.
915
923
| never shrinks | top page busy, or inside the post-grow cool-off (10× down-cycles), or usage above the down-threshold, or below-floor/blocked safety hold | check `hg_stats`; prefer-low needs time to drain the top |
916
924
|`DRY RUN - would grow` but nothing happens |`hg_autoscale_dry_run = 1`| that is the point; set 0 to act |
917
925
|`grow_refused` climbing, gauge 0 | isolated refusals; hysteresis holding | by design — the gauge latches on *sustained* refusal |
918
-
|children SIGBUS at startup on tier 1 | pool sized to caps with zero margin | rule 2|
926
+
|`failed to initialize child process N` / `cannot fork tcp main`at startup, no arena line for that child | a build predating `HG_INIT_INHERITED`: the last no-script child COW-faulted the parent's hugetlb pkg arena on an empty pool | upgrade; meanwhile leave free pages in the pool at fork time|
919
927
| arena runs unpinned (`continuing unpinned`) |`RLIMIT_MEMLOCK` too low for a tier 2–4 arena |`LimitMEMLOCK=infinity` in the unit |
920
928
| testing under `ulimit -l` shows no refusals | you are root — `CAP_IPC_LOCK` bypasses `RLIMIT_MEMLOCK` entirely | test the mlock leg as an unprivileged user (`setpriv`) |
921
929
| pool numbers "prove" shrink is broken | rule 5 | use the exact object-residency formula |
# /etc/sysctl.d/60-opensips.conf — the pool must fit the CAPS:
670
670
# shm cap: 1024 MB = 512 pages
671
671
# pkg cap × 31 workers: 32 MB × 31 = 496 pages
672
-
#fork-window COW margin (~12%) ≈ 120 pages
673
-
vm.nr_hugepages=1128
672
+
#rounding / restart headroom (~5%) ≈ 50 pages
673
+
vm.nr_hugepages=1060
674
674
```
675
675
676
676
```ini
@@ -679,11 +679,16 @@ vm.nr_hugepages=1128
679
679
LimitMEMLOCK=infinity
680
680
```
681
681
682
-
The COW margin is not optional: children briefly copy-on-write-touch
683
-
the parent's pkg arena before building their own, and each touched page
684
-
draws a **free** pool page. A pool with zero free pages at fork time
685
-
SIGBUSes children — measured, and it is the same crash family as a real
686
-
2026-08 production incident. If the pool cannot fit a cap:
682
+
Only the caps draw from the pool. The pre-fork (attendant) pkg arena —
683
+
the one every child inherits copy-on-write — is deliberately **not**
684
+
hugetlb-backed: a child's COW fault inside a hugetlb mapping has no 4K
685
+
fallback and no reservation, so an empty pool at fork time was a silent
686
+
SIGBUS (measured; it is how TCP main, the last no-script child, died at
687
+
startup on a short pool — twice). Its ladder starts at THP instead,
688
+
whose COW splits to 4K pages, and no-script children no longer walk the
689
+
inherited route AST with `pkg_free` before swapping to their own arena.
690
+
Startup therefore cannot SIGBUS on pool state: a short pool only pushes
691
+
late children down the tier ladder. If the pool cannot fit a cap:
687
692
688
693
```
689
694
NOTICE: hugetlb pool cannot back a 1024 MB cap; reserving the 128 MB in use instead - the arena keeps huge pages but cannot grow. Raise vm.nr_hugepages to allow growth.
@@ -870,8 +875,11 @@ All at their exact severities; `%` values are illustrative.
870
875
reserved from the hugetlb pool at map time, and every per-child pkg
871
876
arena reserves its own `-M` cap the same way. Budget
2.**Leave free pages for the fork window** (the COW margin, 12.1).
874
-
Zero free pages at fork time = SIGBUS in children. Measured, twice.
878
+
2.**A short pool degrades, it no longer kills.** The one arena
879
+
children inherit copy-on-write — the attendant's pkg arena — is kept
880
+
off hugetlb (12.1), so a pool with zero free pages at fork time
881
+
pushes late children to THP instead of SIGBUSing them. That SIGBUS
882
+
was measured, twice, before this rule existed.
875
883
3.**pkg caps multiply** — RAM and, on tier 1, pool reservations.
876
884
4.**Tier-1 shrink returns pages to the pool, not to RAM.** Freeing
877
885
host memory requires shrinking `vm.nr_hugepages` as well.
@@ -909,7 +917,7 @@ All at their exact severities; `%` values are illustrative.
909
917
| never shrinks | top page busy, or inside the post-grow cool-off (10× down-cycles), or usage above the down-threshold, or below-floor/blocked safety hold | check `hg_stats`; prefer-low needs time to drain the top |
910
918
|`DRY RUN - would grow` but nothing happens |`hg_autoscale_dry_run = 1`| that is the point; set 0 to act |
911
919
|`grow_refused` climbing, gauge 0 | isolated refusals; hysteresis holding | by design — the gauge latches on *sustained* refusal |
912
-
|children SIGBUS at startup on tier 1 | pool sized to caps with zero margin | rule 2|
920
+
|`failed to initialize child process N` / `cannot fork tcp main`at startup, no arena line for that child | a build predating `HG_INIT_INHERITED`: the last no-script child COW-faulted the parent's hugetlb pkg arena on an empty pool | upgrade; meanwhile leave free pages in the pool at fork time|
913
921
| arena runs unpinned (`continuing unpinned`) |`RLIMIT_MEMLOCK` too low for a tier 2–4 arena |`LimitMEMLOCK=infinity` in the unit |
914
922
| testing under `ulimit -l` shows no refusals | you are root — `CAP_IPC_LOCK` bypasses `RLIMIT_MEMLOCK` entirely | test the mlock leg as an unprivileged user (`setpriv`) |
915
923
| pool numbers "prove" shrink is broken | rule 5 | use the exact object-residency formula |
0 commit comments