Segmentation fault in Reseau epoll poller thread under gVisor (`_backend_poll_once!`, `epoll.jl:235`)

[gsoleilhac]

Environment

Julia	1.12.6 / 1.13-rc1
HTTP.jl	2.0.0
Reseau	1.1.3 (165e3ac) — also reproduced on 1.1.2 (3b6e3aa)
Platform	Scaleway Serverless Jobs (gVisor sandbox)
uname -a	Linux e7f94219-061e-42fc-a1fd-94e8979590e0 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

Description

I'm getting a signal 11 (SIGSEGV) crash in Reseau's background epoll I/O poller thread, in _backend_poll_once!.

Claude suggests that @gcsafe_ccall epoll_wait in _backend_poll_once! allows Julia's GC to run during the syscall, and gVisor's userspace epoll implementation exposes a race that doesn't exist on a real Linux kernel. This would explain why I can reproduce it consistently on Scaleway Serverless Jobs (a serverless container platform that uses gVisor as its sandbox) but not at all on standard Docker locally.

Stack trace

MWE reproduction :

┌ Info: environment
│   julia = "1.12.6"
│   os = :Linux
└   arch = :x86_64
┌ Info: packages
│   HTTP = v"2.0.0"
└   Reseau = v"1.1.3"
┌ Info: kernel
└   uname = "Linux 105e9ccc-a395-48a5-9f20-c2824a20e916 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux"

[1] signal 11 (1): Segmentation fault
in expression starting at /app/mwe.jl:11
== at ./promotion.jl:637 [inlined]
== at ./promotion.jl:487 [inlined]
_backend_poll_once! at /root/.julia/packages/Reseau/chanm/src/iopoll/epoll.jl:235
_poller_thread_main! at /root/.julia/packages/Reseau/chanm/src/iopoll/runtime.jl:384
_poller_thread_entry at /root/.julia/packages/Reseau/chanm/src/iopoll/runtime.jl:401
jlcapi__poller_thread_entry_18496 at /root/.julia/compiled/v1.12/Reseau/vKBJ8_aNuEz.so (unknown line)
unknown function (ip: 0x7ea27c7e0b7a) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7ea27c85e7f7) at /lib/x86_64-linux-gnu/libc.so.6
Allocations: 37680835 (Pool: 37680686; Big: 149); GC: 5294

Line 235 is if n == -1 immediately following @gcsafe_ccall epoll_wait(...). The inlined == frames in promotion.jl point to the type promotion for that comparison.

Minimal reproduction

If you have gVisor (runsc) installed locally, you should be able to reproduce it with:

docker build -t http-mwe .
docker run --runtime=runsc --rm http-mwe

and let it run a couple minutes.

mwe.jl:

using HTTP

@info "environment" julia=string(VERSION) os=Sys.KERNEL arch=Sys.ARCH
deps = Dict(v.name => v.version for (_, v) in Pkg.dependencies())
@info "packages" HTTP=deps["HTTP"] Reseau=deps["Reseau"]
@info "kernel" uname=readchomp(`uname -a`)

server = HTTP.serve!(_ -> HTTP.Response(200, "ok"), "127.0.0.1", 8765)

while true
    HTTP.get("http://127.0.0.1:8765/")
    rand(100_000)
end

Dockerfile:

FROM julia:1.12.6

ENV JULIA_CPU_TARGET="x86-64"

WORKDIR /app
COPY mwe.jl Project.toml ./

RUN julia --project=@. -e 'using Pkg; Pkg.instantiate()'

ENTRYPOINT ["julia", "-t1", "--project=@.", "mwe.jl"]

Project.toml:

[deps]
HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"

[compat]
HTTP = "2.0.0"

# Could also be a reproducer for the precompilation issue ? I gave up after 40+ minutes
[preferences.HTTP]
precompile_workload = false

[quinnj]

I pushed a Reseau experiment branch for this: https://github.com/JuliaServices/Reseau.jl/tree/codex/gvisor-epoll-experiment

It defaults Linux epoll_wait to ordinary @ccall instead of @gcsafe_ccall, and has two env toggles for A/B testing.

To try it in the reproducer project:

using Pkg
Pkg.develop(Pkg.PackageSpec(name="Reseau", url="https://github.com/JuliaServices/Reseau.jl", rev="codex/gvisor-epoll-experiment"))
Pkg.resolve()

Then please run the MWE three ways if possible:

julia -t1 --project=@. mwe.jl
RESEAU_EPOLL_WAIT_GCSAFE=1 julia -t1 --project=@. mwe.jl
RESEAU_EPOLL_MALLOC_EVENTS=1 julia -t1 --project=@. mwe.jl

The first checks whether avoiding the GC-safe transition fixes it; the second restores the old syscall mode; the third keeps the branch default but uses a malloc-backed epoll event buffer.

[gsoleilhac]

Hey, thank you for looking into this so quickly :)
I can confirm that with RESEAU_EPOLL_WAIT_GCSAFE=1 the crash reproduces, while the other two variants have been running fine for over 20 minutes. 👍

[quinnj]

Thanks, that A/B is very useful. I opened a narrower release-shaped Reseau PR here: JuliaServices/Reseau.jl#118

This keeps the normal preserved Julia event buffer, avoids the GC-safe epoll_wait path that reproduced the crash, and caps each GC-unsafe wait so the poller cannot block Julia GC indefinitely while idle.

Could you try the MWE against this branch?

using Pkg
Pkg.develop(Pkg.PackageSpec(name="Reseau", url="https://github.com/JuliaServices/Reseau.jl", rev="codex/gvisor-epoll-release"))
Pkg.resolve()

Then run the normal reproducer command, without the experiment env vars. If that survives in Scaleway/gVisor, I think this is the Reseau fix to merge and release.

[gsoleilhac]

Tested against the branch (now released as v1.1.4). The original epoll crash is gone, but there's a new crash that happens immediately on startup:

[1] signal 11 (1): Segmentation fault
in expression starting at /app/mwe.jl:1
_drain_expired_time_entries! at /root/.julia/packages/Reseau/e3LfV/src/iopoll/timers.jl:352
_poller_thread_main! at /root/.julia/packages/Reseau/e3LfV/src/iopoll/runtime.jl:386
_poller_thread_entry at /root/.julia/packages/Reseau/e3LfV/src/iopoll/runtime.jl:401
Allocations: 1 (Pool: 1; Big: 0); GC: 0

Claude suggests the 25ms cap is what triggers this. Without it (option 1 in the A/B test, RESEAU_EPOLL_WAIT_GCSAFE=false), epoll_wait blocks until a real I/O event arrives, so _drain_expired_time_entries! is only called once the server is active. With the cap, the poller wakes every 25ms and calls _drain_expired_time_entries! immediately on startup — in a context (idle poller thread, gVisor) that crashes.

[quinnj]

Closing as fixed as of latest Reseau/HTTP releases

[gsoleilhac]

Update: still reproduces on Scaleway after updating to latest HTTP and Reseau releases.

┌ Info: environment
│   julia = "1.12.6"
│   os = :Linux
└   arch = :x86_64
┌ Info: packages
│   HTTP = v"2.4.0"
└   Reseau = v"1.3.1"
┌ Info: kernel
└   uname = "Linux 635f1b00-943a-4e63-bcc4-d7c08ecf95ea-attempt-1 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux"

[1] signal 11 (1): Segmentation fault
in expression starting at /app/mwe.jl:19
== at ./promotion.jl:637 [inlined]
_backend_poll_once! at /root/.julia/packages/Reseau/Mkbpx/src/iopoll/epoll.jl:269
_poller_thread_main! at /root/.julia/packages/Reseau/Mkbpx/src/iopoll/runtime.jl:402
_poller_thread_entry at /root/.julia/packages/Reseau/Mkbpx/src/iopoll/runtime.jl:419
jlcapi__poller_thread_entry_18604 at /root/.julia/compiled/v1.12/Reseau/vKBJ8_aNuEz.so (unknown line)
unknown function (ip: 0x7ebed9934b7a) at /lib/x86_64-linux-gnu/libc.so.6
unknown function (ip: 0x7ebed99b27f7) at /lib/x86_64-linux-gnu/libc.so.6
Allocations: 19298264 (Pool: 19298108; Big: 156); GC: 933

[quinnj]

I was able to reproduce and diagnose the issue on an older version of gvisor; the root issue is google/gvisor#12518 which was fixed via google/gvisor@609778e. Confirmed that newer versions of gvisor w/ the included fix do not crash.

[gsoleilhac]

Got it, I'll see with scaleway support for an update then. Thanks for digging into it 👍