[Bug Report] Panic in ChooseDialTarget during staged DNS controller handoff #1054

Checks

• I have searched the existing issues
• I have read the documentation

Current Behavior

During a staged same-port reload, dae can panic while an old control-plane generation is still handling a TCP connection:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1026870]

goroutine 886759 [running]:
github.com/daeuniverse/dae/control.(*DnsController).HasDnsKnowledge(0x0, ...)
        github.com/daeuniverse/dae/control/dns_control.go:935 +0x50
github.com/daeuniverse/dae/control.(*ControlPlane).ChooseDialTarget(...)
        github.com/daeuniverse/dae/control/control_plane.go:1720 +0x16a
github.com/daeuniverse/dae/control.(*ControlPlane).chooseProxyDialer(...)
        github.com/daeuniverse/dae/control/dial.go:110 +0xbb
github.com/daeuniverse/dae/control.(*ControlPlane).routeDial(...)
        github.com/daeuniverse/dae/control/dial.go:234 +0xb4
github.com/daeuniverse/dae/control.(*ControlPlane).handleConn(...)
        github.com/daeuniverse/dae/control/tcp.go:233 +0xeb4
github.com/daeuniverse/dae/control.(*ControlPlane).Serve.func2.1(...)
        github.com/daeuniverse/dae/control/control_plane.go:3132 +0xb3

The panic happened about 55 ms after the staged generation entered Serve and about 5 ms after restoring the DNS reload cache:

WARN Reload: Received reload signal; prepare to reload
WARN Reload: Prepare staged same-port handoff
WARN Reload: Serve
INFO Restored 80 DNS cache entries from previous control plane
panic: runtime error: invalid memory address or nil pointer dereference

This appears to be a DNS-controller ownership handoff race:

1. reuseDNSControllerFrom publishes the replacement controller through the old plane's handoff pointer.
2. It then sets previous.dnsController = nil and assigns the replacement to the new plane.
3. Old accepted/in-flight TCP work can still call ChooseDialTarget while the old generation is draining.
4. ChooseDialTarget reads c.dnsController directly instead of resolving the handoff with c.ActiveDnsController().

The old generation therefore has a valid dnsHandoffController, but c.dnsController is nil. This direct access panics when dial_mode: domain reaches HasDnsKnowledge.

The same direct access and ownership-transfer sequence are present in the v2.0.0 tag and current main:

// control/control_plane.go
if c.dnsController.HasDnsKnowledge(c.dnsController.cacheKey(domain, ...)) {

// control/dns_runtime.go
if publishHandoff != nil {
    publishHandoff(reusedController)
}
previous.dnsController = nil
r.dnsController = reusedController

Expected Behavior

Connections still running on the old control-plane generation should resolve the DNS handoff controller and complete or drain without crashing the process.

ChooseDialTarget should take one stable ActiveDnsController() snapshot and handle nil safely, rather than directly reading the ownership field during reload.

Steps to Reproduce

The race is timing-sensitive; a later reload with the same configuration and active traffic succeeded.

1. Configure dae with dial_mode: domain and a same-port reload configuration.
2. Keep TCP traffic active so the old control plane is accepting or handling connections during handoff.
3. Repeatedly run dae reload without changing the tproxy port.
4. Occasionally an old-generation connection reaches ChooseDialTarget after DNS ownership has transferred and dae panics as above.

A deterministic regression test could:

1. Create old and replacement control planes in domain mode.
2. Call newCP.ReuseDNSControllerFrom(oldCP) and verify oldCP resolves the published handoff while its owned dnsController is detached.
3. Invoke oldCP.ChooseDialTarget(...) for a non-reserved outbound and a non-IP domain.
4. Assert that it does not panic and uses the handoff controller.

Environment

• Dae version: unstable-20260710.r873.f920fa4 (downstream build based on upstream v2.0.0; the relevant code is unchanged from v2.0.0/current main)
• Go runtime: go1.26.3-X:heapminimum512kib linux/amd64
• OS: Debian GNU/Linux 13 (trixie)
• Kernel: 6.12.95+deb13-cloud-amd64
• Architecture: x86_64
• Relevant config: dial_mode: domain; unchanged tproxy port, so reload used staged same-port handoff

Anything else?

This is related to the staged handoff subsystem discussed in #1013/#1016, but appears to be a distinct userspace lifecycle race. #1016 addresses shared BPF/domain-routing-map mutation and does not change ChooseDialTarget, ActiveDnsController, or the ownership detachment shown above.

The crash is also operationally severe with the packaged systemd unit: a Go panic exits with status 2, while Restart=on-abnormal does not restart a process that exited normally with a non-zero status. That is secondary to the controller race, but it can turn this panic into a prolonged outage.