Repository navigation
release (required): igb gateways and DPDK/kernel driver switching - #2003
Conversation
A volatile access leaves the instruction to the compiler. Clang folded an e1000 register load into a VEX vpbroadcastd, and igb_hw_rss_hash_set stores the OR of the hash flags, computed in SSE registers, with movd. KVM emulates every access to an emulated NIC's BAR and its instruction emulator rejects both, so the guest takes SIGILL: on e1000 at probe, on igb as soon as RSS is configured. Patch DPDK's x86 MMIO accessors into inline asm that moves through a general-purpose register. This replaces building the e1000 driver with -mno-avx, which kept the VEX load out but not the SSE store, and covers every driver. Passed-through NICs already got a single mov. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
igb emulates the Intel 82576, the one emulated Intel NIC with RSS and several queue pairs. Each igb gets a pcie-root-port of its own: QEMU reserves the slots its SR-IOV virtual functions would take, 16 and 17 slots on, and an igb in the next slot collides with them. Build the kernel driver into the test kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The e1000 and e1000e have no RSS, so a dataplane on them runs one worker. igb (Intel 82576) hashes, has a redirection table and 16 queue pairs, which is what a port needs to run several data workers and keep RSS off the control queue. Accept it, bind it to vfio-pci, and test that DPDK spreads RSS over the data queues only. Also correct the PMD the e1000e is driven by: net_e1000_em. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A container's /dev is filled in when the container starts, and device
nodes created later do not appear in it. vfio-pci creates a group's
node when it binds the group's first device, which init does from
inside the container, so on the first start after boot DPDK fails to
open the group ("Failed to open VFIO group 9"), skips the device as not
managed by vfio, and the dataplane exits. The restart finds the node
and comes up, so it looked like a one-off crash.
After binding, read each group's device number from sysfs and mknod the
nodes that are missing, and the container node with them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the NIC refuses a control class's flow rule, data workers take that class off the data path themselves and punt it to the kernel, or drop it when it is not addressed to the port. Neither path set a verdict, so dropping the packet logged "Attempted to drop packet with unspecified verdict!" once per control packet. On an emulated NIC, which refuses every class rule, that is every BGP, BFD, ARP and LLDP frame: about ten errors a second on each vlab gateway. Mark punted packets Local and the rest MacNotForUs, as the pipeline does for the same packets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Switching a gateway from the DPDK driver to the kernel driver left its NICs on vfio-pci. Init only handed a device back when its interface was given as pci@ADDR, and the kernel driver's interfaces come as kernel@NAME or a bare name, so the names never appeared and the dataplane could not start. When a named interface is missing, hand back every NIC a DPDK run could have left on vfio-pci, once: a supported vfio-pci NIC still overridden to vfio-pci by our bind, and bound to it or to nothing. A device bound to vfio-pci some other way has no override and is left alone. Then give the name time to appear, since udev may rename the interface. A restart that finds the interface already under its configured name takes it as named, as the pci@ path does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins githedgehog/dplane-plugin#26, which stops the plugin from holding IPv4 address installs until the dataplane answers. Held installs could leave the taps with no address after a dataplane restart when FRR runs in its own pod. The pin, and its entry in scripts/gen-pins.sh, point at the PR branch so CI builds and tests it. Repoint both at hh-master-10.7 once the plugin PR merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
|
Caution Review failedThe pull request is closed. Walkthrough
Priority: Merge Risk:
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
mvachhar
left a comment
There was a problem hiding this comment.
I hate this problem and the fix, but otherwise this is fine. I hate the recovery code too, but again, don't know anything better.
There was a problem hiding this comment.
🔵 Needs a closer look
Broad x86 MMIO changes, privileged PCI driver transitions, and the acknowledged unmerged plugin dependency require final human validation before release.
No open issues 🎉
What changed in this PR
Bundles the required release changes for multi-worker igb gateways, reliable VFIO startup, and switching between DPDK and kernel drivers.
Changes:
- Adds QEMU igb support and VM coverage, replacing the AVX workaround with single-instruction x86 register accesses.
- Creates missing VFIO device nodes and restores kernel interfaces after DPDK use.
- Assigns verdicts to diverted control packets and pins the plugin address-install fix for coordinated dataplane/FRR testing.
| File | Description |
|---|---|
scripts/gen-pins.sh |
Tracks the plugin’s address-install PR branch. |
npins/sources.json |
Pins the plugin fix revision and hash. |
nix/pkgs/linux/fragments/intel-e1000.config |
Enables the igb kernel driver for VM tests. |
nix/pkgs/dpdk/patches/x86-mmio-mov.patch |
Uses single general-purpose-register moves for x86-64 MMIO. |
nix/pkgs/dpdk/default.nix |
Applies the MMIO patch instead of disabling AVX. |
n-vm/tests/integration.rs |
Includes igb in mixed-NIC integration coverage. |
n-vm/src/qemu/mod.rs |
Places igb NICs behind separate PCIe root ports. |
n-vm/src/kernel_feature.rs |
Declares the igb kernel feature. |
n-vm/src/config.rs |
Adds the QEMU-only igb NIC model. |
init/src/vfio.rs |
Finds and creates missing VFIO device nodes, with tests. |
init/src/main.rs |
Exposes VFIO nodes after device binding. |
init/src/kernel_interfaces.rs |
Restores named interfaces from VFIO and handles renamed interfaces on restart. |
hardware/tests/dpdk_in_vm.rs |
Adds igb receive and RSS configuration tests; corrects driver documentation. |
hardware/src/support.rs |
Recognizes Intel 82576 devices as VFIO-capable. |
hardware/src/nic/mod.rs |
Adds the igb kernel driver identifier. |
dataplane/src/drivers/dpdk/worker/tests.rs |
Checks verdicts for diverted control packets. |
dataplane/src/drivers/dpdk/worker.rs |
Marks diverted packets as local or not addressed to the port. |
Review effort: Balanced
The required set for this release. Stacked on it: #2004 (advised), then #1988 (optional).
Combines #1999, #2000, #2001, #2002 and #1985 into one branch, so CI builds one set of images with all of them for the test team. Each commit is unchanged from its PR (rebased onto main); see those PRs for the details and their testing.
fix(dpdk): access x86 device registers with a single movmovs; replaces the e1000-mno-avxworkaround.feat(n-vm): add QEMU's igb as a NIC modelCONFIG_IGB=y.feat(hardware): support QEMU's igb NIC8086:10c9), bind it to vfio-pci; igb DPDK VM tests incl. RSS over the data queues.fix(init): create the VFIO nodes this container's /dev lacksfix(dataplane): give diverted control packets a verdictfix(init): bring NICs back from vfio-pci for named kernel interfacesbump(dplane-plugin): do not hold address installs (draft)Images
CI pushes, for each push to this branch:
ghcr.io/githedgehog/dataplane:v0-<first 9 of the head sha>-{debug,release}ghcr.io/githedgehog/dataplane/frr:v0-<sha9>-{debug,release}ghcr.io/githedgehog/dataplane/validator:v0-<sha9>-releaseUse the dataplane and FRR images together: the FRR image carries the plugin fix.
Also needed outside this repo
0000:05:00.0,0000:06:00.0, ...), and Gatewayworkers> 1./mtu=Nfrom the fabric controller (gw-ctrl: give a DPDK dataplane its MTU and hugepages fabric#1704), and hugepages reserved on the gateway nodes.init-ifacesinit container, which otherwise fails before the dataplane can run.Testing (this branch, rebased on main 977bd98)
-D warnings(hardware, n-vm, init, dpdk, dataplane) and rustfmt clean.TMPDIR=/tmp; incl. the VFIO-node and by-name transition VM tests), dataplane worker tests 8/8,dpdk_in_vmigb/e1000/e1000e 8/8.test-connectivitypasses; DPDK ↔ kernel switching both ways; 8/8 restart races clean with the plugin fix.Draft: no DCO sign-offs yet; the plugin pin points at an unmerged dplane-plugin branch.
🤖 Generated with Claude Code