Skip to content

release (required): igb gateways and DPDK/kernel driver switching - #2003

Merged
Frostman merged 7 commits into
mainfrom
pr/daniel-noland/vlab-igb-combined
Oct 27, 2026
Merged

Frostman merged 7 commits into
mainfrom
pr/daniel-noland/vlab-igb-combined

Conversation

@daniel-noland

@daniel-noland daniel-noland commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

The required set for this release. Stacked on it: #2004 (advised), then #1988 (optional).

Combines #1999, #2000, #2001, #2002 and #1985 into one branch, so CI builds one set of images with all of them for the test team. Each commit is unchanged from its PR (rebased onto main); see those PRs for the details and their testing.

commit from what
fix(dpdk): access x86 device registers with a single mov #1999 DPDK's igb driver configures RSS with an SSE store KVM cannot emulate; the guest took SIGILL at port bring-up. All x86 MMIO accessors become single GPR movs; replaces the e1000 -mno-avx workaround.
feat(n-vm): add QEMU's igb as a NIC model #1999 igb in VM tests, each behind its own PCIe root port (its SR-IOV slots collide otherwise); CONFIG_IGB=y.
feat(hardware): support QEMU's igb NIC #1999 Accept the 82576 (8086:10c9), bind it to vfio-pci; igb DPDK VM tests incl. RSS over the data queues.
fix(init): create the VFIO nodes this container's /dev lacks #2000 First start after boot failed with "Failed to open VFIO group"; init now mknods the group nodes.
fix(dataplane): give diverted control packets a verdict #2001 ~10 "unspecified verdict" ERRORs/s on NICs that refuse the control flow rules (all emulated NICs).
fix(init): bring NICs back from vfio-pci for named kernel interfaces #2002 Test-team report: DPDK → kernel driver switch left the NICs on vfio-pci.
bump(dplane-plugin): do not hold address installs (draft) #1985 Pins githedgehog/dplane-plugin#26 (a PR branch). Fixes taps without addresses after a dataplane restart. vlab: 8/8 stuck with the stock plugin, 8/8 OK with this.

Images

CI pushes, for each push to this branch:

  • ghcr.io/githedgehog/dataplane:v0-<first 9 of the head sha>-{debug,release}
  • ghcr.io/githedgehog/dataplane/frr:v0-<sha9>-{debug,release}
  • ghcr.io/githedgehog/dataplane/validator:v0-<sha9>-release

Use the dataplane and FRR images together: the FRR image carries the plugin fix.

Also needed outside this repo

Testing (this branch, rebased on main 977bd98)

  • clippy -D warnings (hardware, n-vm, init, dpdk, dataplane) and rustfmt clean.
  • n-vm 304/304, init 81/81 (with TMPDIR=/tmp; incl. the VFIO-node and by-name transition VM tests), dataplane worker tests 8/8, dpdk_in_vm igb/e1000/e1000e 8/8.
  • vlab, with the individual commits: igb gateways with 4 workers and RSS, BGP/BFD/LLDP up, test-connectivity passes; DPDK ↔ kernel switching both ways; 8/8 restart races clean with the plugin fix.

Draft: no DCO sign-offs yet; the plugin pin points at an unmerged dplane-plugin branch.

🤖 Generated with Claude Code

daniel-noland and others added 7 commits October 26, 2026 18:17
A volatile access leaves the instruction to the compiler. Clang folded
an e1000 register load into a VEX vpbroadcastd, and igb_hw_rss_hash_set
stores the OR of the hash flags, computed in SSE registers, with movd.
KVM emulates every access to an emulated NIC's BAR and its instruction
emulator rejects both, so the guest takes SIGILL: on e1000 at probe, on
igb as soon as RSS is configured.

Patch DPDK's x86 MMIO accessors into inline asm that moves through a
general-purpose register. This replaces building the e1000 driver with
-mno-avx, which kept the VEX load out but not the SSE store, and covers
every driver. Passed-through NICs already got a single mov.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
igb emulates the Intel 82576, the one emulated Intel NIC with RSS and
several queue pairs. Each igb gets a pcie-root-port of its own: QEMU
reserves the slots its SR-IOV virtual functions would take, 16 and 17
slots on, and an igb in the next slot collides with them. Build the
kernel driver into the test kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The e1000 and e1000e have no RSS, so a dataplane on them runs one
worker. igb (Intel 82576) hashes, has a redirection table and 16 queue
pairs, which is what a port needs to run several data workers and keep
RSS off the control queue. Accept it, bind it to vfio-pci, and test that
DPDK spreads RSS over the data queues only.

Also correct the PMD the e1000e is driven by: net_e1000_em.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A container's /dev is filled in when the container starts, and device
nodes created later do not appear in it. vfio-pci creates a group's
node when it binds the group's first device, which init does from
inside the container, so on the first start after boot DPDK fails to
open the group ("Failed to open VFIO group 9"), skips the device as not
managed by vfio, and the dataplane exits. The restart finds the node
and comes up, so it looked like a one-off crash.

After binding, read each group's device number from sysfs and mknod the
nodes that are missing, and the container node with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the NIC refuses a control class's flow rule, data workers take
that class off the data path themselves and punt it to the kernel, or
drop it when it is not addressed to the port. Neither path set a
verdict, so dropping the packet logged "Attempted to drop packet with
unspecified verdict!" once per control packet. On an emulated NIC,
which refuses every class rule, that is every BGP, BFD, ARP and LLDP
frame: about ten errors a second on each vlab gateway.

Mark punted packets Local and the rest MacNotForUs, as the pipeline
does for the same packets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Switching a gateway from the DPDK driver to the kernel driver left its
NICs on vfio-pci. Init only handed a device back when its interface
was given as pci@ADDR, and the kernel driver's interfaces come as
kernel@NAME or a bare name, so the names never appeared and the
dataplane could not start.

When a named interface is missing, hand back every NIC a DPDK run could
have left on vfio-pci, once: a supported vfio-pci NIC still overridden
to vfio-pci by our bind, and bound to it or to nothing. A device bound
to vfio-pci some other way has no override and is left alone. Then
give the name time to appear, since udev may rename the interface. A
restart that finds the interface already under its configured name
takes it as named, as the pci@ path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins githedgehog/dplane-plugin#26, which stops the plugin from holding
IPv4 address installs until the dataplane answers. Held installs could
leave the taps with no address after a dataplane restart when FRR runs
in its own pod.

The pin, and its entry in scripts/gen-pins.sh, point at the PR branch so
CI builds and tests it. Repoint both at hh-master-10.7 once the plugin
PR merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Walkthrough

This change adds QEMU igb support, updates DPDK packet verdict handling, adds VFIO device-node creation and kernel-interface recovery, applies an x86 MMIO patch, and updates the dplane-plugin source pin.

Changes: Diverted packet verdicts

Layer / File(s) Summary
Classify diverted packets
dataplane/src/drivers/dpdk/worker.rs, dataplane/src/drivers/dpdk/worker/tests.rs
Diverted packets receive Local or MacNotForUs based on their destination MAC. The verdict controls whether punt_diverted punts the packet to the kernel. Tests cover broadcast, local, and nonlocal destinations.

QEMU igb support

Layer / File(s) Summary
Recognize Intel igb devices
hardware/src/nic/mod.rs, hardware/src/support.rs
Adds the igb PCI driver and recognizes Intel 82576 device ID 0x10c9 as a vfio-pci device.
Configure QEMU and guest igb
n-vm/src/config.rs, n-vm/src/kernel_feature.rs, n-vm/src/qemu/mod.rs, n-vm/tests/integration.rs, nix/pkgs/linux/fragments/intel-e1000.config, hardware/tests/dpdk_in_vm.rs
Adds the QEMU igb model and root-port lowering. Enables the guest driver and adds VM coverage for IOMMU modes, ICMPv6, RSS queues, and teardown.
Use x86 MMIO mov helpers
nix/pkgs/dpdk/default.nix, nix/pkgs/dpdk/patches/x86-mmio-mov.patch
The DPDK derivation applies an x86-64, non-MSVC patch with width-specific mov instructions for relaxed MMIO reads and writes.

VFIO initialization and interface recovery

Layer / File(s) Summary
Discover and create VFIO nodes
init/src/vfio.rs, init/src/main.rs
Finds missing VFIO container and group nodes, reads their device numbers from sysfs, and creates character devices. DPDK initialization invokes this process after device preparation.
Recover named kernel interfaces
init/src/kernel_interfaces.rs
Name-based interface preparation can return eligible devices left on vfio-pci to kernel drivers and poll for the source or configured interface name. Tests cover candidate filtering and recovery.

dplane-plugin source pin

Layer / File(s) Summary
Update dplane-plugin source
npins/sources.json, scripts/gen-pins.sh
Updates the source pin and pin-generation script to the pr/daniel-noland/addr-install-no-hold branch.

Possibly related issues

  • githedgehog/dataplane#1977: The source pin changes to a branch described as addressing tap address loss after a dataplane restart, but the available summary does not establish that this issue is resolved.

Possibly related PRs

Suggested reviewers: mvachhar, qmonnet

Priority: Not assessed

Merge Risk: Moderate · up to 08616

Large Igb VMs may fail to start despite passing configuration validation. Address the root-port limit or reject unsupported configurations before merging.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@Frostman Frostman added the ci:+vlab Enable VLAB tests label Oct 27, 2026
@Frostman Frostman closed this Oct 27, 2026
@Frostman Frostman reopened this Oct 27, 2026
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.14773% with 91 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
init/src/kernel_interfaces.rs 66.37% 39 Missing ⚠️
init/src/vfio.rs 80.31% 24 Missing and 1 partial ⚠️
init/src/main.rs 0.00% 22 Missing ⚠️
dataplane/src/drivers/dpdk/worker.rs 81.81% 2 Missing ⚠️
hardware/tests/dpdk_in_vm.rs 0.00% 2 Missing ⚠️
hardware/src/support.rs 83.33% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@mvachhar
mvachhar self-requested a review October 27, 2026 00:52

@mvachhar mvachhar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate this problem and the fix, but otherwise this is fine. I hate the recovery code too, but again, don't know anything better.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Broad x86 MMIO changes, privileged PCI driver transitions, and the acknowledged unmerged plugin dependency require final human validation before release.

No open issues 🎉

What changed in this PR

Bundles the required release changes for multi-worker igb gateways, reliable VFIO startup, and switching between DPDK and kernel drivers.

Changes:

  • Adds QEMU igb support and VM coverage, replacing the AVX workaround with single-instruction x86 register accesses.
  • Creates missing VFIO device nodes and restores kernel interfaces after DPDK use.
  • Assigns verdicts to diverted control packets and pins the plugin address-install fix for coordinated dataplane/FRR testing.
File Description
scripts/​gen-pins.sh Tracks the plugin’s address-install PR branch.
npins/​sources.json Pins the plugin fix revision and hash.
nix/​pkgs/​linux/​fragments/​intel-e1000.config Enables the igb kernel driver for VM tests.
nix/​pkgs/​dpdk/​patches/​x86-mmio-mov.patch Uses single general-purpose-register moves for x86-64 MMIO.
nix/​pkgs/​dpdk/​default.nix Applies the MMIO patch instead of disabling AVX.
n-vm/​tests/​integration.rs Includes igb in mixed-NIC integration coverage.
n-vm/​src/​qemu/​mod.rs Places igb NICs behind separate PCIe root ports.
n-vm/​src/​kernel_feature.rs Declares the igb kernel feature.
n-vm/​src/​config.rs Adds the QEMU-only igb NIC model.
init/​src/​vfio.rs Finds and creates missing VFIO device nodes, with tests.
init/​src/​main.rs Exposes VFIO nodes after device binding.
init/​src/​kernel_interfaces.rs Restores named interfaces from VFIO and handles renamed interfaces on restart.
hardware/​tests/​dpdk_in_vm.rs Adds igb receive and RSS configuration tests; corrects driver documentation.
hardware/​src/​support.rs Recognizes Intel 82576 devices as VFIO-capable.
hardware/​src/​nic/​mod.rs Adds the igb kernel driver identifier.
dataplane/​src/​drivers/​dpdk/​worker/​tests.rs Checks verdicts for diverted control packets.
dataplane/​src/​drivers/​dpdk/​worker.rs Marks diverted packets as local or not addressed to the port.

Review effort: Balanced

@Frostman
Frostman disabled auto-merge October 27, 2026 01:39
@Frostman
Frostman merged commit 2a9e355 into main Oct 27, 2026
36 of 50 checks passed
@Frostman
Frostman deleted the pr/daniel-noland/vlab-igb-combined branch October 27, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:+vlab Enable VLAB tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants