libbpf-tools/netstacklat: Add tool to track latency in ingress network

[simosund]

Description

This PR adds the netstacklat (network stack latency) tool from bpf-examples. Adding netstacklat to libbpf-tools would make it more easily distributable while also complementing the current collection of libbpf-tools. This is a follow-up to issue #5510, which indicated there was some interest for adding netstacklat.

netstacklat summarizes how long packets (or more accurately, SKBs) have spent in the ingress network stack at several different points and reports the latency distribution as histograms. This can be useful to detect latency issues stemming from slow paths through the network stack or applications failing to read new socket data in a timely manner. netstacklat is also designed to be compatible with ebpf_exporter, making it possible to export the measurements to Prometheus.

While tcppktlat and softirq -d can be used for some similar purposes as netstacklat, netstacklat is more focused on measuring the latency for individual SKBs and provides a different set of capabilities compared to those prior tools. tcppktlat only measures the latency between data arriving and being read from TCP sockets (similar to the difference between netstacklat's tcp-socket-enqueued and tcp-socket-read), while netstacklat instead measures the total time each individual SKB has spent up both those and other points (including points in the IP and UDP stack as well). The net_rx distribution from softirq also covers time spent on processing SKB's in the ingress stack, but each measurement there will include the entire napi_poll() duration, which may process multiple SKBs at a time. netstacklat instead gives the per-SKB latency, offers a more granular breakdown of latency at different layers of the ingress stack, and can track the additional delay until it applications actually read the SKB payload.

We have published an academic paper describing the tool, evaluating its performance in a testbed, and sharing some brief results from a fleet-wide deployment at Cloudflare. This also covered in a more summarized manner in this RHRQ magazine article.

Why this approach

The central idea behind netstacklat (originally proposed by @netoptimizer) is to use the software timestamp from SOF_TIMESTAMPING_RX_SOFTWARE that the kernel sets for each SKB (sk_buff->tstamp) to determine the delay since the kernel started processing the SKB at later points (as illustrated in the figure below). This allows us to very efficiently determine the latency for each SKB without having to manage and lookup any external per-SKB state. By aggregating the latency measurements in-kernel, we also avoid a lot of the overhead (both in terms of performance and amount of output) that is associated with reporting each individual measurement, making it much more feasible to continuously run in production than tools like pwru and retis (which are great complementary tools to dig down into any latency issues identified by netstacklat).

Changes to histogram printing

netstacklat was designed to also work together with ebpf_exporter to export metrics to Prometheus. As a result of this, netstacklat uses a slightly different histogram format than other libbpf-tools. Briefly put, we use log2 histograms with right-closed bins instead of left-closed bins to work together with Prometheus' less-than-or-equal (le) format for histograms. On the eBPF side, we essentially compute the bin index as ceil(log2(val)) instead of floor(log2(val))).

To support our slightly different histogram binning, I've taken the liberty to generalize the logic for printing histograms in libbpf-tools, and in the process also made some minor functional changes that I think improves the output of current libbpf-tools. The first two commits describe the functional changes. The third commit generalizes and refactors the histogram printing logic, allowing it to be reused for netstacklat but without any further functional changes for existing tools. As part of the generalization I also added support for custom formatting of the bin labels. netstacklat needs to cover a very wide range (from 10s of nanoseconds to full seconds), where I find it much easier to interpret e.g. "537ms -> 1.07s" than "536870913 -> 1073741824" nanoseconds.

Let me know if you prefer me to break up the histogram printing changes in a separate PR, or just not meddle with the common histogram printing at all (just keep separate logic for histogram printing in netstacklat.c?). I'm open for any ideas on how to best make netstacklat's histogram structure work in libbpf-tools.

---

Checklist

• Commit prefix matches changed area (e.g., tools/toolname:, libbpf-tools/toolname:, src/cc:, docs:, build:, tests/python:)
• Commit body explains why this change is needed

For new tools only

• Explains why this tool is needed and what existing tools cannot cover this use case
  • Explanation can be found in this PR description as well as the commit adding the tool
• Includes at least one real production use case
  • netstacklat has been deployed at Cloudflare to monitor latency within the local host before nginx processes incoming requests. This case is described in more detail in our paper and more briefly summarized in this RHRQ article. You can also find example use cases in this blog post and the netstacklat_examples.txt, although those are artificially set up for demonstration purposes and do not correspond to real production use cases.
• Man page (man/man8/) with an OVERHEAD section
• Example output file (*_example.txt)
• README.md entry added
  • There does not appear to be any entries for prior libbpf-tools in the README
• Smoke test added to tests/python/test_tools_smoke.py
  • This does not seem relevant for libbpf-tools.

---

About AI Code Review: This project uses GitHub Copilot to assist with code review.
If a Copilot review is added, treat its feedback as you would any reviewer comment — you can
agree, disagree (with explanation), or ask questions. The maintainer makes all final decisions.