Troubleshooting Tools

A comprehensive toolset for performance analysis and troubleshooting of virtualized network systems using eBPF technologies.

Overview

This repository provides a collection of eBPF-based tools for monitoring, tracing, and analyzing network performance issues in virtualized environments. The tools are designed to help identify packet drops, measure latency, trace data paths, and analyze system performance bottlenecks.

Repository Structure

`measurement-tools/`

Main directory containing all eBPF-based monitoring and troubleshooting tools organized by system component:

`cpu/`

CPU and scheduler monitoring tools:

Off-CPU time analysis
Scheduler latency monitoring
Futex and pthread lock tracing

`kvm-virt-network/`

KVM virtualization network stack monitoring:

kvm/ - KVM IRQ injection and interrupt statistics
tun/ - TUN/TAP device monitoring (ring buffer, GSO, TX stats)
vhost-net/ - vhost eventfd, queue correlation, buffer peek stats
virtio-net/ - virtio-net polling, IRQ monitoring, RX path tracing

`linux-network-stack/`

Linux kernel network stack tools:

Connection tracking (conntrack) monitoring
IP fragmentation/defragmentation tracing
packet-drop/ - Comprehensive packet drop detection and analysis

`ovs/`

Open vSwitch specific monitoring:

Userspace megaflow analysis
Kernel module drop monitoring
Upcall latency measurement

`performance/`

Network performance measurement tools:

system-network/ - System-level network latency and metrics (ICMP RTT, TCP latency)
vm-network/ - VM-specific network performance analysis
- VM network latency decomposition
- vm_pair_latency/ - Inter-VM latency monitoring
- vm_pair_latency_gap/ - Latency gap and jitter analysis

`other/`

Additional tracing tools:

Abnormal ARP detection
OVS connection tracking invalid states
Network offloading and segmentation tracing
Qdisc and TX queue monitoring

`docs/`

Documentation and guides:

publish/ - User manuals and deployment guides
Design documents for various monitoring approaches

`test/`

Test configurations and specifications for all tool categories

Prerequisites

System Requirements

Linux kernel with eBPF support (4.1+ recommended)
Root privileges for BPF operations

Software Dependencies

BCC (BPF Compiler Collection) - Required for Python BPF tools
bpftrace - Required for .bt scripts
Python 2/3 - Most tools support both versions
- Python 2 for el7 systems with bcc package
- Python 3 for oe1 systems with bpfcc package

Installation

Install BCC:

# For RHEL/CentOS 7
sudo yum install bcc-tools python2-bcc

# For openEuler distributions
sudo apt-get install bpfcc-tools python3-bpfcc

Install bpftrace:

# For RHEL/CentOS
sudo yum install bpftrace

# For Ubuntu/Debian
sudo apt-get install bpftrace

Clone the repository:

git clone <repository-url>
cd troubleshooting-tools

Usage Examples

Packet Drop Analysis

Monitor packet drops for specific connections:

# Monitor TCP drops for specific connection
sudo ./bcc-tools/packet-drop/multi-protocol-drop-monitor.py \
    --src 192.168.1.10 --dst 192.168.1.20 --protocol tcp --dst-port 443

# Monitor all protocol drops
sudo ./bcc-tools/packet-drop/drop.py

Network Latency Measurement

Measure network latency at various layers:

# System network ICMP latency
sudo ./bcc-tools/performance/system-network/icmp_rtt_latency.py \
    --src-ip 192.168.1.10 --dst-ip 192.168.1.20 \
    --phy-iface1 eth0 --phy-iface2 eth1

# VM network latency
sudo ./bcc-tools/performance/vm-network/vm_latency.py \
    --src-ip 192.168.1.10 --dst-ip 192.168.1.20 \
    --vm-interface vnet0 --phy-interface eth0 --direction tx

OVS Analysis

Monitor OVS operations:

# OVS upcall monitoring
sudo ./bcc-tools/ovs-measurement/ovs-upcall-execute.py

# OVS megaflow analysis
sudo ./bcc-tools/ovs-measurement/ovs_userspace_megaflow.py --debug

CPU Performance Analysis

Monitor CPU-related performance issues:

# Off-CPU time analysis
sudo ./bcc-tools/cpu-measurement/offcputime-ts.py

# Scheduler latency monitoring
sudo ./bcc-tools/cpu-measurement/sched_latency_monitor.sh

Bpftrace Scripts

Use bpftrace for quick analysis:

# Trace abnormal ARP events
sudo bpftrace bpftrace-tools/trace-abnormal-arp.bt

# Monitor OVS connection tracking issues
sudo bpftrace bpftrace-tools/trace-ovs-ct-invalid.bt

Network Architecture

This toolset is designed for virtualized network environments with the following architecture:

System Network: Physical host network interfaces and kernel network stack
VM Network: Virtual machine network interfaces (kvm virtlization env , with TUN/TAP devices, vnet interfaces)
OVS Integration: Open vSwitch datapath monitoring (ovs as vswitch in virtualization env)
Virtio Network: Virtio network device performance monitoring (vhost-net && virtio-net)

For detailed architecture information, see the documentation in the md/ directory.

Output and Logging

Most tools output to stdout by default. To capture logs:

# Redirect output to log file
sudo ./tool-name --options > output.log 2>&1

# Real-time monitoring with logging
sudo ./tool-name --options | tee output.log

Performance Considerations

Tools may introduce overhead in production environments
Use filtering options to reduce noise and performance impact
Monitor system resources when running intensive tracing
Consider using sampling or time-limited tracing for high-traffic systems: summary version