This repository provides a custom kernel tracepoint, scx_custom_trace_event, that can be emitted from eBPF functions.
This tracepoint can be recorded by ftrace for visualization with Perfetto or other viz tools like Kernel Shark.
A set of kfuncs (scx_custom_trace_event_begin, scx_custom_trace_event_end, scx_custom_trace_event_instant) are provided for conveniently emitting the tracepoint in a format ftrace and Perfetto understand.
An example use case for this tracepoint is to track the occurance and duration of certain scheduler functions in an eBPF sched-ext (SCX) scheduler. Normally, you could just collect timestamps in an eBPF map at the beginning and end of each function to understand the timing behavior. However, it's hard to understand how the scheduler and the SCX-scheduled tasks are interacting with other tasks in the system.
For example, collecting just timestamps at the beginning and end of your enqueue or dispatch functions can tell you when and for how long each occurance lasts. However, it can be cumbersome to match this up with system traces from ftrace to see if this was expected timing behavior, given the rest of the system's state.
This tracepoint allows eBPF program developers to emit a marker into an ftrace report, so the full set of timing and interaction data can be understood. The resulting ftrace report can be either parsed with a script for automatic analysis, or can be visualized with tools like Perfetto to create an interactive timeline view.
git clone https://github.com/wagler/scx-tracer.git
cd scx-tracer
If you're using the default kernel that came with your Linux distro:
make
If you have a custom kernel you built with a custom compiler (like if you built the kernel with Clang and special BPF options enabled for sched-ext), then you need to build the repo like you built your kernel.
For example, if your kernel source code is located in ~/kernel-source-dir, and you built it with clang-21:
make LLVM=-21 CLANG=clang-21 CC=clang-21 KDIR=~/kernel-source-dir
In the compiler output, make sure you see the following BTF [M] scx_tracer.ko. If you don't, then you need to recompile the kernel with CONFIG_DEBUG_INFO_BTF_MODULES enabled.
Then, insert the module into the kernel:
sudo insmod scx_tracer.ko
Verify it's inserted without error by viewing the dmesg log:
sudo dmesg | tail
It should look like this:
scx_tracer: Initializing...
scx_tracer: View events under /sys/kernel/tracing/events/scx_tracer/
Also, verify that ls /sys/kernel/tracing/events/scx_tracer/ looks like this:
enable filter scx_custom_trace_event
You can now enable this event like any other kernel tracepoint:
echo 1 | sudo tee /sys/kernel/tracing/events/scx_tracer/enable
echo 1 | sudo tee /sys/kernel/tracing/tracing_on
Now that the tracepoint is loaded in your kernel, you can emit it from your eBPF code.
To simply emit an instant tracepoint event (not a time-spanning slice event), call scx_custom_trace_event_instant(__func__).
If you want a time-spanning slice event (for Perfetto visualization), call scx_custom_trace_event_begin(__func__) to start a trace slice event, and then scx_custom_trace_event_end() to end the slice.
For example, to instrument your sched-ext scheduler, call scx_custom_trace_event_begin(__func__) at the beginning and scx_custom_trace_event_end() at the end of each function.
An example of an SCX scheduler with the markers can be found in the file scx_simple_tracer_annotated.bpf.c. It's a modified version of the scx_simple.bpf.c scheduler found here.
You can use any tracing tool you'd normally use (ftrace, bpftrace, perf, Perfetto, etc).
sudo trace-cmd record -e scx_tracer/scx_custom_trace_event
trace-cmd report > my_report.txt
sudo bpftrace -e 'tracepoint:scx_tracer:scx_custom_trace_event { <your code here> }'
Perfetto provides a cli utility tracebox to easily capture traces.
Use the provided trace config file scx_tracer.cfg as a template. It records a 5s trace, capturing our tracepoint as well as sched_switch and sched_waking.
Record a trace:
sudo /path/to/tracebox -c scx_tracer.cfg --txt -o my_scx_trace.pftrace
Then, go to the Perfetto UI and upload the my_scx_trace.pftrace file.