Debug kernel module / kernel driver

[QuangNguyenMinh123]

A kernel module is a piece of code that extends system capabilities — such as adding device drivers or file system support.
It can do many things: toggle an LED, drive a motor, or handle a monitor, a camera, and so on.
Kernel modules account for up to 70% of the source code in the upstream Linux kernel.

There are two types of kernel module: loadable and built-in.
Built-in kernel modules are "baked into" or "embedded in" the kernel at build time. They are essentially part of the kernel itself, so their load addresses are predetermined.

Loadable kernel modules are built into .ko files. A .ko file is essentially an ELF file in position-independent executable (PIE) format, which means it can be loaded and run at any arbitrary address.

As mentioned above, the load addresses of built-in kernel modules and of the Linux kernel itself are predetermined, which makes debugging them easy with JTAG.

The load address of a loadable kernel module, however, is unknown and only determined at runtime, which makes debugging more challenging.

Solution:
There is a Linux kernel function that can help us: load_module().
It is responsible for parsing a kernel module, checking its compatibility, and loading it into the kernel's memory layout. It decides where each segment of the module — .text, .data, .rodata, and so on — will be placed.
By reading the relevant kernel variables, we can tell GDB which part of the module's memory is mapped to which part of the kernel's memory layout, and thus debug kernel modules through a GUI.

[QuangNguyenMinh123]

A particular example Rpi4 openWRT image:

1. To build image: run make docker and make owrt, the generated image will be openwrt-bcm27xx-bcm2711-rpi-4-squashfs-factory.img. This image contains loadable kernel driver helloworld. Flash it to Rpi4, just like you flashed yocto previously to Rpi4
2. The source code of this kernel module is located at seer/tests/Rpi4/helloworld_owrt/src/helloworld.c
   It does basic test: Loading, unloading, create sysfs, perform basic fibinacci calculation:

root@OpenWrt:/# rmmod helloworld
[ 2292.184940] Goodbye, World! Cleaning up module on Raspberry Pi 4
root@OpenWrt:/# insmod helloworld
[ 2294.766332] Hello, World! Initializing advanced module on Raspberry Pi 4
[ 2294.773133] helloworld: Sysfs interface created at /sys/kernel/helloworld/fibonacci
root@OpenWrt:/# echo 1 > /sys/kernel/helloworld/fibonaci
[ 2306.848252] helloworld: Calculated Fibonacci(1) = 1
root@OpenWrt:/# echo 2 > /sys/kernel/helloworld/fibonacci \n
[ 2322.801048] helloworld: Invalid input for n
ash: write error: Invalid argument
root@OpenWrt:/# echo 5 > /sys/kernel/helloworld/fibonaci
[ 2336.245255] helloworld: Calculated Fibonacci(5) = 5
root@OpenWrt:/# echo 7 > /sys/kernel/helloworld/fibonaci
[ 2343.176934] helloworld: Calculated Fibonacci(7) = 13
root@OpenWrt:/# cat /sys/kernel/helloworld/fibonacci
Fibonacci(7) = 13

[QuangNguyenMinh123]

Typically, the location of kernel module is /lib/modules/<kernel version>/helloworld.ko
You could place kernel module at /etc and insmod kernel modules in that folder

When you do insmod helloworld, the kernel will find the helloworld.ko, calculate and allocate each segment address for kernel module.
You could test this by placing break point right before kernel invokes the do_init_module
Run -data-evaluate-expression "*mod->sect_attrs->attrs@mod->sect_attrs->nsections" to read allocated address of each segment of kernel module

The output:

From Widget:-data-evaluate-expression "*mod->sect_attrs->attrs@mod->sect_attrs->nsections"
From GdbMonitor: ^done,value="{{battr = {attr = {name = 0xffffff80410a1b00 ".text", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840147968}, {battr = {attr = {name = 0xffffff80410a1100 ".init.text", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798839791616}, {battr = {attr = {name = 0xffffff80410a1300 ".exit.text", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840148536}, {battr = {attr = {name = 0xffffff80410a1500 ".plt", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840148672}, {battr = {attr = {name = 0xffffff80410a1380 ".init.plt", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798839791872}, {battr = {attr = {name = 0xffffff80410a1680 ".text.ftrace_trampoline", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840148684}, {battr = {attr = {name = 0xffffff80410a1980 ".rodata", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840152064}, {battr = {attr = {name = 0xffffff80410a1880 ".note.gnu.property", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840152544}, {battr = {attr = {name = 0xffffff80410a1900 ".note.Linux", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840152576}, {battr = {attr = {name = 0xffffff80410a1700 ".data", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840156160}, {battr = {attr = {name = 0xffffff80410a1780 ".gnu.linkonce.this_module", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840156224}, {battr = {attr = {name = 0xffffff80410a1180 ".bss", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798840156864}, {battr = {attr = {name = 0xffffff8042143f00 ".symtab", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798839795712}, {battr = {attr = {name = 0xffffff80439f5980 ".strtab", mode = 256}, size = 19, private = 0x0, f_mapping = 0x0, read = 0xffffffc0080d3700 <module_sect_read>, write = 0x0, mmap = 0x0}, address = 18446743798839796312}}"

The output shows me that .text is allocated to address 18446743798840147968 = 0xFFFFFFC00081C000

Double check in terminal:

root@OpenWrt:/# cat /sys/module/helloworld/sections/.text
0xffffffc00081c000

Do the same with others segments: .bss, .data, .rodata ...
After that, we load these addresses accordingly to the gdb
The entrypoint of a kernel module is __init function. After loading correct address of each segment to gdb, only then you could set breakpoint to __int function, a stop at very entrypoint of kernel module

-> These are the main goals of this MR:

1. Read and find correct address of kernel module and load it to gdb
2. Do debug kernel module on int (stop at entrypoint)
3. Test functionality of kernel module functions

[QuangNguyenMinh123]

Demo video: https://youtu.be/jpcxMeApOQU
Ready for review