gnu/stub-32.h required on amd64

[nela]

Hello, thanks for these examples to help newbies like me get into BPF!

Issue

I am trying to write some uprobes for tracing nginx and am struggling to build the bpf program. More specifically, when including nginx header files in order to parse data, the build does not register my 64-bit architecture and wants to include gnu/stubs-32.h. Building without ngx_http.h works fine. This may not be specifically a libbpf issue, however I do not have the same problems with bpftrace or BCC where inclusion works without problems.

Any suggestions as to how to fix this is greatly appreciated.

Code

nginx.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include "ngx_http.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("uprobe//usr/sbin/nginx:ngx_http_finalize_request")
int handle_ngx_http_finalize_request(struct pt_regs* ctx)
{
    int pid = bpf_get_current_pid_tgid() >> 32;
    bpf_printk("Is this getting triggered? %d.\n", pid);
    return 0;
}

nginx.c

#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>
#include <bpf/libbpf.h>
#include "nginx.skel.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

int main(int argc, char **argv)
{
    struct nginx_bpf *skel;
    int err;

    libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
    libbpf_set_print(libbpf_print_fn);

    skel = nginx_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

    err = nginx_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "Failed to auto-attach BPF skeleton: %d\n", err);
        goto cleanup;
    }

    printf("Successfully started!\n");

    for (;;) {
        sleep(1);
    }

cleanup:
    nginx_bpf__destroy(skel);
    return -err;
}

error message

❯ make
  BPF      .output/nginx.bpf.o
In file included from nginx.bpf.c:4:
In file included from /home/nela/master/nginx-1.18.0/src/http/ngx_http.h:12:
In file included from /home/nela/master/nginx-1.18.0/src/core/ngx_config.h:26:
In file included from /home/nela/master/nginx-1.18.0/src/os/unix/ngx_linux_config.h:18:
In file included from /usr/include/x86_64-linux-gnu/sys/types.h:25:
In file included from /usr/include/features.h:510:
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:12: fatal error: 'gnu/stubs-32.h' file not found
 # include <gnu/stubs-32.h>
           ^~~~~~~~~~~~~~~~
1 error generated.
make: *** [Makefile:106: .output/nginx.bpf.o] Error 1

System Information

❯ echo $CPATH
/home/nela/master/nginx-1.18.0/src/http/modules:/home/nela/master/nginx-1.18.0/src/http/v2:/home/nela/master/nginx-1.18.0/src/http:/home/nela/master/nginx-1.18.0/src/event:/home/nela/master/nginx-1.18.0/src/core:/home/nela/master/nginx-1.18.0/src:/home/nela/master/nginx-1.18.0/src/os/unix:/home/nela/master/nginx-1.18.0/objs:/home/nela/master/nginx-1.18.0:/usr/src/linux-headers-5.15.0-56-generic/include:/usr/src/linux-headers-5.15.0-56/include:/usr/include

❯ uname -a 
Linux nelasus 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

❯ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

[anakryiko]

Avoid including sys/types.h would be my recommendation. As you said, it's not libbpf problem. BPF-side code can't include a lot of system-wide headers.

[nela]

Interesting. This seems to limit the capabilities of uprobes quite a lot? How is this achieved in bpftrace? The following certainly works:

#include "/usr/include/stdint.h"
#include "ngx_http.h"

uprobe:/usr/sbin/nginx:ngx_http_finalize_request
{
    $req = (struct ngx_http_request_s *)arg0;
    printf("Req line: %s\n", str($req->request_line.data));
}

I did manage to build the program by forcing it to use gnu/stubs-64.h (by commenting out conditionals, so not really a solution). However, I had quite a lot of problems trying to do something as simple as bpftrace script as above. I have posted the issue on StackOverflow, but essentially I was unable to access the data of the first function argument of ngx_http_finalize_request function. Could that as well be due to type collisions between system and kernel?

PS: Read all the posts in your blog! Very informative - looking forward to reading more!

[anakryiko]

Interesting. This seems to limit the capabilities of uprobes quite a lot? How is this achieved in bpftrace? The following certainly works:

I don't know what sort of magic bpftrace do. Maybe they substitute empty gnu/stubs-32.h or something to avoid this error. Sorry, I don't have a solution.

In practice people have a copy/pasted type definitions they are working with. They don't use real headers from some other application. This also allows to support multiple versions of application using single BPF program, as you can have multiple incompatible versions of (in this case nginx's) types.

#include "/usr/include/stdint.h"
#include "ngx_http.h"

uprobe:/usr/sbin/nginx:ngx_http_finalize_request
{
    $req = (struct ngx_http_request_s *)arg0;
    printf("Req line: %s\n", str($req->request_line.data));
}

I did manage to build the program by forcing it to use gnu/stubs-64.h (by commenting out conditionals, so not really a solution). However, I had quite a lot of problems trying to do something as simple as bpftrace script as above. I have posted the issue on StackOverflow, but essentially I was unable to access the data of the first function argument of ngx_http_finalize_request function. Could that as well be due to type collisions between system and kernel?

No, you are not defining uprobe programs correctly. I'll reply on StackOverflow, if you already asked question there.

PS: Read all the posts in your blog! Very informative - looking forward to reading more!

Thanks! I'm glad you found it useful!

[anakryiko]

https://stackoverflow.com/a/74736806/14352337 for the answer to your follow up question.

[nela]

Alright, thanks for taking the time to reply.