In https://lore.kernel.org/bpf/20260326145444.2076244-1-alan.maguire@oracle.com/ BTF layout support was added to the kernel and libbpf. pahole added support in https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=178b76972b1279c062dbc2c48a0b306cc33552b2.
BTF had a compatibility problem when it comes to parsing types. Each type has a common header which lists its "kind" and an optional "vlen" among other info. This common header is then optionally followed by a kind specific struct and/or an array kind specific structs with length of vlen.
This means that the parser needs to understand every "kind" in order to proceed with parsing. When an unknown "kind" of BTF type is encountered we have to bail out because we do not know how many bytes to consume.
BTF layouts fixes this. A BTF encoder can optionally include layouts which is an array which tells us how many bytes to always consume after the common header and how many bytes to consume per vlen. The index of the array being equal to the kind value. This means that even if we do not understand what the bytes of a BTF type means, we should still be able to proceed parsing the BTF and continue functioning in some capacity.
This seems like a useful feature to support since it should make our library more forwards compatible. Meaning our library will be able to work with newer future kernels without requiring mandatory patching just to keep working when new BTF types are added.
We need to decide to which level we want to support / tolerate unknown types.
There are three levels of "support", listed here in increasing order of effort / refactoring / redesign required:
- We tolerate unknown BTF kinds using the layout info. We parse BTF containing them, but users are unable to use unknown types. We do not marshal them in the kernel facing BTF and querying them via the BTF spec will result in errors.
- We add an opaque type for "unknown" kinds which stores all info needed to go from the parser to the BTF builder / encoder. Our builder is able to emit BTF with layout info for kernels that support it. This means that "unknown" types can be part of the type graph for map key/values or CO-RE relocations, however, since we do not understand them, this only work for leaf types without children. If an "unknown" type requires valid pointers to other types, the emited BTF will be wrong.
- When an "unknown" type is part of the type graph for BTF we wish to load into the kernel we load the full BTF spec as it was presented to us in the ELF. This in contrast to the current situation where we marshal the bare minimum. By preserving all types with the exact same IDs as during the original encoding we ensure that any pointers encoded in "unknown" types stay valid.
Options 1 and 2 are fairly easy to implement. But option 3 is a bit of a departure from our current way of doing BTF. On the one hand our intent is to always update the library to understand newly added BTF kinds as soon as we can. On the other hand the whole point of layouts is that older parsers/loaders can continue to work without needing library updates.
Would like to hear some other opinions as well.
In https://lore.kernel.org/bpf/20260326145444.2076244-1-alan.maguire@oracle.com/ BTF layout support was added to the kernel and libbpf. pahole added support in https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=178b76972b1279c062dbc2c48a0b306cc33552b2.
BTF had a compatibility problem when it comes to parsing types. Each type has a common header which lists its "kind" and an optional "vlen" among other info. This common header is then optionally followed by a kind specific struct and/or an array kind specific structs with length of vlen.
This means that the parser needs to understand every "kind" in order to proceed with parsing. When an unknown "kind" of BTF type is encountered we have to bail out because we do not know how many bytes to consume.
BTF layouts fixes this. A BTF encoder can optionally include layouts which is an array which tells us how many bytes to always consume after the common header and how many bytes to consume per vlen. The index of the array being equal to the kind value. This means that even if we do not understand what the bytes of a BTF type means, we should still be able to proceed parsing the BTF and continue functioning in some capacity.
This seems like a useful feature to support since it should make our library more forwards compatible. Meaning our library will be able to work with newer future kernels without requiring mandatory patching just to keep working when new BTF types are added.
We need to decide to which level we want to support / tolerate unknown types.
There are three levels of "support", listed here in increasing order of effort / refactoring / redesign required:
Options 1 and 2 are fairly easy to implement. But option 3 is a bit of a departure from our current way of doing BTF. On the one hand our intent is to always update the library to understand newly added BTF kinds as soon as we can. On the other hand the whole point of layouts is that older parsers/loaders can continue to work without needing library updates.
Would like to hear some other opinions as well.