feat(aya): add support for map-of-maps (HashOfMaps, ArrayOfMaps)#1446
feat(aya): add support for map-of-maps (HashOfMaps, ArrayOfMaps)#1446Brskt wants to merge 10 commits intoaya-rs:mainfrom
Conversation
✅ Deploy Preview for aya-rs-docs ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@codex review |
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive support for BPF map-of-maps (BPF_MAP_TYPE_HASH_OF_MAPS and BPF_MAP_TYPE_ARRAY_OF_MAPS) to the Aya framework, building upon the foundation from PR #70.
Changes:
- Added
innerattribute to#[map]macro for declaring map-of-maps templates in eBPF code - Implemented
HashMapOfMapsandArrayOfMapstypes withget(),iter(), and other helper methods - Added support for program array population via
EbpfLoader::set_prog_array_entry()andEbpf::populate_prog_arrays()
Reviewed changes
Copilot reviewed 43 out of 43 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| xtask/public-api/aya.txt | Updated public API surface with new map-of-maps types and test run functionality |
| xtask/public-api/aya-obj.txt | Added inner map bindings and map creation helpers to object parser API |
| xtask/public-api/aya-ebpf.txt | Introduced InnerMap trait and map-of-maps types for eBPF side |
| test/integration-test/src/tests/prog_array.rs | Added integration tests for program array population |
| test/integration-test/src/tests/map_of_maps.rs | Added integration tests for map-of-maps functionality |
| test/integration-ebpf/src/prog_array.rs | eBPF test program for tail calls using program arrays |
| test/integration-ebpf/src/map_of_maps.rs | eBPF test program demonstrating map-of-maps usage |
| ebpf/aya-ebpf/src/maps/*.rs | Implemented InnerMap trait across all compatible map types |
| aya/src/sys/bpf.rs | Added inner_map_fd parameter to map creation and test run functionality |
| aya/src/maps/of_maps/*.rs | Implemented HashMapOfMaps and ArrayOfMaps with iterators |
| aya/src/maps/mod.rs | Added map-of-maps variants to Map enum and error handling |
| aya/src/bpf.rs | Enhanced loader to handle map-of-maps creation and program array population |
| aya-obj/src/obj.rs | Added parsing for .maps.inner section and inner map bindings |
| aya-obj/src/maps.rs | Extended map definitions with inner map support and helper constructors |
| aya-ebpf-macros/src/map.rs | Implemented inner attribute processing in map macro |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f2593b39d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
aya/src/maps/of_maps/hash_map.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, key, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
There was a problem hiding this comment.
Here get() treats the value returned by bpf_map_lookup_elem as a map ID and calls bpf_map_get_fd_by_id, but this API inserts raw map FDs (insert passes value.as_fd().as_raw_fd()), so the lookup is expected to return an FD in common map-in-map setups. In that case this path will fail (EINVAL/ENOENT) or open a different map whose ID happens to match the FD integer. Consider constructing MapData directly from the returned FD (or otherwise aligning with the stored value type) instead of resolving it as an ID.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This implementation is correct. For map-of-maps types, the kernel uses an asymmetric API:
- Update (
bpf_map_update_elem): expects the FD of the inner map - Lookup (
bpf_map_lookup_elem): returns the ID of the inner map
This is documented behavior in the Linux kernel: https://docs.kernel.org/bpf/map_of_maps.html
The lookup value must be converted to an FD using bpf_map_get_fd_by_id, which is exactly what this code does
There was a problem hiding this comment.
Perhaps this deserves some inline comments?
aya/src/maps/of_maps/array.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, index, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
There was a problem hiding this comment.
This get() path interprets the lookup value as a map ID and calls bpf_map_get_fd_by_id, but set() stores raw map FDs in the outer array. If the kernel returns the stored FD (as it commonly does for map-in-map values), bpf_map_get_fd_by_id will fail or resolve the wrong map. Using MapData::from_fd on the returned value would keep the value interpretation consistent with set().
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same as above - this is the correct behavior. The kernel's map-of-maps syscall API is asymmetric by design:
BPF_MAP_UPDATE_ELEMtakes an FDBPF_MAP_LOOKUP_ELEMreturns an ID
See: https://docs.ebpf.io/linux/map-type/BPF_MAP_TYPE_ARRAY_OF_MAPS/
Using bpf_map_get_fd_by_id(id) to convert the returned ID to an FD is the intended pattern.
There was a problem hiding this comment.
Perhaps this deserves some inline comments?
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird made 3 comments.
Reviewable status: 0 of 43 files reviewed, 3 unresolved discussions (waiting on @Brskt).
-- commits line 70 at r35:
The commits in this PR are mostly a mess, but e.g. this one looks useful on its own. Did you intend for the commits history to be preserved? If yes, we will need you to rewrite it into something coherent. If not, then this PR is 3k lines that have to be reviewed in one shot, which is quite difficult.
Code quote:
New commits in r8 on 1/17/2026 at 4:21 PM:
- d1f0cb8: feat(aya): Add prog_array population support for tail calls
Add EbpfLoader::set_prog_array_entry() to declaratively specify which
programs should be placed in program arrays at which indices.
Add Ebpf::populate_prog_arrays() to populate the declared entries with
loaded program file descriptors after programs are loaded.
This enables easier setup of tail call jump tables without manually
managing program array entries.
aya/src/maps/of_maps/array.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, index, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
There was a problem hiding this comment.
Perhaps this deserves some inline comments?
aya/src/maps/of_maps/hash_map.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, key, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
There was a problem hiding this comment.
Perhaps this deserves some inline comments?
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
f2593b3 to
333e272
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 3 comments and resolved 2 discussions.
Reviewable status: 0 of 43 files reviewed, 1 unresolved discussion (waiting on @tamird).
Previously, tamird (Tamir Duberstein) wrote…
The commits in this PR are mostly a mess, but e.g. this one looks useful on its own. Did you intend for the commits history to be preserved? If yes, we will need you to rewrite it into something coherent. If not, then this PR is 3k lines that have to be reviewed in one shot, which is quite difficult.
Yes, I've kept the commit history and rewritten it as requested:
- aya-ebpf: eBPF-side map-of-maps implementation
- aya-ebpf-macros: inner attribute for #[map] macro
- aya-obj: Map constructors and .maps.inner parsing
- aya: userspace map-of-maps support
- tests: integration and unit tests
- public API updates
Should be easier to review now.
aya/src/maps/of_maps/array.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, index, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
aya/src/maps/of_maps/hash_map.rs
Outdated
| let value: Option<u32> = | ||
| bpf_map_lookup_elem(fd, key, flags).map_err(|io_error| SyscallError { | ||
| call: "bpf_map_lookup_elem", | ||
| io_error, | ||
| })?; | ||
| if let Some(id) = value { | ||
| let inner_fd = bpf_map_get_fd_by_id(id)?; | ||
| let info = MapInfo::new_from_fd(inner_fd.as_fd())?; | ||
| let map_data = MapData::from_id(info.id())?; |
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird partially reviewed 15 files and made 2 comments.
Reviewable status: 1 of 43 files reviewed, 2 unresolved discussions (waiting on @Brskt).
Previously, Brskt wrote…
Yes, I've kept the commit history and rewritten it as requested:
- aya-ebpf: eBPF-side map-of-maps implementation
- aya-ebpf-macros: inner attribute for #[map] macro
- aya-obj: Map constructors and .maps.inner parsing
- aya: userspace map-of-maps support
- tests: integration and unit tests
- public API updates
Should be easier to review now.
it's still just one big blob, right? the commits are now cut along which crates they touch, which is maybe easier for review but they need to be squashed on merge. do I understand correctly?
ebpf/aya-ebpf/src/maps/mod.rs line 47 at r36 (raw file):
/// /// Only implement this trait for map types that can be safely used as inner maps. pub unsafe trait InnerMap {}
🤔 does this need to be pub?
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 2 comments.
Reviewable status: 1 of 43 files reviewed, 2 unresolved discussions (waiting on @tamird).
Previously, tamird (Tamir Duberstein) wrote…
it's still just one big blob, right? the commits are now cut along which crates they touch, which is maybe easier for review but they need to be squashed on merge. do I understand correctly?
Yes, that's correct. Split for easier review, feel free to squash on merge.
ebpf/aya-ebpf/src/maps/mod.rs line 47 at r36 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
🤔 does this need to be pub?
I tested with pub(crate) and here's what happens:
Build results:
cargo build:⚠️ 3 warnings (private_bounds)cargo clippy -D warnings: ❌ 3 errors - fails CIcargo test: ✅ passintegration tests: ✅ 127 passedpublic-api check: ❌ fails (InnerMap removed from API)
Why it fails:
InnerMap is used as a trait bound on public types:
pub struct ArrayOfMaps<T: InnerMap> { ... }A private trait in a public bound triggers private_bounds, which becomes an error with -D warnings.
Conclusion:
pub is required to pass CI. The unsafe marker already discourages external implementations, and the kernel validates map types at load time anyway.
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird partially reviewed 42 files, made 8 comments, and resolved 1 discussion.
Reviewable status: 19 of 43 files reviewed, 8 unresolved discussions (waiting on @Brskt).
ebpf/aya-ebpf/src/maps/mod.rs line 47 at r36 (raw file):
Previously, Brskt wrote…
I tested with
pub(crate)and here's what happens:Build results:
cargo build:⚠️ 3 warnings (private_bounds)cargo clippy -D warnings: ❌ 3 errors - fails CIcargo test: ✅ passintegration tests: ✅ 127 passedpublic-api check: ❌ fails (InnerMap removed from API)Why it fails:
InnerMapis used as a trait bound on public types:pub struct ArrayOfMaps<T: InnerMap> { ... }A private trait in a public bound triggers private_bounds, which becomes an error with -D warnings.
Conclusion:
pub is required to pass CI. The unsafe marker already discourages external implementations, and the kernel validates map types at load time anyway.
I see. This should be a sealed trait then since we don't want external implementations.
-- commits line 25 at r38:
this commit is ...bad. it's adding a bunch of code that is unused, making review impossible.
ebpf/aya-ebpf/src/maps/hash_of_maps.rs line 14 at r36 (raw file):
pub struct HashOfMaps<K, V> { def: UnsafeCell<bpf_map_def>, _k: PhantomData<K>,
See #1447; use a single phantom plz
aya-ebpf-macros/src/map.rs line 20 at r37 (raw file):
let mut args = syn::parse2(attrs)?; let name = name_arg(&mut args).unwrap_or_else(|| item.ident.to_string()); let inner = pop_string_arg(&mut args, "inner");
while you're here, please add err_on_unknown_args(args)?; (see #1448)
aya-ebpf-macros/src/map.rs line 40 at r37 (raw file):
#[used] static #binding_ident: [u8; #binding_len] = [#(#binding_bytes),*]; }
are we following libbpf conventions here? needs citations
Code quote:
// Create a unique identifier for the binding
let binding_ident = format_ident!("__inner_map_binding_{}", name);
// Format: "outer_name\0inner_name\0" (null-terminated strings)
let binding_value = format!("{}\0{}\0", name, inner);
let binding_len = binding_value.len();
let binding_bytes = binding_value.as_bytes();
quote! {
#[unsafe(link_section = ".maps.inner")]
#[used]
static #binding_ident: [u8; #binding_len] = [#(#binding_bytes),*];
}aya-ebpf-macros/src/map.rs line 42 at r37 (raw file):
} } else { quote! {}
we can drop this b/c Options impls ToTokens
https://docs.rs/quote/latest/quote/trait.ToTokens.html#impl-ToTokens-for-Option%3CT%3E
aya-ebpf-macros/src/map.rs line 118 at r37 (raw file):
); // "OUTER\0INNER_TEMPLATE\0" = 21 bytes assert!(expanded_str.contains("21usize"), "expected 21 bytes");
these assertions are problematic because they emit no information on failure
Code quote:
assert!(
expanded_str.contains(".maps.inner"),
"expected .maps.inner section"
);
assert!(
expanded_str.contains("__inner_map_binding_OUTER"),
"expected binding identifier"
);
// "OUTER\0INNER_TEMPLATE\0" = 21 bytes
assert!(expanded_str.contains("21usize"), "expected 21 bytes");ebpf/aya-ebpf/src/maps/array_of_maps.rs line 19 at r36 (raw file):
unsafe impl<T: InnerMap> Sync for ArrayOfMaps<T> {} impl<T: InnerMap> ArrayOfMaps<T> {
let's reduce some of this boilerplate, see #1447
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
333e272 to
60f6d7c
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 8 comments.
Reviewable status: 19 of 43 files reviewed, 8 unresolved discussions (waiting on @tamird).
Previously, tamird (Tamir Duberstein) wrote…
this commit is ...bad. it's adding a bunch of code that is unused, making review impossible.
Done, is this the way u wanted ?
aya-ebpf-macros/src/map.rs line 20 at r37 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
while you're here, please add
err_on_unknown_args(args)?;(see #1448)
Done.
aya-ebpf-macros/src/map.rs line 40 at r37 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
are we following libbpf conventions here? needs citations
No, this is not following libbpf conventions. Documentation has been added to clarify this.
libbpf uses BTF relocations within the .maps section for inner map bindings (see https://patchwork.ozlabs.org/comment/2418417/), where u declare .values = { [0] = &inner_map, ... } and libbpf processes the relocations.
The .maps.inner section is an aya-specific mechanism. This approach was chosen because:
- aya-ebpf doesn't require BTF for map definitions
- It provides a simpler mechanism that works with both legacy and BTF-style maps
The format is now documented in both aya-ebpf-macros/src/map.rs and aya-obj/src/obj.rs with references to the libbpf implementation.
aya-ebpf-macros/src/map.rs line 42 at r37 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
we can drop this b/c Options impls ToTokens
https://docs.rs/quote/latest/quote/trait.ToTokens.html#impl-ToTokens-for-Option%3CT%3E
Done.
aya-ebpf-macros/src/map.rs line 118 at r37 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
these assertions are problematic because they emit no information on failure
Done.
ebpf/aya-ebpf/src/maps/array_of_maps.rs line 19 at r36 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
let's reduce some of this boilerplate, see #1447
Acknowledged. This PR can be rebased on top of #1447 once it's merged to use the MapDef abstraction, which will eliminate the duplicated UnsafeCell<bpf_map_def> wrapper, unsafe impl Sync, and constructor boilerplate.
Should I wait for #1447 to land first, or would you prefer I implement a similar pattern in this PR?
ebpf/aya-ebpf/src/maps/hash_of_maps.rs line 14 at r36 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
See #1447; use a single phantom plz
Done.
ebpf/aya-ebpf/src/maps/mod.rs line 47 at r36 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
I see. This should be a sealed trait then since we don't want external implementations.
Done.
Each map type now implements Sealed (e.g., impl<T> Sealed for Array<T> {}), preventing external implementations while keeping InnerMap public to satisfy the trait bounds on ArrayOfMaps<T: InnerMap> and HashOfMaps<K, V: InnerMap>.
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
60f6d7c to
581a00e
Compare
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird partially reviewed 26 files, made 3 comments, and resolved 3 discussions.
Reviewable status: 3 of 43 files reviewed, 7 unresolved discussions (waiting on @Brskt).
aya-ebpf-macros/src/map.rs line 40 at r37 (raw file):
Previously, Brskt wrote…
No, this is not following libbpf conventions. Documentation has been added to clarify this.
libbpf uses BTF relocations within the .maps section for inner map bindings (see https://patchwork.ozlabs.org/comment/2418417/), where u declare
.values = { [0] = &inner_map, ... }and libbpf processes the relocations.The .maps.inner section is an aya-specific mechanism. This approach was chosen because:
- aya-ebpf doesn't require BTF for map definitions
- It provides a simpler mechanism that works with both legacy and BTF-style maps
The format is now documented in both aya-ebpf-macros/src/map.rs and aya-obj/src/obj.rs with references to the libbpf implementation.
Doesn't this mean that libbpf can't load aya programs that use map-in-map, and vice versa? That's generally not the approach we have taken.
A better link: torvalds/linux@646f02ffdd49
aya-ebpf-macros/src/map.rs line 106 at r47 (raw file):
#[test] fn test_map_with_inner() {
the tests above check for the exact generated code, can we follow the same style? if not, please add a comment explaining why
aya-ebpf-macros/src/map.rs line 171 at r47 (raw file):
), ); assert!(result.is_err());
pretty weak assertion
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
581a00e to
cf9be0a
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 3 comments.
Reviewable status: 2 of 48 files reviewed, 7 unresolved discussions (waiting on @tamird).
aya-ebpf-macros/src/map.rs line 40 at r37 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
Doesn't this mean that libbpf can't load aya programs that use map-in-map, and vice versa? That's generally not the approach we have taken.
A better link: torvalds/linux@646f02ffdd49
Done. #[btf_map] with btf_maps::ArrayOfMaps/HashOfMaps now works with both aya and libbpf loaders (tested both). Uses [*const V; 0] for the values field per the BTF relocation format libbpf expects.
Legacy #[map(inner = "...")] remains aya-specific but is now documented as such.
Does this address your concern ?
aya-ebpf-macros/src/map.rs line 106 at r47 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
the tests above check for the exact generated code, can we follow the same style? if not, please add a comment explaining why
Done.
aya-ebpf-macros/src/map.rs line 171 at r47 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
pretty weak assertion
Done.
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird partially reviewed 41 files and made 2 comments.
Reviewable status: 2 of 48 files reviewed, 9 unresolved discussions (waiting on @Brskt).
ebpf/aya-ebpf/src/btf_maps/array.rs line 11 at r53 (raw file):
/// /// This map type stores elements of type `T` indexed by `u32` keys. /// The struct layout is designed to be compatible with both aya and libbpf loaders.
what does that mean?
ebpf/aya-ebpf/src/btf_maps/array.rs line 23 at r53 (raw file):
#[repr(C)] #[allow(dead_code)] pub struct Array<T, const M: usize, const F: usize = 0> {
why did we need to toss bpf_map_def!?
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
cf9be0a to
fd9cb5b
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 2 comments.
Reviewable status: 2 of 48 files reviewed, 9 unresolved discussions (waiting on @tamird).
ebpf/aya-ebpf/src/btf_maps/array.rs line 11 at r53 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
what does that mean?
I've improved the comments. Is it clearer now ?
ebpf/aya-ebpf/src/btf_maps/array.rs line 23 at r53 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why did we need to toss
bpf_map_def!?
The existing btf_maps that use btf_map_def! (RingBuf, SkStorage) aren't libbpf-compatible either - they only work with aya's loader. For this PR, you requested that map-of-maps be loadable by both aya and libbpf, so I used flat #[repr(C)] structs instead.
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
fd9cb5b to
0e4c970
Compare
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird made 1 comment.
Reviewable status: 2 of 48 files reviewed, 9 unresolved discussions (waiting on @Brskt).
ebpf/aya-ebpf/src/btf_maps/array.rs line 23 at r53 (raw file):
Previously, Brskt wrote…
The existing btf_maps that use
btf_map_def!(RingBuf, SkStorage) aren't libbpf-compatible either - they only work with aya's loader. For this PR, you requested that map-of-maps be loadable by both aya and libbpf, so I used flat#[repr(C)]structs instead.
Ah, yeah this is also #1455. Would you be willing to send a separate PR to fix that for all the maps?
|
Previously, tamird (Tamir Duberstein) wrote…
Or at least a separate commit would be great. |
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 1 comment.
Reviewable status: 2 of 48 files reviewed, 9 unresolved discussions (waiting on @tamird and @vadorovsky).
ebpf/aya-ebpf/src/btf_maps/array.rs line 23 at r53 (raw file):
Previously, vadorovsky (Michal R) wrote…
Or at least a separate commit would be great.
Done in #1457
Brskt
force-pushed
the
hashmapofmaps-new
branch
5 times, most recently
from
c3d2e4a to
eef6947
Compare
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
4a936a3 to
f15ade0
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 22 comments.
Reviewable status: 17 of 50 files reviewed, 22 unresolved discussions (waiting on @tamird and @vadorovsky).
aya/src/bpf.rs line 130 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
do we like this name? it isn't very descriptive of what's stored in here nor what it's used for.
I renamed to pending_prog_array_inserts. if this better for you.
aya/src/bpf.rs line 224 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
inconsistent text width
Done.
aya/src/bpf.rs line 247 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
can you help me understand how this functionality fits into the map-of-maps work in this PR?
Yikes, this is unrelated to map-of-maps. Removed it entirely.
aya/src/bpf.rs line 542 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
useless comment
Done.
aya/src/bpf.rs line 564 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
i dont understand what's going on here
Yeah that heuristic was nonsense, replaced it with an explicit error if no inner binding is provided.
aya/src/bpf.rs line 566 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this comment would be more helpful if it explained why
The kernel needs inner_map_fd when creating a map-of-maps, so we create regular maps first. Updated the comment.
aya/src/bpf.rs line 656 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this code is very similar to the block above. can we find a way to avoid the duplication?
Unified into a single loop using chain(). The only difference is resolving inner_map_fd for map-of-maps, None for the rest.
aya/src/bpf.rs line 862 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
??
Removed.
aya/src/maps/mod.rs line 107 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
lol wat
Yeah fair enough, replaced with MapError::MissingInnerMapBinding.
aya/src/maps/array/array.rs line 97 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why is this a separate impl block from the one above?
It needs T = MapData concretely since create constructs an owned map via the syscall (the equivalent of bpf_map_create() in libbpf for dynamically creating inner maps at runtime). Can't go in the generic T: Borrow<MapData> block.
aya/src/maps/hash_map/hash_map.rs line 104 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
ditto, why a separate impl block with a different bound?
Same as above.
aya/src/maps/of_maps/array.rs line 15 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
period?
Done.
aya/src/maps/of_maps/array.rs line 60 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
can you help me understand how this knows to return an Array? we don't know the inner map type, do we?
You're right, we don't. get() got rewored to be generic over M: FromMapData so the caller picks the inner map type. Validation still happens at runtime through each map's constructor. Applied the same to HashMapOfMaps::get() and the iterators.
aya/src/maps/of_maps/array.rs line 89 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
do we need this? seems like we can get by without.
Right, for arrays you can just iterate by index with get(). Removed it.
aya/src/maps/of_maps/hash_map.rs line 13 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
ditto
Done.
aya/src/maps/of_maps/hash_map.rs line 47 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
same question here
Done with the get() generic.
aya/src/maps/of_maps/hash_map.rs line 76 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
i think we can perhaps skip these
After some research, yup, we can drop iter() and the iterator type. Kept keys() though since it's just generic BPF_MAP_GET_NEXT_KEY iteration and it's the only way to discover existing keys in the map.
aya/src/maps/of_maps/mod.rs line 1 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
what's with all these sentence fragments?
Done.
aya/src/sys/bpf.rs line 599 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
i'm confused about this being in the diff. isn't this #1461?
Removed.
ebpf/aya-ebpf/src/btf_maps/mod.rs line 48 at r96 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
please revert these spurious whitespace changes
Restored.
ebpf/aya-ebpf/src/btf_maps/mod.rs line 126 at r96 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
did this need to move? looking at the diff of this file i see a number of spurious whitespace changes + this function moving from the bottom of this arm to the top.
No it didn't, moved it back. Whitespace restored too.
aya/src/programs/mod.rs line 128 at r102 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
period!
Done.
|
|
||
| // Inner map template - must be declared before outer map | ||
| #[btf_map] | ||
| static INNER: Array<u32, 10> = Array::new(); |
There was a problem hiding this comment.
this is only consumed by the macro right?
couldn't we have #[btf_map(of_map)] or something and parse the
inner type from the outer definition, so we don't need this extra ghost definition?
There was a problem hiding this comment.
I see what you are talking about, since we declare the map in the OUTER, there is no need to put an INNER map.
Brskt
changed the title
|
I'm testing this change here. Seems to work ok functionally but saw a 10x slow down (10s vs 1s) in ebpf verification against my array-of-maps implementation, I'm digging a little into why.. |
I tried on my end and see no slowdown at all; the maps, loading or executing the program. Can u describe more? |
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 29 files and all commit messages, made 11 comments, and resolved 22 discussions.
Reviewable status: 42 of 51 files reviewed, 11 unresolved discussions (waiting on alessandrod, Brskt, and vadorovsky).
aya/src/bpf.rs line 130 at r102 (raw file):
Previously, Brskt wrote…
I renamed to
pending_prog_array_inserts.if this better for you.
Looks like it's just removed. Great.
aya/src/bpf.rs line 530 at r108 (raw file):
// Create regular maps first, then map-of-maps. The kernel requires // inner_map_fd when creating map-of-maps, so inner maps must exist first.
can you word this better? it's repeating the comment above in large part. this is classic LLM slop =/
Code quote:
// Create regular maps first, then map-of-maps. The kernel requires
// inner_map_fd when creating map-of-maps, so inner maps must exist first.aya/src/bpf.rs line 554 at r108 (raw file):
let btf_fd = btf_fd.as_deref().map(|fd| fd.as_fd()); // For BTF map-of-maps, create a temporary inner map from the definition
how do you know if it's a BTF map-of-maps or not?
aya/src/bpf.rs line 557 at r108 (raw file):
// embedded in the `values` BTF field. The temp map provides the fd that the // kernel needs during outer map creation; it is dropped afterwards. let btf_inner_map = if is_map_of_maps(map_type) {
this is a weird check -- you know this is true for maps_of_maps
you can do
for ((name, mut map_obj), is_map_of_maps) in regular_maps.into_iter().zip(iter::repeat(false)).chain(....zip(repeat(true))
aya/src/maps/mod.rs line 117 at r108 (raw file):
mod sealed { #[expect(unnameable_types, reason = "intentionally unnameable sealed trait")] pub trait Sealed {}
since this is the trait you end up implementing, it should have a descriptive name
blanket impl for the outer trait over the sealed trait completes the pattern
aya/src/maps/array/array.rs line 116 at r108 (raw file):
/// # Ok::<(), aya::maps::MapError>(()) /// ``` pub fn create(max_entries: u32, flags: u32) -> Result<Self, MapError> {
does this need to be pub? ditto below
aya/src/maps/hash_map/hash_map.rs line 123 at r108 (raw file):
/// # Ok::<(), aya::maps::MapError>(()) /// ``` pub fn create(max_entries: u32, flags: u32) -> Result<Self, MapError> {
do these need to be pub?
also, won't we need these additions for every inner map type? i'm surprised only array and hashmap are getting them. what am i missing?
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
/// Returns [`MapError::OutOfBounds`] if `index` is out of bounds, [`MapError::SyscallError`] /// if `bpf_map_update_elem` fails. pub fn set(&mut self, index: u32, value: &MapFd, flags: u64) -> Result<(), MapError> {
allowing an arbitrary mapfd here seems bad?
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
self.inner.fd() } }
do these need to be pub?
Code quote:
impl ArrayOfMaps<MapData> {
/// Returns a reference to the underlying [`MapData`].
pub const fn map_data(&self) -> &MapData {
&self.inner
}
/// Returns a file descriptor reference to the underlying map.
pub const fn fd(&self) -> &MapFd {
self.inner.fd()
}
}aya/src/maps/of_maps/hash_map.rs line 76 at r108 (raw file):
&mut self, key: impl Borrow<K>, value: &MapFd,
this seems wrong
aya/src/maps/of_maps/hash_map.rs line 103 at r108 (raw file):
self.inner.fd() } }
why pub?
Code quote:
impl<K: Pod> HashOfMaps<MapData, K> {
/// Returns a reference to the underlying [`MapData`].
pub const fn map_data(&self) -> &MapData {
&self.inner
}
/// Returns a file descriptor reference to the underlying map.
pub const fn fd(&self) -> &MapFd {
self.inner.fd()
}
}
it seems that the In my case it seems the the different code structure causes the BPF verifier's state tracking to explode, because I have a binary search loop going over that. I'll share a fix I'll do on my end, and see if there's room for improvement here |
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
593a014 to
5058009
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 10 comments.
Reviewable status: 42 of 51 files reviewed, 11 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).
aya/src/bpf.rs line 530 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
can you word this better? it's repeating the comment above in large part. this is classic LLM slop =/
Reworded, better?
aya/src/bpf.rs line 554 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
how do you know if it's a BTF map-of-maps or not?
We don't actually, reworded the comment to reflect that.
aya/src/bpf.rs line 557 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this is a weird check -- you know this is true for maps_of_maps
you can do
for ((name, mut map_obj), is_map_of_maps) in regular_maps.into_iter().zip(iter::repeat(false)).chain(....zip(repeat(true))
Thanks, done.
aya/src/maps/mod.rs line 117 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
since this is the trait you end up implementing, it should have a descriptive name
blanket impl for the outer trait over the sealed trait completes the pattern
Done. Renamed to sealed::FromMapData with the method, added blanket impl of the outer trait.
aya/src/maps/array/array.rs line 116 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
does this need to be pub? ditto below
Yes, MapData::create is already pub but requires constructing an aya_obj::Map manually. Array::create/HashMap::create are typed convenience wrappers so users don't have to reach into aya-obj to create standalone inner maps.
aya/src/maps/hash_map/hash_map.rs line 123 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
do these need to be pub?
also, won't we need these additions for every inner map type? i'm surprised only array and hashmap are getting them. what am i missing?
Yes, moved both to macros in mod.rs. FromMapData now covers all types the kernel supports as inner maps, create() covers the data-map types (Array, PerCpuArray, HashMap, PerCpuHashMap, LpmTrie, BloomFilter, Queue, Stack). Made new_legacy pub in aya-obj and dropped the redundant new_hash/new_array wrappers.
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
allowing an arbitrary mapfd here seems bad?
This is consistent with the rest of aya - ProgramArray::set() accepts any &ProgramFd, SockMap accepts any AsRawFd. The kernel already validates compatibility at insertion time and returns EINVAL if the inner map doesn't match, so we're safe here.
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
do these need to be pub?
Yes - same pattern as Array, HashMap, and other map types. fd() is needed so users can e.g. pin the map, and map_data() gives access to the underlying metadata.
aya/src/maps/of_maps/hash_map.rs line 76 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this seems wrong
Same as above - consistent with the rest of aya, kernel validates at insertion time.
aya/src/maps/of_maps/hash_map.rs line 103 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why pub?
Same as ArrayOfMaps - consistent with Array, HashMap, and other map types.
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 3 files and all commit messages, made 6 comments, and resolved 6 discussions.
Reviewable status: 40 of 51 files reviewed, 6 unresolved discussions (waiting on alessandrod, Brskt, and vadorovsky).
aya/src/bpf.rs line 554 at r108 (raw file):
Previously, Brskt wrote…
We don't actually, reworded the comment to reflect that.
I still don't understand where this condition ("If this map-of-maps has a BTF values definition") is being evaluated
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
Previously, Brskt wrote…
This is consistent with the rest of aya -
ProgramArray::set()accepts any&ProgramFd,SockMapaccepts anyAsRawFd. The kernel already validates compatibility at insertion time and returnsEINVALif the inner map doesn't match, so we're safe here.
I agree it's safe, but is there any reason not to validate at compile time? SockMap is hard because it's just an FD, I don't know the story with ProgramFd and ProgramArray but here it seems clear we can do better?
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
Previously, Brskt wrote…
Yes - same pattern as
Array,HashMap, and other map types.fd()is needed so users can e.g. pin the map, andmap_data()gives access to the underlying metadata.
Sorry, I'm not seeing it. Where is fd() on Array or HashMap? I do see a function called map that returns &MapData but it's part of the generic block on those maps and has a different name, so this doesn't look consistent to me.
aya/src/maps/of_maps/hash_map.rs line 76 at r108 (raw file):
Previously, Brskt wrote…
Same as above - consistent with the rest of aya, kernel validates at insertion time.
But why would we do that? We know the type, why not check at compile time?
aya/src/maps/of_maps/hash_map.rs line 103 at r108 (raw file):
Previously, Brskt wrote…
Same as
ArrayOfMaps- consistent withArray,HashMap, and other map types.
i don't think it is
aya/src/bpf.rs line 569 at r109 (raw file):
let inner_map_fd = if is_map_of_maps { if let Some(inner) = &btf_inner_map {
this stack of checks makes no sense, btf_inner_map can only be Some if is_map_of_maps is true. so the outer check is useless
Code quote:
let inner_map_fd = if is_map_of_maps {
if let Some(inner) = &btf_inner_map {- Add sealed InnerMap trait; set()/insert() now take &impl InnerMap instead of &MapFd for compile-time validation - Implement InnerMap for all kernel-supported inner map types, MapData, and MapFd - Add pub(crate) map_fd() to PerfEventArray and RingBuf for InnerMap impls (different field layout than other map types) - Remove fd()/map_data() from Array, HashMap; remove fd() from ArrayOfMaps, HashOfMaps - Flatten nested if in bpf.rs inner_map_fd logic - Update integration tests to pass &map instead of map.fd()
Brskt
force-pushed
the
hashmapofmaps-new
branch
from
5058009 to
2b3e706
Compare
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 6 comments.
Reviewable status: 40 of 51 files reviewed, 6 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).
aya/src/bpf.rs line 554 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
I still don't understand where this condition ("If this map-of-maps has a BTF
valuesdefinition") is being evaluated
Reworded.
aya/src/bpf.rs line 569 at r109 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this stack of checks makes no sense, btf_inner_map can only be Some if is_map_of_maps is true. so the outer check is useless
You're right, simplified.
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
I agree it's safe, but is there any reason not to validate at compile time? SockMap is hard because it's just an FD, I don't know the story with ProgramFd and ProgramArray but here it seems clear we can do better?
Done. set() and insert() now take &impl InnerMap where InnerMap is a sealed trait. Only map types the kernel supports as inner maps implement it. MapData and MapFd are also included as escape hatches.
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
Sorry, I'm not seeing it. Where is
fd()onArrayorHashMap? I do see a function calledmapthat returns&MapDatabut it's part of the generic block on those maps and has a different name, so this doesn't look consistent to me.
Sure, removed fd() and map_data() from Array and HashMap. ArrayOfMaps/HashOfMaps keep map_data() since they don't implement IterableMap.
aya/src/maps/of_maps/hash_map.rs line 76 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
But why would we do that? We know the type, why not check at compile time?
Done.
aya/src/maps/of_maps/hash_map.rs line 103 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
i don't think it is
Removed.
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 6 files and all commit messages, made 3 comments, and resolved 3 discussions.
Reviewable status: 42 of 53 files reviewed, 4 unresolved discussions (waiting on alessandrod, Brskt, and vadorovsky).
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
Previously, Brskt wrote…
Done.
set()andinsert()now take&impl InnerMapwhereInnerMapis a sealed trait. Only map types the kernel supports as inner maps implement it.MapDataandMapFdare also included as escape hatches.
But why that and not the specific map type? We know what T is, why can't we just use &T?
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
Previously, Brskt wrote…
Sure, removed
fd()andmap_data()from Array and HashMap. ArrayOfMaps/HashOfMaps keepmap_data()since they don't implementIterableMap.
you didn't remove map_data
aya/src/bpf.rs line 562 at r110 (raw file):
.inner() .map(|inner| MapData::create(inner, &format!("{name}.inner"), btf_fd)) .transpose()?
ah i see what this is doing now. can we write
// The kernel ...
let inner_map = if is_map_of_maps {
// Try using a BTF definition of the inner map.
map_ob.inner().map(|inner| MapData::create(inner, &format!("{name}.inner"), btf_fd)).or_else(|| {
// No BTF inner definition; fall back to the `.maps.inner` binding.
let inner_name = obj.inner_map_binding(&name).ok_or_else(|| {
EbpfError::MapError(MapError::MissingInnerMapBinding { name: name.clone() })
})?;
let inner_map = maps.get(inner_name).ok_or_else(|| {
EbpfError::MapError(MapError::InnerMapNotFound {
name: name.clone(),
inner_name: inner_name.to_owned(),
})
})?;
Ok(inner_map)
}).transpose()?
} else {
None
}something like that?
Code quote:
map_obj
.inner()
.map(|inner| MapData::create(inner, &format!("{name}.inner"), btf_fd))
.transpose()?
@Brskt the workaround I have is in zz85/profile-bee#74 The ArrayOfMaps<Array> results in a 2 step lookup in the call chain which LLVM seem to be generating different/larger BPF instructions. In my use case there's a binary search of up to 17 interaction so the additional instructions overshoots. One suggest for your change would be to perform both the outer and inner lookup in a single call, so the inner map value gets returned without going through the Array::get() indirection. eg. |
Enrich the sealed Map trait with Key and Value associated types so that map-of-maps containers can perform fused two-level lookups without intermediate struct indirection. This reduces BPF verifier state explosion in tight loops. eBPF side: - Add Key/Value to private::Map and public Map with blanket forwarding. - Introduce impl_private_map! macro to replace per-file boilerplate. - Add get_value/get_value_ptr_mut to ArrayOfMaps and HashOfMaps. Userspace side: - Restructure inner map BTF/fallback logic in bpf.rs. - Add V type parameter to ArrayOfMaps and HashOfMaps. - Refactor impl_try_from_map! with @impl internal rule and add impl_try_from_map_of_maps! for unconstrained V.
Test fused lookups on both ArrayOfMaps and HashOfMaps: userspace pre-populates inner maps, the eBPF program reads via get_value and writes via get_value_ptr_mut, then userspace verifies the results.
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 3 comments.
Reviewable status: 42 of 53 files reviewed, 4 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).
aya/src/bpf.rs line 562 at r110 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
ah i see what this is doing now. can we write
// The kernel ... let inner_map = if is_map_of_maps { // Try using a BTF definition of the inner map. map_ob.inner().map(|inner| MapData::create(inner, &format!("{name}.inner"), btf_fd)).or_else(|| { // No BTF inner definition; fall back to the `.maps.inner` binding. let inner_name = obj.inner_map_binding(&name).ok_or_else(|| { EbpfError::MapError(MapError::MissingInnerMapBinding { name: name.clone() }) })?; let inner_map = maps.get(inner_name).ok_or_else(|| { EbpfError::MapError(MapError::InnerMapNotFound { name: name.clone(), inner_name: inner_name.to_owned(), }) })?; Ok(inner_map) }).transpose()? } else { None }something like that?
Done. Used if let instead of or_else since the BTF path produces an owned MapData (for lifetime) while the fallback borrows from maps.
aya/src/maps/of_maps/array.rs line 85 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
But why that and not the specific map type? We know what T is, why can't we just use &T?
Yup, we can. Added V type parameter, set()/insert() now take &V instead of &impl InnerMap.
aya/src/maps/of_maps/array.rs line 109 at r108 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
you didn't remove map_data
Done, removed.
@zz85 Some changes have been made about this; it should be better. |
|
@Brskt - thanks! I can confirm that the |
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 23 files and all commit messages, made 10 comments, and resolved 3 discussions.
Reviewable status: 43 of 53 files reviewed, 11 unresolved discussions (waiting on alessandrod, Brskt, and vadorovsky).
aya/src/bpf.rs line 539 at r96 (raw file):
.unwrap_or_else(|| Path::new("/sys/fs/bpf")); let path = path.join(&name);
nit: restore
aya/src/maps/ring_buf.rs line 115 at r111 (raw file):
impl RingBuf<MapData> { pub(crate) const fn map_fd(&self) -> &MapFd {
why do you need this? most of our maps already define fn map(&self) -> &MapData { which we should probably centralize and then just use to get the FD. I am confused by all the new code that seems to duplicate existing functionality.
aya/src/maps/of_maps/array.rs line 64 at r111 (raw file):
/// /// The inner map type `V` is determined by the type parameter on the /// `ArrayOfMaps` itself. Use `MapData` to retrieve an untyped handle.
why are we telling folks how to get an untyped handle?
aya/src/maps/of_maps/array.rs line 154 at r111 (raw file):
let map = new_map(test_utils::new_obj_map::<u8>(BPF_MAP_TYPE_ARRAY_OF_MAPS)); assert_matches!( ArrayOfMaps::<_, MapData>::new(&map),
this is the default value of the parameter, is it needed? here and everywhere
aya/src/maps/of_maps/array.rs line 251 at r111 (raw file):
fn test_get_not_found() { let map = new_map(new_obj_map()); let arr: ArrayOfMaps<_> = ArrayOfMaps::new(&map).unwrap();
why do you need this ascription?
aya/src/maps/of_maps/hash_map.rs line 60 at r111 (raw file):
/// /// The inner map type `V` is determined by the type parameter on the /// `HashOfMaps` itself. Use `MapData` to retrieve an untyped handle.
ditto
aya/src/maps/of_maps/hash_map.rs line 172 at r111 (raw file):
aya_obj::generated::bpf_map_type::BPF_MAP_TYPE_HASH, )); let mut hm: HashOfMaps<_, u32, MapData> = HashOfMaps::new(&mut map).unwrap();
same question here about all the unnecessary ascription
aya/src/maps/perf/perf_event_array.rs line 187 at r111 (raw file):
impl PerfEventArray<MapData> { pub(crate) fn map_fd(&self) -> &MapFd {
same as the ring_buf comment
ebpf/aya-ebpf/src/maps/mod.rs line 79 at r111 (raw file):
} macro_rules! impl_private_map {
unclear how much this macro helps, though if you move fn map into it, that would be nice (see other comments)
ebpf/aya-ebpf/src/maps/mod.rs line 149 at r111 (raw file):
type Key; /// The value type declared in this map's definition. type Value;
do we need these? we can't just use them through the supertrait?
Code quote:
/// The key type declared in this map's definition.
type Key;
/// The value type declared in this map's definition.
type Value;Replace `map_fd()` with `map_data()` on `RingBuf` and `PerfEventArray`, returning `&MapData`. Update sealed `InnerMap` impls to use `map_data().fd()`. Also clean up `of_maps` docs/tests: - remove untyped-handle wording - remove redundant type ascriptions/default type parameters - use typed literals where inference needs help (`1u32`, `&1u32`)
Remove redundant `Key`/`Value` associated types from the public `Map` trait; they resolve through the sealed `private::Map` supertrait. Drop `impl_private_map!` in favor of explicit `private::Map` impls in map modules, and simplify projections from `<T as Map>::Key` to `T::Key` (and same for `Value`).
Brskt
left a comment
Brskt
left a comment
There was a problem hiding this comment.
@Brskt made 10 comments and resolved 1 discussion.
Reviewable status: 21 of 53 files reviewed, 10 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).
aya/src/bpf.rs line 539 at r96 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
nit: restore
Done.
aya/src/maps/ring_buf.rs line 115 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why do you need this? most of our maps already define
fn map(&self) -> &MapData {which we should probably centralize and then just use to get the FD. I am confused by all the new code that seems to duplicate existing functionality.
Done. Replaced map_fd() with map_data() -> &MapData on both types, which mirrors the IterableMap::map() pattern. They don't implement IterableMap (not iterable), so this fills that gap.
aya/src/maps/of_maps/array.rs line 64 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why are we telling folks how to get an untyped handle?
Should not, removed.
aya/src/maps/of_maps/array.rs line 154 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this is the default value of the parameter, is it needed? here and everywhere
Right, removed the explicit default everywhere.
aya/src/maps/of_maps/array.rs line 251 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
why do you need this ascription?
Removed the local type ascription. Type inference is now driven by typed usage in the test (u32 keys), and I kept explicit type hints only where inference would otherwise be ambiguous.
aya/src/maps/of_maps/hash_map.rs line 60 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
ditto
Done.
aya/src/maps/of_maps/hash_map.rs line 172 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
same question here about all the unnecessary ascription
Done.
aya/src/maps/perf/perf_event_array.rs line 187 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
same as the ring_buf comment
Done.
ebpf/aya-ebpf/src/maps/mod.rs line 79 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
unclear how much this macro helps, though if you move
fn mapinto it, that would be nice (see other comments)
Agreed, reverted. The macro didn’t provide enough value as-is. I kept explicit impl private::Map blocks in this PR; fn map() centralization can be a separate cleanup.
ebpf/aya-ebpf/src/maps/mod.rs line 149 at r111 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
do we need these? we can't just use them through the supertrait?
Yep, good point. We can use the associated types via the supertrait, so I removed Key/Value from the public Map trait and updated the signatures accordingly.
Summary
This PR is a continuation of #70, rebased onto the current main branch and extended with additional functionality.
It adds comprehensive support for BPF map-of-maps (
BPF_MAP_TYPE_HASH_OF_MAPSandBPF_MAP_TYPE_ARRAY_OF_MAPS):btf_maps::ArrayOfMapsandbtf_maps::HashOfMaps(libbpf-compatible, uses BTF relocations)maps::ArrayOfMapsandmaps::HashOfMaps(legacy)innerattribute to#[map]macro for specifying inner map templates (uses.maps.innersection).maps.innersection for inner map bindingsEbpfLoaderArrayOfMapsandHashOfMapsuserspace types withget(),set()/insert(),keys(),fd()methodsArray::create()andHashMap::create()for dynamic inner map creationExample usage (eBPF side)
BTF (libbpf-compatible) - Recommended:
Legacy (aya-only):
Example usage (userspace side)
Test plan
This change is