bpf, riscv64: Support load-acquire and store-release instructions #8868
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![kernel-patches-daemon-bpf[bot]](https://avatars.githubusercontent.com/u/70728002?s=60&v=4){kind=small}
|
Upstream branch: 38d976c |
|
Upstream branch: 38d976c |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
7ca96ef to
36ba674
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
993c0c1 to
97831b9
Compare
|
Upstream branch: 358b1c0 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
36ba674 to
d6f4598
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
97831b9 to
c60a0e6
Compare
|
Upstream branch: 358b1c0 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
d6f4598 to
4552c2b
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
c60a0e6 to
bebc5be
Compare
|
Upstream branch: 7b05f43 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
4552c2b to
7c9f1ba
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
bebc5be to
bc80b4f
Compare
|
Upstream branch: 7b05f43 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
7c9f1ba to
3e10273
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
bc80b4f to
ad4665c
Compare
|
Upstream branch: 7b05f43 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
3e10273 to
882bbed
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
ad4665c to
f363929
Compare
|
Upstream branch: 86870d0 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
882bbed to
2246571
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
f363929 to
632a377
Compare
|
Upstream branch: f263336 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
2246571 to
e5252ef
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
632a377 to
6bce87f
Compare
|
Upstream branch: f263336 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
e5252ef to
f90ec18
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
6bce87f to
19783fc
Compare
|
Upstream branch: 41d4ce6 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
f90ec18 to
e5f8e08
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
19783fc to
786af9a
Compare
|
Upstream branch: 41948af |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
e5f8e08 to
2530faf
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
786af9a to
309dda5
Compare
In preparation for supporting BPF load-acquire and store-release instructions for architectures where bpf_jit_needs_zext() returns true (e.g. riscv64), make insn_def_regno() handle load-acquires properly. Signed-off-by: Peilin Ye <[email protected]>
We're planning to add support for the load-acquire and store-release BPF instructions. Define emit_load_<size>() and emit_store_<size>() to enable/facilitate the (re)use of their code. Tested-by: Peilin Ye <[email protected]> Signed-off-by: Andrea Parri <[email protected]> [[email protected]: cosmetic change to commit title] Signed-off-by: Peilin Ye <[email protected]> Reviewed-by: Pu Lehui <[email protected]>
Support BPF load-acquire (BPF_LOAD_ACQ) and store-release (BPF_STORE_REL) instructions in the riscv64 JIT compiler. For example, consider the following 64-bit load-acquire (assuming little-endian): db 10 00 00 00 01 00 00 r1 = load_acquire((u64 *)(r1 + 0x0)) 95 00 00 00 00 00 00 00 exit opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000100): BPF_LOAD_ACQ The JIT compiler will emit an LD instruction followed by a FENCE R,RW instruction for the above, e.g.: ld x7,0(x6) fence r,rw Similarly, consider the following 16-bit store-release: cb 21 00 00 10 01 00 00 store_release((u16 *)(r1 + 0x0), w2) 95 00 00 00 00 00 00 00 exit opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000110): BPF_STORE_REL A FENCE RW,W instruction followed by an SH instruction will be emitted, e.g.: fence rw,w sh x2,0(x4) 8-bit and 16-bit load-acquires are zero-extending (cf., LBU, LHU). The verifier always rejects misaligned load-acquires/store-releases (even if BPF_F_ANY_ALIGNMENT is set), so the emitted load and store instructions are guaranteed to be single-copy atomic. Introduce primitives to emit the relevant (and the most common/used in the kernel) fences, i.e. fences with R -> RW, RW -> W and RW -> RW. Rename emit_atomic() to emit_atomic_rmw() to make it clear that it only handles RMW atomics, and replace its is64 parameter to allow to perform the required checks on the opsize (BPF_SIZE(code)). Tested-by: Peilin Ye <[email protected]> Signed-off-by: Andrea Parri <[email protected]> [[email protected]: whitespace changes; cosmetic changes to commit title and message] Signed-off-by: Peilin Ye <[email protected]> Reviewed-by: Pu Lehui <[email protected]>
Currently, the verifier inserts a zext instruction right after every 8-, 16- or 32-bit load-acquire, which is already zero-extending. Skip such redundant zext instructions. While we are here, update that already-obsolete comment about "skip the next instruction" in build_body(). Also change emit_atomic_rmw()'s parameters to keep it consistent with emit_atomic_ld_st(). Note that checking 'insn[1]' relies on 'insn' not being the last instruction, which should have been guaranteed by the verifier; we already use 'insn[1]' elsewhere in the file for similar purposes. Additionally, we don't check if 'insn[1]' is actually a zext for our load-acquire's dst_reg, or some other registers - in other words, here we are relying on the verifier to always insert a redundant zext right after a 8/16/32-bit load-acquire, for its dst_reg. Signed-off-by: Peilin Ye <[email protected]> Reviewed-by: Pu Lehui <[email protected]>
Instead of open-coding the conditions, use '#ifdef CAN_USE_LOAD_ACQ_STORE_REL' to guard the following tests: verifier_precision/bpf_load_acquire verifier_precision/bpf_store_release verifier_store_release/* Note that, for the first two tests in verifier_precision.c, switching to '#ifdef CAN_USE_LOAD_ACQ_STORE_REL' means also checking if '__clang_major__ >= 18', which has already been guaranteed by the outer '#if' check. Signed-off-by: Peilin Ye <[email protected]>
Currently, we pass 0x1234567890abcdef to __retval() for the following two tests: verifier_load_acquire/load_acquire_64 verifier_store_release/store_release_64 However, the upper 32 bits of that value are being ignored, since __retval() expects an int. Actually, the tests would still pass even if I change '__retval(0x1234567890abcdef)' to e.g. '__retval(0x90abcdef)'. Restructure the tests a bit to test the entire 64-bit values properly. Do the same to their 8-, 16- and 32-bit variants as well to keep the style consistent. Fixes: ff3afe5 ("selftests/bpf: Add selftests for load-acquire and store-release instructions") Signed-off-by: Peilin Ye <[email protected]>
Verify that 8-, 16- and 32-bit load-acquires are zero-extending by using immediate values with their highest bit set. Do the same for the 64-bit variant to keep the style consistent. Signed-off-by: Peilin Ye <[email protected]>
…for riscv64
For riscv64, enable all BPF_{LOAD_ACQ,STORE_REL} selftests except the
arena_atomics/* ones (not guarded behind CAN_USE_LOAD_ACQ_STORE_REL),
since arena access is not yet supported.
Signed-off-by: Peilin Ye <[email protected]>
|
Upstream branch: 62e23f1 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/958319=>bpf-next
branch
from
2530faf to
b5c6052
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: bpf, riscv64: Support load-acquire and store-release instructions
version: 1
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=958319