EL2 boot on wdk

[pbo-linaro]

Thanks for the great work you've done here!

I'm using a Windows Devkit, and booting from a usb-c disk, with mini DP -> HDMI output.
I've been able to boot Ubuntu_Desktop_24.10_wdk2023_6.13rc, and update to kernel 6.16-rc2 prebuilt provided.

I would be interested to boot in EL2 to support virtualization.
I tried 6.13-rc7-el2-1 prebuilt, but boot is hanging early (just before resolution is updated in early boot). I'm sure it's not video output that has a problem only, as ssh server is not coming up.

Before sending any dmesg output:

• Which branch/kernel do you recommend?
• Are prebuilt el2 kernels supposed to work?
• Any special kernel parameter needed?

Thanks,
Pierrick

[jglathe]

Hi Pierrick,

there are some limitations and pitfalls regarding EL2 on the WDK. I have one running on EL2, although there are certain things not working:

• no adsp and cdsp firmware gets loaded

This has some not-so obvious implications. Without adsp up, you don't have:

• battery management
• sound
• orientation switching on the type-c ports
• mode switching for dp altmode

Which is quite a bit. Battery and sound, who cares. But orientation switching is not good, mode switching is bad if you rely on dp altmode.

That said, I'm using the WDK for over a year now on EL2, and its pretty viable.

To set up a WDK with EL2 I would suggest to read this Wiki as preparation. This can be done on an external SSD as well. Type-c also works. It has orientation dependency, though - once booted in swapped mode with only USB2 support it stays that way. Best and fastest is internal SSD. I have a Silicon Power 2TB 2230 in mine, works brilliantly with BTRFS, can be dual booted to Windows (you sometimes need it).
There are no special parameters necessary for EL2, only the normal ones. But you need slbounce as discussed in the Wiki. And either the dtb-patch tool from slbounce or a dedicated el2-ready dtb. Those are contained in the pre-built kernels I sometimes publish, and by default in all 6.16rc kernels since @TravMurav 's patch is now upstream.
To make it somewhat easier, I'll build and upload 6.15.0-jg-el2-8, which should be best / most stable as of now.

The miniDP on the WDK is sort of perfect for a display, btw. I have it running with a 4k@60 display. Only speciality I had to do was blacklist camcc_sc8280xp for a clean dpms recovery in etc/modprobe.d.
wdk2023.conf.txt

with best regards,
Jens

[jglathe]

6.15.0-jg-el2-7 is uploaded now, no difference regarding wdk to -8.

[pbo-linaro]

Thanks for the clear instructions, I missed there was a dedicated wiki page. It could be worth to add a link in README.

I've been able to boot in EL2 with the prebuilt kernel 6.15.0-jg-el2-7 uploaded by @jglathe.

Besides the excellent wiki link, a few additional information that could help the next developer trying this:

• slbounce must be compiled with make SLBOUNCE_ALWAYS_SWITCH=1, without this flag it didn't boot
• No need to use dtbhack with the provided kernel 6.15.0-jg-el2-7
• HDMI display is working fine
• KVM is working fine
• efi shell can be conveniently found in efi-shell-aa64 ubuntu package: /usr/share/efi-shell-aa64/shellaa64.efi
• Indeed, external drive is limited to USB 2.0 speed.
• Updated to ubuntu 25.04 and everything went fine

I'll try to automate boot using a startup.nsh boot script, as it seems kernel is compiled with CONFIG_EFI_STUB, so it should be bootable directly from EFI Shell. As well, I'll move installation to internal storage, should be much faster.

Many thanks again!

[TravMurav]

slbounce must be compiled with make SLBOUNCE_ALWAYS_SWITCH=1, without this flag it didn't boot

FWIW new slbounce heuristic only allows el2 if zap-shader exists and explicitly disabled (which is a "cautious" heuristic -- that would only happen when el2 overlays are used), if zap shader was just removed from the base dts, then slbounce detection will not switch.

[pbo-linaro]

Thanks for the information @TravMurav, this explains why it didn't work in the setup I was trying.

[jglathe]

You can try to change orientation of the SSD type-c plug, it should work in one direction with USB3.2. Usually label down on the WDK

[jglathe]

I'll try to automate boot using a startup.nsh boot script, as it seems kernel is compiled with CONFIG_EFI_STUB, so it should be bootable directly from EFI Shell. As well, I'll move installation to internal storage, should be much faster.

Would be cool to document it here. Although I should really use dtbloader from @TravMurav, this would be way more elegant.

[pbo-linaro]

You can try to change orientation of the SSD type-c plug, it should work in one direction with USB3.2. Usually label down on the WDK

I can confirm switching the cable orientation allows to have usb 3 speed. I didn't expect orientation to matter for usb-c.

[pbo-linaro]

@jglathe I gave a try to boot the kernel directly from EFI Shell.
Alas, the compiled kernel is missing CONFIG_EFI_ARMSTUB_DTB_LOADER=y, so it's impossible to pass a dtb through command line.

I would be happy to try that, but I'm not sure which branch exactly you used (I don't see any 6.15.0-el2 branch), as well, not sure if you have a script generating all needed packages (rebuild.sh is not working on my side). Would you mind sharing instructions, or simply sharing a prebuilt el2 kernel with config above enabled?

Thanks!

[jglathe]

I didn't expect orientation to matter for usb-c.

Actually, who does 😀 But its a fact of (Linux) life. Those orientation switches are fairly complex, and part of the logic is managed by firmware in adsp. Which isn't fully up on the WDK when booting (x13s neither), and never up on EL2. So... loss of functionality, maybe partially fixable for type-c orientation switch, but a helluva lot of work for dp altmode.

[jglathe]

Alas, the compiled kernel is missing CONFIG_EFI_ARMSTUB_DTB_LOADER=y, so it's impossible to pass a dtb through command line.

I can add that one, sure. It's Ubuntu packaging, so not the easiest way to handle (but overall quite neat).

[pbo-linaro]

Thanks, that would be convenient!

[pbo-linaro]

After recompiling the kernel, I can confirm it's possible to boot automatically in EL2 using an EFI Shell script.
I'll send detailed information when you publish the new kernel, to make sure paths match.

[jglathe]

I have added this config and pushed and uploaded 6.16.0-rc3-jg-0. Looking forward to the script.

[pbo-linaro]

Kernel can be booted as an EFI app thanks to CONFIG_EFI_STUB. As we'll skip the bootloader in this case, there are two importants details we need to take care of:

• For arm64, there is no compressed kernel support: kernel Image must be uncompressed.
• For the ARM and arm64 architectures, a device tree must be provided to the kernel: either it can be loaded in UEFI configuration table (I didn't find any easy tool for that, allowing to load an arbitrary dtb), or it can simply be passed to kernel via dtb parameter, which needs CONFIG_EFI_ARMSTUB_DTB_LOADER=y.

I used dtb coming from 6.15.0-jg-el2-7, and kernel and initrd 6.16.0-rc3-jg-0. To be able to boot in EL2, only the dtb matters, kernel is the same.

First, slbounce needs to be compiled as explained on the wiki. It needs to be compiled with make SLBOUNCE_ALWAYS_SWITCH=1 to force switch to EL2 when it is called. As well, tcblaunch.exe must be copied from the Windows install, as explained in slbounce README.

From there, as root:

# copy data needed in EFI partition: uncompressed kernel, initrd, dtb (el2) 
cd /boot/efi
# uncompress kernel image using zcat
zcat /boot/vmlinuz-6.16.0-rc3-jg-0-qcom-x1e > linux-6.16.0-rc3-jg-0-qcom-x1e.efi
# copy initrd
cp /boot/initrd.img-6.16.0-rc3-jg-0-qcom-x1e .
# copy el2 dtb
cp /boot/dtb-6.15.0-jg-el2-7-qcom-x1e sc8280xp-microsoft-blackrock-el2.dtb

At this point, you should get this structure:

/boot/efi/
├── EFI
│   ├── BOOT
│   │   ├── BOOTAA64.EFI
│   │   └── shellaa64.efi
│   └── ubuntu
│       ├── grub.cfg
│       └── grubaa64.efi
├── initrd.img-6.16.0-rc3-jg-0-qcom-x1e
├── linux-6.16.0-rc3-jg-0-qcom-x1e.efi
├── sc8280xp-microsoft-blackrock-el2.dtb
├── slbounce.efi
└── tcblaunch.exe

---

Then, we can boot kernel directly from EFI shell:

fs3: # on internal drive, may be different number from external disk
load slbounce.efi
linux-6.16.0-rc3-jg-0-qcom-x1e.efi initrd=\initrd.img-6.16.0-rc3-jg-0-qcom-x1e dtb=\sc8280xp-microsoft-blackrock-el2.dtb root=/dev/nvme0n1p2 ro  clk_ignore_unused pd_ignore_unused cma=128M rootdelay=20 rootwait pd_ignore_unused clk_ignore_unused arm64.nopauth efi=noruntime dyndbg='file drivers/base/firmware_loader/main.c +p' earlycon=efifb keepearlycon
# root=... must be set for your / partition

Please note this bypass grub completely, which is only used to launch the EFI shell, so all kernel parameters have to be set. Most notably, the root= where you're booting. I didn't try to use UUID, but it should be possible.

---

From now, if you want to automate the boot, you can simply write those instructions in
/boot/efi/EFI/BOOT/startup.nsh.
As well, EFI Shell can be set as default for grub in /etc/default/grub with GRUB_DEFAULT="EFI Shell", or any other name you chose for this entry. Don't forget to run sudo update-grub to apply this change.

---

This solution allows to boot automatically in EL2, to the price of some manual configuration, that has to be repeated in case you update your kernel. Good enough for my use case of a devkit staying far from any screen/keyboard.

The general direction for windows-arm64 devices is to detect the correct dtb at boot time, based on various criterias (like dtbloader). While this is nice, and will eventually work out of the box one day, I feel it adds a lot of complexity (device tree overlay, another EFI driver) when trying to understand the boot process. As well, I didn't see how it's possible to selectively apply different dtb depending if you boot in EL1 or EL2.

Booting in EL2 is still a niche case, considering some important things are not working. However, if the current subset of features is enough for you, it's a nice way to have a reasonably powered arm device with virtualization enabled.

Thanks to @jglathe (dtb, prebuilt kernels and image and all your help!) and @TravMurav (amazing reverse engineering effort on slbounce) for your work.