Draft: NEEDS HELP surface: linux 6.12.19 -> 6.12.21

[matthiasdotsh]

Description of changes

Starting with linux 6.12.20 kernel bumps are failing.

# Sorry, my nixos config isn't public yet.
sudo nixos-rebuild switch --flake .#surface-pro9 

last lines from nix log /nix/store/dvikj3jr164y07ffrz8j88d81c9f53mh-linux-6.12.21.drv (nothing special before)

  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp_psp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp1_execution.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp1_transition.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_execution.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_transition.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_isp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_1.o
  LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.o
  AR      drivers/gpu/built-in.a
make[2]: *** [../scripts/Makefile.build:478: drivers] Error 2
make[1]: *** [/build/linux-6.12.21/Makefile:1944: .] Error 2
make: *** [../Makefile:224: __sub-make] Error 2

Could someone who also has a surface please try if it works for him or does someone have an idea how to get more debug infos?

Things done

• Tested the changes in your own NixOS Configuration
• Tested the changes end-to-end by using your fork of nixos-hardware and
  importing it via <nixos-hardware> or Flake input

[8bitbuddhist]

@matthiasdotsh I'm currently testing a PR for kernel 6.13.6. At some point a new patch was added, so maybe that's causing your build error

Edit: Wait nevermind, the patch is only for 6.13

[matthiasdotsh]

I'm currently testing a PR for kernel 6.13.6

Is it working? I tried with 6.13.8 but got a similar error.

[8bitbuddhist]

I'm currently testing a PR for kernel 6.13.6

Is it working? I tried with 6.13.8 but got a similar error.

I'm getting a conflict with the hardware.ipu6 module that I'm looking into, but otherwise it's all working as expected. Here's my branch: https://code.8bitbuddhism.com/aires/nixos-hardware/src/branch/surface/kernel-6.13.6

In your commit, the only thing I could see that might be problematic is that repos.nix is pointing to rev 6.13.6 instead of 6.13.8

[matthiasdotsh]

but otherwise it's all working as expected. Here's my branch: https://code.8bitbuddhism.com/aires/nixos-hardware/src/branch/surface/kernel-6.13.6

Mh, I compiled your code and it works on my system. I'll have to take another look at what exactly the differences are compared to my 6.13. branch. I tested with 6.13.6 before and got the same errors as with 6.13.8.

I'm not sure if it wouldn't be better to stay on the lts kernel (6.12.) (because I'm using zfs and there is always some delay until they support new kernels).

[8bitbuddhist]

but otherwise it's all working as expected. Here's my branch: https://code.8bitbuddhism.com/aires/nixos-hardware/src/branch/surface/kernel-6.13.6

Mh, I compiled your code and it works on my system. I'll have to take another look at what exactly the differences are compared to my 6.13. branch. I tested with 6.13.6 before and got the same errors as with 6.13.8.

I'm not sure if it wouldn't be better to stay on the lts kernel (6.12.) (because I'm using zfs and there is always some delay until they support new kernels).

100% agree, there are a lot of modules that still don't support 6.13. Maybe we could make 6.12 the default and include 6.13 as an option? Something like linux-surface_default and linux-surface_latest?

[matthiasdotsh]

Mh, I compiled your code and it works on my system. I'll have to take another look at what exactly the differences are compared to my 6.13. branch.

Ok, I looked into my 6.13. branch, downgraded it to 6.13.6 and now it works.

I'm now building 6.13.8.

@8bitbuddhist Do you have enough resources to try the 6.12.21 kernel from this PR?

I'm not sure if it wouldn't be better to stay on the lts kernel (6.12.) (because I'm using zfs and there is always some delay until they support new kernels).

100% agree, there are a lot of modules that still don't support 6.13. Maybe we could make 6.12 the default and include 6.13 as an option? Something like linux-surface_default and linux-surface_latest?

I really like the idea and would be super happy to help. However, I might need some assistance.

[8bitbuddhist]

I think the problem is that linux-surface hasn't released any packaged kernels past 6.12.7, so if we try to change repos.nix to pull a version other than what's listed in their releases, the build will fail. Or, it will just update the package version without actually updating the kernel. AFAIK the only way to use a different kernel version is to update this variable and run build-linux-surface.py. There's work being done to update the kernel version automatically, but it's still a draft PR.

In the meantime, the only way (I think) we'd be able to use kernel versions other than the releases is to use an overlay and run the entire build process from start to finish. I could be wrong though - I'm still figuring this out 😅

[matthiasdotsh]

I think the problem is that linux-surface hasn't released any packaged kernels past 6.12.7, so if we try to change repos.nix to pull a version other than what's listed in their releases, the build will fail. Or, it will just update the package version without actually updating the kernel.

Are you sure about this?

Most PRs that have updated the Surface kernel here in nixos-hardware in the past have only updated the version and sha256 in microsoft/surface/common/kernel/linux-surface/default.nix.

I'm pretty sure that this worked, as uname -a was always reporting the new kernel version.
My understanding has always been that we are more or less only using the patches from the linux-surface project.

So there might be a change in the upstream kernel version that now require some adjustments in the linux-surface project?

[8bitbuddhist]

I think the problem is that linux-surface hasn't released any packaged kernels past 6.12.7, so if we try to change repos.nix to pull a version other than what's listed in their releases, the build will fail. Or, it will just update the package version without actually updating the kernel.

Are you sure about this?

Most PRs that have updated the Surface kernel here in nixos-hardware in the past have only updated the version and sha256 in microsoft/surface/common/kernel/linux-surface/default.nix.

I'm pretty sure that this worked, as uname -a was always reporting the new kernel version. My understanding has always been that we are more or less only using the patches from the linux-surface project.

So there might be a change in the upstream kernel version that now require some adjustments in the linux-surface project?

Nope, I'm not sure 😆 I'm still learning how this all works.

[matthiasdotsh]

Just for reference: Same error with latest 6.12.22 kernel

[8bitbuddhist]

After some (a lot of) trial and error, I've got kernel switching working between 6.12 and 6.13. Changing the version should only require changing this line and this line.

Now I'll look at why the other kernel versions aren't building

Edit 4/11: missed a variable. Now it should work!

[8bitbuddhist]

One thing I'd like your thoughts on, @matthiasdotsh:

For choosing the kernel version, I originally set the options to 6.12 and 6.13. This works, but it does mean updating these each time there's a new minor version. Instead, I set the options to lts and latest. They still point to 6.12 and 6.13, but I think it'd be more intuitive than remembering specific kernel versions. Do you have a preference?

https://github.com/8bitbuddhist/nixos-hardware/blob/d3c5202b089facf64a2e67e40ce06b3e79ddb730/microsoft/surface/common/default.nix#L7

[matthiasdotsh]

Thanks a lot for your work @8bitbuddhist

For choosing the kernel version, I originally set the options to 6.12 and 6.13. This works, but it does mean updating these each time there's a new minor version. Instead, I set the options to lts and latest. They still point to 6.12 and 6.13, but I think it'd be more intuitive than remembering specific kernel versions. Do you have a preference?

I like the idea a lot. This means we can maintain this and others can just update nixos-hardware and are good.

I understand why you chose lts and latest as names.
However looking at https://kernel.org/ what do you think about longterm and stable?

[8bitbuddhist]

Just tried building 6.12.21 and I got a build error when compiling nouveau. Something about GPU drivers is mucking things up. I'm trying now with 6.12.22 since that's the latest, and maybe it'll output more useful info

[matthiasdotsh]

Does it look similar to this traceback? I already tried with 6.12.22 and it failed with a very similar message.

[8bitbuddhist]

Does it look similar to this traceback? I already tried with 6.12.22 and it failed with a very similar message.

Yep, pretty much exactly the same, down to the line numbers:

linux>   CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_isp.o
linux>   CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_0.o
linux>   CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_1.o
linux>   LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.o
linux>   AR      drivers/gpu/built-in.a
linux> make[2]: *** [../scripts/Makefile.build:478: drivers] Error 2
linux> make[1]: *** [/build/linux-6.12.22/Makefile:1944: .] Error 2
linux> make: *** [../Makefile:224: __sub-make] Error 2
error: builder for '/nix/store/jvyjlx2pgb6yxjb02ynyjpyld46wr1kf-linux-6.12.22.drv' failed with exit code 2;
       last 25 log lines:
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn303.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn31.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn314.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn315.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn316.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn32.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn35.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn351.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dmub/src/dmub_dcn401.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp_ddc.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp_log.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp_psp.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp1_execution.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp1_transition.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_execution.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/../display/modules/hdcp/hdcp2_transition.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_isp.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_0.o
       >   CC [M]  drivers/gpu/drm/amd/amdgpu/isp_v4_1_1.o
       >   LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.o
       >   AR      drivers/gpu/built-in.a
       > make[2]: *** [../scripts/Makefile.build:478: drivers] Error 2
       > make[1]: *** [/build/linux-6.12.22/Makefile:1944: .] Error 2
       > make: *** [../Makefile:224: __sub-make] Error 2
       For full logs, run 'nix log /nix/store/jvyjlx2pgb6yxjb02ynyjpyld46wr1kf-linux-6.12.22.drv'.
error: 1 dependencies of derivation '/nix/store/4xhpy02djw7f5w284clqxjg66zscvnsy-linux-6.12.22-modules.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qxzh634l0cqvpy0hn57kiqx7ig8hr0cq-boot.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9brpfsznpjzhw7ilsn4vginyipw33ihs-nixos-system-Khanda-24.11.20250410.f9ebe33.drv' failed to build

[matthiasdotsh]

The upstream linux-surface project just published a 6.14. release.

[8bitbuddhist]

The upstream linux-surface project just published a 6.14. release.

0012cff

[matthiasdotsh]

The upstream linux-surface project just published a 6.14. release.

0012cff

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Is there an easy way to test this without manually integrating the changes?

$ git checkout 0012cffb692c640807d1f19b395502499cc9d80b                                        
fatal: unable to read tree (0012cffb692c640807d1f19b395502499cc9d80b)

[8bitbuddhist]

The easiest way is probably to point flake.nix to the branch:

nixos-hardware.url = "github:8bitbuddhist/nixos-hardware.git?ref=surface/kernel-multiple-versions";

[8bitbuddhist]

I thought I'd be clever and try rebuilding without the 0013-amd-gpio patch, to see if it was introducing some bug, but I got the same AMD build error anyway. I'll keep trying though