Pink screen with artifacts when using Vulkan AV1 encoder on NVIDIA

[shoulderrr]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

A pink glitchy screen appears when using KMS capture and the new Vulkan encoder (AV1 specifically).
Audio is working fine, however the pink noisy screen displays for a few seconds then holds on a single frame. Sunshine doesn't detect any pause or disconnection and audio keeps going. Have previously used the VA-API encoder without issue (besides expected performance hit), still using KMS capture as recommended by the docs.

Quick info:
Arch Linux
5090
KDE Plasma (Wayland)
4k 120FPS target

Screenshot of issue (NeoMoonlight beta, also happens with MoonlightXROS):

Expected Behavior

Normal stream appears like with VA API encoder

Additional Context

First time filing a bug on here, sorry if anything is wrong or weird with this! Double checked a few places to make sure I wasn't missing some dependencey or if NVIDIA isn't supported yet but didn't find anything

Host Operating System

Linux

Operating System Version

Arch (not sure how to convey version here, everything up to date)

Architecture

amd64/x86_64

Package

Linux - LizardByte/pacman-repo

GPU Type

NVIDIA

GPU Model

RTX 5090

GPU Driver/Mesa Version

nvidia-open 595.58.03-3

Capture Method

KMS (Linux)

Apps

Log output

[2026-04-18 22:13:40.582]: Info: Sunshine version: 2026.417.201619 commit: 5ffa66ee52fc4f57c018ae5ae831df3bf7088de5
[2026-04-18 22:13:40.582]: Info: Package Publisher: LizardByte
[2026-04-18 22:13:40.582]: Info: Publisher Website: https://app.lizardbyte.dev
[2026-04-18 22:13:40.582]: Info: Get support: https://app.lizardbyte.dev/support
[2026-04-18 22:13:40.582]: Info: config: 'capture' = kms
[2026-04-18 22:13:40.582]: Info: config: 'controller' = disabled
[2026-04-18 22:13:40.582]: Info: config: 'nvenc_preset' = 1
[2026-04-18 22:13:40.582]: Info: config: 'vk_tune' = 0
[2026-04-18 22:13:40.582]: Info: config: 'encoder' = vulkan
[2026-04-18 22:13:40.582]: Info: config: 'keyboard' = disabled
[2026-04-18 22:13:40.582]: Info: config: 'min_threads' = 4
[2026-04-18 22:13:40.582]: Info: config: 'mouse' = disabled
[2026-04-18 22:13:40.582]: Info: config: 'nvenc_spatial_aq' = enabled
[2026-04-18 22:13:40.582]: Info: config: 'nvenc_vbv_increase' = 400
[2026-04-18 22:13:40.582]: Info: config: 'vk_rc_mode' = 0
[2026-04-18 22:13:40.583]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:40.583]: Info: /dev/dri/card2 -> amdgpu
[2026-04-18 22:13:40.585]: Info: [wayland] Found display [wayland-0]
[2026-04-18 22:13:40.585]: Info: [wayland] Found display [wayland-0]
[2026-04-18 22:13:40.585]: Info: [wayland] Found interface: zxdg_output_manager_v1(30) version 3
[2026-04-18 22:13:40.585]: Info: [wayland] Found interface: zwp_linux_dmabuf_v1(56) version 5
[2026-04-18 22:13:40.585]: Info: [wayland] Found interface: wl_output(63) version 4
[2026-04-18 22:13:40.585]: Info: [wayland] Resolution: 3840x2160
[2026-04-18 22:13:40.585]: Info: [wayland] Offset: 0x0
[2026-04-18 22:13:40.585]: Info: [wayland] Logical size: 2194x1234
[2026-04-18 22:13:40.585]: Info: [wayland] Name: HDMI-A-2
[2026-04-18 22:13:40.585]: Info: [wayland] Found monitor: LG Electronics LG TV SSCR2
[2026-04-18 22:13:40.585]: Info: -------- Start of KMS monitor list --------
[2026-04-18 22:13:40.585]: Warning: Mismatch on expected Resolution compared to actual resolution: 0x0 vs 3840x2160
[2026-04-18 22:13:40.585]: Info: Monitor 0 is HDMI-A-2: LG Electronics LG TV SSCR2
[2026-04-18 22:13:40.585]: Info: --------- End of KMS monitor list ---------
[2026-04-18 22:13:40.725]: Info: Trying encoder [vulkan]
[2026-04-18 22:13:40.725]: Info: Screencasting with KMS
[2026-04-18 22:13:40.725]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:40.725]: Info: Found monitor for DRM screencasting
[2026-04-18 22:13:40.725]: Info: Found connector ID [128]
[2026-04-18 22:13:40.725]: Info: Found cursor plane [56]
[2026-04-18 22:13:40.725]: Info: Creating encoder [h264_vulkan]
[2026-04-18 22:13:40.725]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:40.725]: Info: Color depth: 8-bit
[2026-04-18 22:13:40.725]: Info: Color range: JPEG
[2026-04-18 22:13:41.078]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:41.082]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:41.185]: Info: Creating encoder [hevc_vulkan]
[2026-04-18 22:13:41.186]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:41.186]: Info: Color depth: 8-bit
[2026-04-18 22:13:41.186]: Info: Color range: JPEG
[2026-04-18 22:13:41.481]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:41.486]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:41.582]: Info: Creating encoder [av1_vulkan]
[2026-04-18 22:13:41.582]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:41.582]: Info: Color depth: 8-bit
[2026-04-18 22:13:41.582]: Info: Color range: JPEG
[2026-04-18 22:13:41.858]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:41.863]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:41.950]: Info: Screencasting with KMS
[2026-04-18 22:13:41.950]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:41.950]: Info: Found monitor for DRM screencasting
[2026-04-18 22:13:41.950]: Info: Found connector ID [128]
[2026-04-18 22:13:41.950]: Info: Found cursor plane [56]
[2026-04-18 22:13:41.950]: Info: Creating encoder [hevc_vulkan]
[2026-04-18 22:13:41.950]: Info: Color coding: SDR (Rec. 709)
[2026-04-18 22:13:41.950]: Info: Color depth: 10-bit
[2026-04-18 22:13:41.950]: Info: Color range: JPEG
[2026-04-18 22:13:42.226]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:42.230]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:42.322]: Info: Creating encoder [av1_vulkan]
[2026-04-18 22:13:42.322]: Info: Color coding: SDR (Rec. 709)
[2026-04-18 22:13:42.322]: Info: Color depth: 10-bit
[2026-04-18 22:13:42.322]: Info: Color range: JPEG
[2026-04-18 22:13:42.582]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:42.587]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:42.668]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-04-18 22:13:42.668]: Info: 
[2026-04-18 22:13:42.668]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-04-18 22:13:42.668]: Info: 
[2026-04-18 22:13:42.668]: Info: Found H.264 encoder: h264_vulkan [vulkan]
[2026-04-18 22:13:42.668]: Info: Found HEVC encoder: hevc_vulkan [vulkan]
[2026-04-18 22:13:42.668]: Info: Found AV1 encoder: av1_vulkan [vulkan]
[2026-04-18 22:13:42.668]: Info: Starting system tray
[2026-04-18 22:13:42.670]: Info: Configuration UI available at [https://localhost:47990]
[2026-04-18 22:13:42.688]: Info: System tray created
[2026-04-18 22:13:42.689]: Info: Starting main loop
[2026-04-18 22:13:43.588]: Info: Adding avahi service fivenine
[2026-04-18 22:13:44.475]: Info: Avahi service fivenine successfully established.
[2026-04-18 22:13:56.706]: Info: Trying encoder [vulkan]
[2026-04-18 22:13:56.706]: Info: Screencasting with KMS
[2026-04-18 22:13:56.706]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:56.706]: Info: Found monitor for DRM screencasting
[2026-04-18 22:13:56.706]: Info: Found connector ID [128]
[2026-04-18 22:13:56.706]: Info: Found cursor plane [56]
[2026-04-18 22:13:56.706]: Info: Creating encoder [h264_vulkan]
[2026-04-18 22:13:56.706]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:56.706]: Info: Color depth: 8-bit
[2026-04-18 22:13:56.706]: Info: Color range: JPEG
[2026-04-18 22:13:57.039]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:57.043]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:57.132]: Info: Creating encoder [hevc_vulkan]
[2026-04-18 22:13:57.132]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:57.132]: Info: Color depth: 8-bit
[2026-04-18 22:13:57.132]: Info: Color range: JPEG
[2026-04-18 22:13:57.407]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:57.411]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:57.507]: Info: Creating encoder [av1_vulkan]
[2026-04-18 22:13:57.507]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:57.507]: Info: Color depth: 8-bit
[2026-04-18 22:13:57.507]: Info: Color range: JPEG
[2026-04-18 22:13:57.784]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:57.788]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:57.873]: Info: Screencasting with KMS
[2026-04-18 22:13:57.873]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:57.873]: Info: Found monitor for DRM screencasting
[2026-04-18 22:13:57.873]: Info: Found connector ID [128]
[2026-04-18 22:13:57.873]: Info: Found cursor plane [56]
[2026-04-18 22:13:57.873]: Info: Creating encoder [hevc_vulkan]
[2026-04-18 22:13:57.873]: Info: Color coding: SDR (Rec. 709)
[2026-04-18 22:13:57.873]: Info: Color depth: 10-bit
[2026-04-18 22:13:57.873]: Info: Color range: JPEG
[2026-04-18 22:13:58.166]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:58.171]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:58.262]: Info: Creating encoder [av1_vulkan]
[2026-04-18 22:13:58.262]: Info: Color coding: SDR (Rec. 709)
[2026-04-18 22:13:58.262]: Info: Color depth: 10-bit
[2026-04-18 22:13:58.262]: Info: Color range: JPEG
[2026-04-18 22:13:58.529]: Info: Streaming bitrate is 1000000
[2026-04-18 22:13:58.533]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:58.613]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-04-18 22:13:58.613]: Info: 
[2026-04-18 22:13:58.613]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-04-18 22:13:58.613]: Info: 
[2026-04-18 22:13:58.613]: Info: Found H.264 encoder: h264_vulkan [vulkan]
[2026-04-18 22:13:58.613]: Info: Found HEVC encoder: hevc_vulkan [vulkan]
[2026-04-18 22:13:58.613]: Info: Found AV1 encoder: av1_vulkan [vulkan]
[2026-04-18 22:13:58.613]: Info: Executing [Desktop]
[2026-04-18 22:13:58.681]: Info: New streaming session started [active sessions: 1]
[2026-04-18 22:13:58.708]: Info: CLIENT CONNECTED
[2026-04-18 22:13:58.718]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:58.718]: Info: /dev/dri/card2 -> amdgpu
[2026-04-18 22:13:58.719]: Info: [wayland] Found display [wayland-0]
[2026-04-18 22:13:58.720]: Info: [wayland] Found interface: zxdg_output_manager_v1(30) version 3
[2026-04-18 22:13:58.720]: Info: [wayland] Found interface: zwp_linux_dmabuf_v1(56) version 5
[2026-04-18 22:13:58.720]: Info: [wayland] Found interface: wl_output(63) version 4
[2026-04-18 22:13:58.720]: Info: [wayland] Resolution: 3840x2160
[2026-04-18 22:13:58.720]: Info: [wayland] Offset: 0x0
[2026-04-18 22:13:58.720]: Info: [wayland] Logical size: 2194x1234
[2026-04-18 22:13:58.720]: Info: [wayland] Name: HDMI-A-2
[2026-04-18 22:13:58.720]: Info: [wayland] Found monitor: LG Electronics LG TV SSCR2
[2026-04-18 22:13:58.720]: Info: -------- Start of KMS monitor list --------
[2026-04-18 22:13:58.720]: Warning: Mismatch on expected Resolution compared to actual resolution: 0x0 vs 3840x2160
[2026-04-18 22:13:58.720]: Info: Monitor 0 is HDMI-A-2: LG Electronics LG TV SSCR2
[2026-04-18 22:13:58.720]: Info: --------- End of KMS monitor list ---------
[2026-04-18 22:13:58.720]: Info: Screencasting with KMS
[2026-04-18 22:13:58.720]: Info: /dev/dri/card1 -> nvidia-drm
[2026-04-18 22:13:58.720]: Info: Found monitor for DRM screencasting
[2026-04-18 22:13:58.720]: Info: Found connector ID [128]
[2026-04-18 22:13:58.720]: Info: Found cursor plane [56]
[2026-04-18 22:13:58.720]: Info: Creating encoder [av1_vulkan]
[2026-04-18 22:13:58.720]: Info: Color coding: SDR (Rec. 601)
[2026-04-18 22:13:58.720]: Info: Color depth: 8-bit
[2026-04-18 22:13:58.720]: Info: Color range: MPEG
[2026-04-18 22:13:59.043]: Info: Streaming bitrate is 77452000
[2026-04-18 22:13:59.048]: Info: Vulkan encode using GPU: NVIDIA GeForce RTX 5090
[2026-04-18 22:13:59.049]: Info: Minimum FPS target set to ~60fps (16.6667ms)
[2026-04-18 22:13:59.209]: Info: Setting default sink to: [sink-sunshine-surround71]
[2026-04-18 22:13:59.209]: Info: Found default monitor by name: sink-sunshine-surround71.monitor
[2026-04-18 22:13:59.222]: Info: Opus initialized: 48 kHz, 8 channels, 2048 kbps (total), LOWDELAY
[2026-04-18 22:17:05.092]: Info: Setting default sink to: [alsa_output.pci-0000_01_00.1.hdmi-stereo]

Online logs

No response

[neatnoise]

Is the image corruption only when using AV1 or HEVC, H264 too?

[neatnoise]

Could you test this PR #5030 and provide the feedback with log included?

[Signynt]

Could you test this PR #5030 and provide the feedback with log included?

@neatnoise I'm still getting the same artifacts with AV1, I've sent you a log on Discord.

[shoulderrr]

Could you test this PR #5030 and provide the feedback with log included?

Will test this later, was running off the sunshine-git package so I need to get the builds running from source later. IIRC this same issue was happening with HEVC but I didn’t mess with it too much yet, will verify that too

[neatnoise]

@Signynt @connorjohnhalloran From Sunshine perspective the code should be the same for h264, hevc, av1. If you see the correct h264 stream, corrupted hevc, av1 stream I think It might be ffmpeg issue. Please test https://github.com/LizardByte/build-deps with ffmpeg 8.1 if possible.

[shoulderrr]

Starting to do a bit more testing now, still have the issue on the latest prelease (2026.419.214410-1) from sunshine.pkg.tar.gz (wanted to do a sanity check after releasing there were more recent prereleases). Going to to try the patch associated with PR #5030 next

[ReenigneArcher]

FFmpeg 8.1 has been merged into Sunshine now, and I kicked off a build for #5030

[neatnoise]

I had a chat with @Signynt . The issue seems to be outside of Sunshine. From Sunshine perspective the code and behaviour should be similar when used each codec (h264, hevc, av1) - that's where ffmpeg is for the convenience. If h264 works ok, hevc or/and av1 not, It points to the issue with FFmpeg/drivers the most probably.

[shoulderrr]

@neatnoise I sent a discord message last night asking for some assistance with getting your branch running, if you want some more testing data on that branch let me know (was able to get the main repo building fine I think but cmake was having issues with the vulkan-fixes branch on your fork for me).

I just did some further testing of the sunshine-git 2026.419.214410-1 Arch package with the different codec and this is what I'm seeing (all at 4k 120FPS 100Mbps):

• AV1: Same pink screen as before
• HEVC: Streams video correctly, but with poor performance (60ish FPS, visible artifacts)
• H.264: Seems to work fine, hits 120fps but motion looks weird (never used this codec before so it might just be what it looks like)

Here's what the HEVC artifacts look like, ignore the blurring (that's just how the AVP screenshots work due to DFR). There's some weird blocks of color in the title bar and the fps is only 54 avg despite using so much GPU:

Based on this I guess it does make sense that it's some sort of driver or FFMPEG issue, especially given that sunshine's logs don't seem to think anything is wrong. The only thing that stood out was the warning about resolution:
[2026-04-18 22:13:58.720]: Warning: Mismatch on expected Resolution compared to actual resolution: 0x0 vs 3840x2160
Is there anything related to this that might be an issue?