Svix server went to limbo when redis connection lost

[vanhtuan0409]

Bug Report

We have a setup where the SVIX server uses redis as a queue mechanism.
Whenever redis get restart, the SVIX server will go to a limbo state because it cannot fetch new message from redis. Even after redis server recover, SVIX cannot detect such changes and recover itself

The log returned during the limbo state is

2025-03-25T05:25:03.359590Z ERROR svix_server::worker: Error receiving task: Error { trace: [Location { file: "/rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs", line: 250, col: 5 }, Location { file: "/app/svix-server/src/queue/mod.rs", line: 225, col: 14 }], typ: Queue("NOGROUP: No such key '{queue}_svix_v3_main' or consumer group 'svix_workers_group' in XREADGROUP with GROUP option") }

Which match the logic code at
https://github.com/svix/svix-webhooks/blob/main/server/svix-server/src/worker.rs#L994C1-L995C1

Version

v1.62.0

Platform

Linux svix-579b7c56d4-4qvhr 6.1.119-129.201.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Dec 3 21:07:35 UTC 2024 x86_64 GNU/Linux

Description