message":"worker.container-sweeper.tick.failed-to-dial","data":{"error":"all worker SSH gateways unreachable","session"

[helloparas]

We suddenly started to get these error in concourse .

worker SSH gateways unreachable","session":"7.12"}}
{"timestamp":"2025-02-20T08:53:40.103422960Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-connect-to-tsa","data":{"error":"dial tcp 172.20.68.15:2222: connect: connection refused","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:40.103569451Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.dial.failed-to-connect-to-any-tsa","data":{"error":"all worker SSH gateways unreachable","session":"6.12.1"}}
{"timestamp":"2025-02-20T08:53:40.103603917Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-dial","data":{"error":"all worker SSH gateways unreachable","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:40.103631612Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-report-containers","data":{"error":"all worker SSH gateways unreachable","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:41.127473239Z","level":"error","source":"worker","message":"worker.volume-sweeper.tick.failed-to-connect-to-tsa","data":{"error":"dial tcp 172.20.68.15:2222: connect: connection refused","session":"7.12"}}
{"timestamp":"2025-02-20T08:53:41.127549640Z","level":"error","source":"worker","message":"worker.volume-sweeper.tick.dial.failed-to-connect-to-any-tsa","data":{"error":"all worker SSH gateways unreachable","session":"7.12.3"}}
{"timestamp":"2025-02-20T08:53:41.127568997Z","level":"error","source":"worker","message":"worker.volume-sweeper.tick.failed-to-dial","data":{"error":"all worker SSH gateways unreachable","session":"7.12"}}
{"timestamp":"2025-02-20T08:53:41.127581953Z","level":"error","source":"worker","message":"worker.volume-sweeper.tick.failed-to-get-volumes-to-destroy","data":{"error":"all worker SSH gateways unreachable","session":"7.12"}}
{"timestamp":"2025-02-20T08:53:41.127472963Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-connect-to-tsa","data":{"error":"dial tcp 172.20.68.15:2222: connect: connection refused","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:41.127611359Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.dial.failed-to-connect-to-any-tsa","data":{"error":"all worker SSH gateways unreachable","session":"6.12.2"}}
{"timestamp":"2025-02-20T08:53:41.127631353Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-dial","data":{"error":"all worker SSH gateways unreachable","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:41.127643327Z","level":"error","source":"worker","message":"worker.container-sweeper.tick.failed-to-get-containers-to-destroy","data":{"error":"all worker SSH gateways unreachable","session":"6.12"}}
{"timestamp":"2025-02-20T08:53:42.063858838Z","level":"info","source":"worker","message":"worker.beacon-runner.restarting","data":{"session":"4"}}
{"timestamp":"2025-02-20T08:53:42.064019795Z","level":"debug","source":"worker","message":"worker.beacon-runner.beacon.start","data":{"session":"4.1"}}

[taylorsilva]

Looks like these logs are from the worker. It sounds like your worker can no longer connect to your web node(s). I would guess the issue is on the web node since the error says connection refused.

Did you try restarting both the worker(s) and web node(s)?
If yes, did anything change networking-wise recently? Is there a firewall in front of the web node that could be blocking this traffic?