[Bug]: Randomly receiving proxy IP as X-Forwarded-For header using Traefik or Caddy

[toverux]

Description

Hello!

I am having an issue with Traefik and Caddy proxies where they frequently (and kinda randomly) send me a forwarded-for IP address header that is actually a proxy local IP.
Ex.

• When it works I'm reading this in my debug logs as the IPs received by my app: ["172.18.0.2", "xxx.xxx.xxx.xxx"] with the first IP being a proxy IP, second is the end user IP, and I have the right X-Forwarded-For header.
• When it doesn't work I get this: ["172.18.0.2", "172.18.0.1"] with X-Forwarded-For being the second and "wrong" IP.

I tried adding traefik.entryPoints.http.forwardedHeaders.insecure=true to my project's Traefik config to see if it would just forward headers as-is, but it didn't change a thing.

I use pretty much all the defaults coming with a proxy install, didn't change anything. This is the project's config:

traefik.enable=true
traefik.http.middlewares.gzip.compress=true
traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.entryPoints=http
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.middlewares=redirect-to-https
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.rule=Host(`example.com`) && PathPrefix(`/`)
traefik.http.routers.http-0-bk4sk40wgcwc4k0gc8skwogw.service=http-0-bk4sk40wgcwc4k0gc8skwogw
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.entryPoints=https
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.middlewares=gzip
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.rule=Host(`example.com`) && PathPrefix(`/`)
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.service=https-0-bk4sk40wgcwc4k0gc8skwogw
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.tls.certresolver=letsencrypt
traefik.http.routers.https-0-bk4sk40wgcwc4k0gc8skwogw.tls=true
traefik.http.services.http-0-bk4sk40wgcwc4k0gc8skwogw.loadbalancer.server.port=4000
traefik.http.services.https-0-bk4sk40wgcwc4k0gc8skwogw.loadbalancer.server.port=4000
caddy_0.encode=zstd gzip
caddy_0.handle_path.0_reverse_proxy={{upstreams 4000}}
caddy_0.handle_path=/*
caddy_0.header=-Server
caddy_0.try_files={path} /index.html /index.php
caddy_0=https://example.com
caddy_ingress_network=coolify

I have no load balancing between server, both the proxy and the projects are on the same machine, one running instance only.

I'm completely unfamiliar with the stack (Coolify, Docker, Traefik, Caddy, etc) so I don't know where to look. I posted in the Discord, but no one was able to help.

I thought this is a serious enough bug though (and blocking, for me), since it makes us unable to track users' IPs (I need this for rate limiting and IP-banning), so I thought I could post an issue here.

Minimal Reproduction (if possible, example repository)

console.log(req.ip) with proxy headers trusting enabled in your application's HTTP router.

Exception or Error

No response

Version

4.0.0-beta.332

Cloud?

• Yes
• No

[algora-pbc]

💎 $50 bounty • algora0

Steps to solve:

1. Start working: Comment /attempt #3436 with your implementation plan
2. Submit work: Create a pull request including /claim #3436 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to coollabsio/coolify!

Add a bounty • Share on socials

[peaklabs-dev]

Are you sure the request where you do not seem to get the right IP are not just internal request in coolify? For example the healthcheck of your container?

[toverux]

@peaklabs-dev Yes, it has been verified that those are attached to users.

By the way, another info I didn't mention, the IP is somewhat sticky, when a user gets logged with the wrong IP, it stays so for the duration of their session, but next time the right IP can be used. It might be tied to a keep-alive socket they use when connecting, sometimes the right IP is passed, sometimes not, until they get another socket and the dice is rolled again. Just a theory and not sure if that helps.
Edit: that being said, I once made a few manual requests with curl (so no keepalive) and each time got the wrong IP.

[toverux]

This was too much of a pressing issue for me, so in the meantime I've disabled the Coolify-managed proxy and apt-installed Nginx that I configured manually. Sadly, I loose load balancing and rolling releases, so a fix would still be much appreciated.

[enkiark]

same issue

[toverux]

I've disabled the Coolify-managed proxy and apt-installed Nginx that I configured manually.

So this was on the host machine and it worked well. Then I migrated Nginx to a container again and had the exact same issue.

So I guess this is not an issue with Coolify's handling of proxies specifically, but something about Coolify's Docker network or Docker in general that I don't understand.