Rootless, IPv6, Caddy as Proxy: Domain Check fails/External IP not reachable from out of the Container

[obel1x]

Steps to reproduce

1. Install docker-rootless + docker-compose- plugin
2. Set up Caddy as a reverse proxy as described for nextcloud.testdomain.com redirected to AIO- Apache Port 11000
3. Enable IPv6 as described here, make nextcloud part of that network
4. Start Nextcloud- Setup via Docker- Compose

Expected behavior

Domaincheck should work, Nextcloud would reach the FQDN given from within the Nextcloud- container

Actual behavior

Domain-Check fails. When ignoring, checks in admin- overview will fail.

Host OS

OpenSUsE Leap 15.5 / Docker v26

Nextcloud AIO version

Nextcloud AIO v8.2.1

Current channel

Stable

Other valuable info

In the given scenario i have not been able to get the Containers reach the Host with its FQDN and external IP while there seems to be no route inside of the container to the destination.

I the following scenario nextcloud.testdomain.com is a fqdn, which can be reached from the internet.

My current setup after running AIO is the following - accessing the Container (aio-netxcloud) via Docker Console:

30057875aac2:/var/www/html# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
38: eth0@if39: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.11/16 brd 172.19.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:db8:3::b/64 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe13:b/64 scope link 
       valid_lft forever preferred_lft forever

The host is part of network nextcloud-aio with address 2001:db8:3::b.

Name lookup is fine, but not modified:

30057875aac2:/var/www/html# nslookup nextcloud.testdomain.com
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   nextcloud.testdomain.com
Address: 2a00:1f:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

30057875aac2:/var/www/html# curl nextcloud-aio-apache:11000
(OK)

2a00:1f:xxxx:... is the external IPv6 which is reachable from the internet which works using name resolved proxy via caddy, i can access that when skipping domain_ceck in compose env.

Problem:

30057875aac2:/var/www/html# curl https://nextcloud.testdomain.com
curl: (7) Failed to connect to nextcloud.testdomain.com port 443 after 1051 ms: Couldn't connect to server

Workaround:

echo "2001:db8:3::2 nextcloud.testdomain.com" >> /etc/hosts

Where 2001:db8:3::2 is the internal ipv6 of the caddy- container. This makes connections work and checks succeed (but makes Domain-Cecking at setup nonsense somewhat).

My current goal would be to make containers be able to access the right ipv6 given by dns.

I have:

• changed the Firewall as described here (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-resolve-firewall-problems-with-fedora-linux-rhel-os-centos-suse-linux-and-others)
• enabled net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables
• started br_netfilter module at startup

but none helped so far. Maybe the problem is, that the container has no tools for ipv6? (ipv6 route is not working)

What a i missing to get it working?

[szaimen]

Hi, have you followed https://github.com/nextcloud/all-in-one/blob/main/docker-ipv6-support.md?

[obel1x]

yes, i have. The problem seems to be any container is not able to reach the internet with my setup and additionally the internal dns of docker doe not serve that domain as internal adress, but always the global ip (which is fine basically, but won't work due to first problem).

my daemon.json:

{
 "experimental": true,
 "ipv6": true,
 "fixed-cidr-v6": "fd12:3456:789a:1::/64",
 "ip6tables": true,
 "log-opts": {
   "max-size": "10m",
   "max-file": "5"
 }
}

my aio network:

docker@hostx:~/.config/docker> docker network inspect nextcloud-aio
[
    {
        "Name": "nextcloud-aio",
        "Id": "c1a8b8b39f46f81868fe3776edf8a1dc493c148798fadfdb3314dfa0a8b9d157",
        "Created": "2024-06-04T19:02:45.27230824+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                },
                {
                    "Subnet": "fd12:3456:789a:2::/64",
                    "Gateway": "fd12:3456:789a:2::1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "30a378be6d2b91a8e84f843d24747e83b297ed457b8111faa79d46b7235fcde5": {
                "Name": "nextcloud-aio-mastercontainer",
                "EndpointID": "8b2eeeb3aed1517a59cf2568dc80e11b7a74d6947b52dd12916345a221afe705",
                "MacAddress": "02:42:ac:12:00:06",
                "IPv4Address": "172.18.0.6/16",
                "IPv6Address": "fd12:3456:789a:2::6/64"
            },
            "5479727f27af7c614fd68cf7c56ae8ada8b72b5f7594a409383f6c5e4f4176f1": {
                "Name": "pcserver2023-mariadb-1",
                "EndpointID": "40d26448fcded6d9529990bd5c72738f86198dd64785e146898a45295df140c3",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": "fd12:3456:789a:2::3/64"
            },
            "cca94450611b2ceae6375f83815940cdffe73dca921d88a9a77c268bd16068db": {
                "Name": "pcserver2023-phpmyadmin-1",
                "EndpointID": "85a3b8bfd673fb8c5bf35e71548aa4682ab55247802d34e8a27139935fcf8660",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": "fd12:3456:789a:2::4/64"
            },
            "d00f3d5868d8d8624867c8a473727877f7c40016d660ccc655470e303d3dd121": {
                "Name": "pcserver2023-dokuwiki-1",
                "EndpointID": "fb0ab30ce63cd923c3cec76ee2bad3eb6bae95f3e124578063b3a0e9df9d7ea9",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": "fd12:3456:789a:2::2/64"
            },
            "d106399299dc466b1045b09c38f9341f6aaf39a125ae9cbdc35f8071533a3b9d": {
                "Name": "pcserver2023-caddy-1",
                "EndpointID": "2d7202686e9e2a8eb89802a3914d09004be2e5a7eba5ef42c542d4d464633857",
                "MacAddress": "02:42:ac:12:00:07",
                "IPv4Address": "172.18.0.7/16",
                "IPv6Address": "fd12:3456:789a:2::7/64"
            },
            "eba59c2693d09f7a16b2b3d7e174baf5979e6fa499cb7f3a83c52115675409e6": {
                "Name": "pcserver2023-portainer-1",
                "EndpointID": "2c84415a3b372cf00dc853de2555cadccf8c4852bfa6c9511a9e91d5fb2a646d",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": "fd12:3456:789a:2::5/64"
            }
        },
        "Options": {},
        "Labels": {}
    }
]

inside the container, i can ping ipv4 adresses like this:

455fd8f01d52:/var/www/docker-aio# ip route
default via 172.18.0.1 dev eth0 
172.18.0.0/16 dev eth0 scope link  src 172.18.0.6 
455fd8f01d52:/var/www/docker-aio# ping www.github.com
PING www.github.com (140.82.121.3): 56 data bytes
64 bytes from 140.82.121.3: seq=0 ttl=254 time=11.241 ms
64 bytes from 140.82.121.3: seq=1 ttl=254 time=11.274 ms
^C
--- www.github.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 11.241/11.257/11.274 ms

but i cannot ping6 anything:

455fd8f01d52:/var/www/docker-aio# ip -6 route
fd12:3456:789a:2::/64 dev eth0  metric 256 
fe80::/64 dev eth0  metric 256 
default via fd12:3456:789a:2::1 dev eth0  metric 1024 
multicast ff00::/8 dev eth0  metric 256 
455fd8f01d52:/var/www/docker-aio# ping6 www.heise.de
PING www.heise.de (2a02:2e0:3fe:1001:7777:772e:2:85): 56 data bytes
^C
--- www.heise.de ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

so i can only reach the host from the host/internet but not from inside the container.

What could be the problem? Maybe some ipv6 problem, maybe because of rootless?

[szaimen]

Hi, sounds like you should open a thread in a docker forum in order to find a solution. I fear I cannot help further on the topic as it sounds like a problem with docker in general.

[obel1x]

Well yes, that may be a problem with IPV6 and Docker at rootless in general, but it leads to the Documentation not beeing usable here and my goal would be to find a working solution and to make the described setup work. I am searching since weeks not finding a solution, the given workaround for me won't really work either.

Currently, IPV6 only on rootless with Caddy does not seem to work at all. So please don't mark as answered as long as it is not.

[szaimen]

Did you already open a thread in a docker forum and can post the link to it here?

[obel1x]

it indeed looks like a general problem with docker+rootless not being able to make outgoing ipv6 connections. When setting up rootful or using ipv4 everything works.
Have opened an issue in docker moby/moby#48257

[szaimen]

Thanks for opening an issue upstrean!

Btw, we have an updated ipv6 documentation now: https://github.com/nextcloud/all-in-one/blob/main/docker-ipv6-support.md

[obel1x]

Fyi: That issue has been fixed with new docker / rootlesskit (now 2.3.2) - and works as expected with pasta- drivers.