Unable boot rabbitmq 3.9 on docker compose: failed to open log file at '/var/log/rabbitmq/rabbit@xxxxxx_upgrade.log', reason: permission denied

[juanbassist]

Hello I am expirementing with docker compose, the update to rabbitmq 3.9 but I always get the following error regarding permission problems. If I do the same exercise in rabbitmq 3.8, it works without a problem.

Ubuntu 20.04 LTS, uname -a:
Linux server 5.4.0-89-generic # 100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 x86_64 x86_64 GNU / Linux

Docker compose RabbitMq 3.9 - Fail

version: "3.8"
services:
  rabbitmq:
    image: rabbitmq:3.9-management-alpine
    container_name: 'rabbitmq'
    ports:
        - 5672:5672
        - 15672:15672
    volumes:
        - ./rabbitmq/data/:/var/lib/rabbitmq/
        - ./rabbitmq/log/:/var/log/rabbitmq/

Error Logs....

rabbitmq         | 2021-11-17 23:42:17.634897+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
rabbitmq         | 2021-11-17 23:42:17.645693+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
rabbitmq         | 2021-11-17 23:42:17.645735+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
rabbitmq         | 2021-11-17 23:42:17.645759+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
rabbitmq         | 2021-11-17 23:42:17.645820+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
rabbitmq         | 2021-11-17 23:42:17.645845+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
rabbitmq         | 2021-11-17 23:42:17.645860+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
rabbitmq         | 2021-11-17 23:42:17.645879+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
rabbitmq         | 2021-11-17 23:42:17.929262+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
rabbitmq         | 2021-11-17 23:42:17.929344+00:00 [noti] <0.222.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
rabbitmq         | 
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> 
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> BOOT FAILED
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> ===========
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> failed to open log file at '/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log', reason: permission denied
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> 
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> 
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> BOOT FAILED
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> ===========
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> failed to open log file at '/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log', reason: permission denied
rabbitmq         | 2021-11-17 23:42:17.951623+00:00 [erro] <0.222.0> 
rabbitmq         | BOOT FAILED
rabbitmq         | ===========
rabbitmq         | failed to open log file at '/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log', reason: permission denied
rabbitmq         | 
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>   crasher:
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     initial call: application_master:init/4
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     pid: <0.221.0>
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     registered_name: []
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     exception exit: {{cannot_log_to_file,
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                          "/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log",
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                          eacces},
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                      {rabbit,start,[normal,[]]}}
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>       in function  application_master:init/4 (application_master.erl, line 142)
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     ancestors: [<0.220.0>]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     message_queue_len: 1
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     messages: [{'EXIT',<0.222.0>,normal}]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     links: [<0.220.0>,<0.44.0>]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     dictionary: []
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     trap_exit: true
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     status: running
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     heap_size: 610
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     stack_size: 28
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     reductions: 178
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>   neighbours:
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0> 
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>   crasher:
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     initial call: application_master:init/4
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     pid: <0.221.0>
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     registered_name: []
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     exception exit: {{cannot_log_to_file,
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                          "/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log",
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                          eacces},
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>                      {rabbit,start,[normal,[]]}}
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>       in function  application_master:init/4 (application_master.erl, line 142)
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     ancestors: [<0.220.0>]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     message_queue_len: 1
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     messages: [{'EXIT',<0.222.0>,normal}]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     links: [<0.220.0>,<0.44.0>]
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     dictionary: []
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     trap_exit: true
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     status: running
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     heap_size: 610
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     stack_size: 28
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>     reductions: 178
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0>   neighbours:
rabbitmq         | 2021-11-17 23:42:18.952454+00:00 [erro] <0.221.0> 
rabbitmq         | 2021-11-17 23:42:18.958877+00:00 [noti] <0.44.0> Application rabbit exited with reason: {{cannot_log_to_file,"/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log",eacces},{rabbit,start,[normal,[]]}}
rabbitmq         | 2021-11-17 23:42:18.958877+00:00 [noti] <0.44.0> Application rabbit exited with reason: {{cannot_log_to_file,"/var/log/rabbitmq/rabbit@261eb6b93a6a_upgrade.log",eacces},{rabbit,start,[normal,[]]}}

I tried

• Create directories manually ./rabbitmq/data/ ./rabbitmq/log/ in host with root permissions.
• Run docker compose with sudo

Unsuccessfully.

Docker compose RabbitMq 3.8 - works Fine

version: "3.8"
services:
  rabbitmq:
    image: rabbitmq:3.8.25-management-alpine
    container_name: 'rabbitmq'
    ports:
        - 5672:5672
        - 15672:15672
    volumes:
        - ./rabbitmq/data/:/var/lib/rabbitmq/
        - ./rabbitmq/log/:/var/log/rabbitmq/

regards.

[wglambert]

It wants access to that upgrade.log but your bind-mounted volume doesn't allow for the rabbitmq user to access it

The container runs as rabbitmq not root, and running docker-compose as root doesn't affect what user the container runs as.

$ docker run -d --name rabbit rabbitmq:management
2e9395be4a14324b28b669918b761b68f811f7a2ce403a2003b82136e1cee360

$ docker exec -it rabbit bash
root@2e9395be4a14:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
rabbitmq     1  0.3  0.0   2612  1648 ?        Ss   16:32   0:00 /bin/sh /opt/rabbitmq/sbin/rabbitmq-server
rabbitmq    20 55.8  7.2 1820000 148808 ?      Sl   16:32   0:07 /usr/local/lib/erlang/erts-12.1.3/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlm
rabbitmq    27  0.0  0.0   2508   524 ?        Ss   16:32   0:00 erl_child_setup 1048576
rabbitmq    51  0.0  0.0   3888    92 ?        S    16:32   0:00 /usr/local/lib/erlang/erts-12.1.3/bin/epmd -daemon
rabbitmq    72  0.0  0.0   3888   852 ?        Ss   16:32   0:00 inet_gethost 4
rabbitmq    73  0.0  0.0   3968  1756 ?        S    16:32   0:00 inet_gethost 4
root        74  1.0  0.1   5996  3700 pts/0    Ss   16:32   0:00 bash
root        87  0.0  0.1   7652  3340 pts/0    R+   16:32   0:00 ps aux

[smorokin]

I have the same problem. The container does create its directories (in my case under ~/.docker/rabbitmq) with the following identical uids and gids for 3.8 and 3.9:

user@machine:~/.docker/rabbitmq$ ll -n
total 16
drwxr-xr-x 4   0 0 4096 Jan 12 15:58 ./
drwxr-xr-x 3   0 0 4096 Jan 12 15:58 ../
drwxr-xr-x 3 999 0 4096 Jan 12 15:58 data/
drwxr-xr-x 2   0 0 4096 Jan 12 15:58 log/

Both 3.8 and 3.9 fail to write to log, but only the 3.9 container crashes after this:
3.8:

rabbitmq_1    | 2022-01-12 15:33:09.842 [error] <0.283.0> Failed to open crash log file /var/log/rabbitmq/log/crash.log with error: permission denied
rabbitmq_1    | 2022-01-12 15:33:09.842 [debug] <0.285.0> Lager installed handler lager_backend_throttle into lager_event
rabbitmq_1    | 2022-01-12 15:33:14.508 [info] <0.44.0> Application mnesia started on node rabbit@rabbitmq
rabbitmq_1    | 2022-01-12 15:33:14.509 [info] <0.273.0> 
rabbitmq_1    |  Starting RabbitMQ 3.8.27 on Erlang 24.2 [jit]
rabbitmq_1    |  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
rabbitmq_1    |  Licensed under the MPL 2.0. Website: https://rabbitmq.com

3.9:

rabbitmq_1    | BOOT FAILED
rabbitmq_1    | 2022-01-12 15:35:26.970342+00:00 [erro] <0.222.0> 
rabbitmq_1    | 2022-01-12 15:35:26.970342+00:00 [erro] <0.222.0> BOOT FAILED
rabbitmq_1    | 2022-01-12 15:35:26.970342+00:00 [erro] <0.222.0> ===========
rabbitmq_1    | 2022-01-12 15:35:26.970342+00:00 [erro] <0.222.0> failed to open log file at '/var/log/rabbitmq/rabbit@rabbitmq_upgrade.log', reason: permission denied

They also crash on different files. It does not matter if its the "normal" or -management containers.

Is root:root correct for the log directory? It should be changed to rabbitmq:rabbitmq in the Dockerfile here but somehow that does not work. First I thought the -f flag for chown is suppressing an error

$ chown --help
[...]
  -f, --silent, --quiet  suppress most error messages

but it did not throw an error when I removed it. Btw: That -f is maybe a bug caused by a very poor choice for the parameter flag.

Now I'm out of ideas.

[yosifkit]

@smorokin, the chown in the Dockerfile only applies to the folder in the image. The entrypoint script fixes permissions of /var/lib/rabbitmq on container run; any other directories provided as a volume need to already have owner/permissions such that the rabbitmq user of the container can access it.

It looks like you are doing the same volume mount ./rabbitmq/log/:/var/log/rabbitmq/. If that local directory (./rabbitmq/log/) does not exist before the docker run (or docker compose), then docker will create it before starting the container and it will unfortunately be owned by root (and then the mounted folder overrides the permissions of the container folder).

Possible solutions:

• (pre-creating and) chowning the folder to user 999 (i.e. rabbitmq of the container)
• create both folders (logs and data) as your user id (e.g. 1000) and then run the container as your user/group: docker run --user 1000:1000

[smorokin]

@yosifkit thank you for the fast reply. Yes, I have the same setup as juanbassist.
In my circumstances it would not be good to do additional steps to running docker-compose up.

Is it possible for the entrypoint script to also change the permissions of the log folder?

Or can i just remove the volume mount for log and just lose that one migration log? As far as I saw that was the only thing written to disk.

[smorokin]

@yosifkit You can ignore my last reply. I have just used docker volumes.
For everybody looking for a solution to run the rabbitmq image with docker compose, try the following:

version: "3.7"
services:
  your_service:
    build:
      context: .
      target: production
    networks:
      - rabbitmq_net
  rabbitmq:
    image: rabbitmq:3.9-management
    hostname: rabbitmq
    ports:
      - 5672:5672
      - 15672:15672
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq/
      - rabbitmq_log:/var/log/rabbitmq/
    networks:
      - rabbitmq_net

volumes:
    rabbitmq_data:
    rabbitmq_log:

networks:
  rabbitmq_net:
    driver: bridge

[smorokin]

Since this got some attention: The docker-compose file above tells docker to create volumes (In the example above rabbitmq_data & rabbitmq_log) for the mounted directories. This fixes the issues with the permissions at the cost that you can no longer inspect the logs that easily from the host. You can find the new mount point by running docker volume inspect [your_volume_name] and looking for the Mountpoint entry. This is the directory on your host that contains the files of the volume (on my machine it's /var/lib/docker/volumes/[volume_name]/_data).

[chudaozhe]

注意，如果是Linux，第一次启动可能失败。提示./rabbitmq/log/目录没有写入权限，这时执行chmod -R 777 log/即可，另一个data目录千万不要更改权限，无需处理

[mojtabajahannia]

Hello
I tried very hard to get the rabbit logs out of its directory, but I could not solve the problem, but this way we can prevent errors so that we have the logs inside the data directory.
It's not very good, at least for me because I like my program logs to be all in one place, but I did it to get through this crisis and deal with the rest of the issues.
In the end, if I reach a more favorable result, I will definitely inform

   ...

    volumes:
        - ./rabbitmq/:/var/lib/rabbitmq/
        - ./rabbitmq/log/:/var/log/rabbitmq/

  ...

[tyknkd]

Thank you! This worked for me!

[chudaozhe]

@mojtabajahannia No, you can put the logs anywhere. Note that if it is Linux, the first boot may fail. Tip./rabbitmq/log/directory does not have write permissions, then execute chmod -R 777 ./rabbitmq/log/, and the other ./rabbitmq/data/directory must not change the permissions, no need to deal with it

[Juanes2499]

Hello, I solved this problem in this way:

1. Go to the volume folder in your machine, execute ll command:
   

2. Check if the log folder has the same user of data folder, if not, change the user folder to the user than data folder, execute sudo chwon -R systemd-network:root log/
   

[karolskolasinski]

sudo chwon -R systemd-network:root log/

it's a typo here, should be:

sudo chown -R systemd-network:root log/

[RuntimeRacer]

Hello, I solved this problem in this way:

1. Go to the volume folder in your machine, execute ll command:
   
2. Check if the log folder has the same user of data folder, if not, change the user folder to the user than data folder, execute sudo chwon -R systemd-network:root log/
   

OMG thank you!! I was searching over 30 minutes for a solution; the answer marked as the correct one did not work for me; but this finally worked!

[stolbov-stanislav]

If someone has the same problem, but the solution will not be found for a long time: take a closer look at the paths that you specified in the volume bindings.

For example, my docker compose yml that caused the error looked something like this:

version: "3.8"

services:

  rabbitmq:
    image: rabbitmq:3.11-management-alpine
    hostname: rabbitmq
    ports:
        ...
    environment:
      RABBITMQ_DEFAULT_USER: rmuser
      RABBITMQ_DEFAULT_PASS: rmpassword
    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq/
...

Downgrading to version 3.6-management-alpine helped eliminate the error, but later, when I had to go back to version 3.11-management-alpine, I found that the error was caused by an incorrect path inside the container, which is specified in the bindings. For example:

    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq/

need to fix as following:

    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq

That is, remove the slash at the end of the path that leads to the RabbitMQ data inside the container.

[tyknkd]

The presence or absence of the final slash does not seem to make a difference for me (on Ubuntu 22.04)