[Questions] Activate Khepri

[deeway]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify) 4.1.4

Erlang version used

27.3.4.4

Operating system (distribution) used

Ubuntu 24.04

How is RabbitMQ deployed?

Debian package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Status of node rabbit@svqal1202 ... Runtime

OS PID: 2151299
OS: Linux
Uptime (seconds): 4829
Is under maintenance?: false
RabbitMQ version: 4.1.4
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: rabbit@svqal1202
Erlang configuration: Erlang/OTP 27 [erts-15.2.7.3] [source] [64-bit] [smp:240:2] [ds:240:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.0.13 30 Jan 2024
Erlang processes: 606 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_management
• rabbitmq_management_agent
• rabbitmq_web_dispatch
• amqp_client
• cowboy
• oauth2_client
• jose

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@svqal1202
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@svqal1202/quorum/rabbit@svqal1202

Config files

• /etc/rabbitmq/advanced.config
• /etc/rabbitmq/rabbitmq.conf

Log file(s)

• /rabbitmq_data/logs/[email protected]

Alarms

(none)

Tags

(none)

Memory

Total memory used: 0.3003 gb
Calculation strategy: rss
Memory high watermark setting: 0.6 of available memory, computed to: 3.668 gb

allocated_unused: 0.2347 gb (56.46 %)
other_system: 0.127 gb (30.55 %)
code: 0.0251 gb (6.05 %)
other_proc: 0.0224 gb (5.38 %)
metrics: 0.0066 gb (1.59 %)
atom: 0.0012 gb (0.28 %)
msg_index: 0.0012 gb (0.28 %)
plugins: 0.0008 gb (0.2 %)
metadata_store: 0.0002 gb (0.04 %)
mgmt_db: 0.0002 gb (0.04 %)
binary: 0.0001 gb (0.03 %)
metadata_store_ets: 0.0 gb (0.01 %)
quorum_ets: 0.0 gb (0.0 %)
connection_other: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
mnesia: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)
other_ets: -0.0038 gb (-0.92 %)

File Descriptors

Total: 0, limit: 32671

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 7.0539 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: 0.0.0.0, port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: 0.0.0.0, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-10-30 12:17:10.662031+00:00 [info] <0.211.0> Running boot step tracking_metadata_store defined by app rabbit 2025-10-30 12:17:10.662121+00:00 [info] <0.554.0> Setting up a table for connection tracking on this node: tracked_connection 2025-10-30 12:17:10.662159+00:00 [info] <0.554.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost 2025-10-30 12:17:10.662208+00:00 [info] <0.554.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user 2025-10-30 12:17:10.662252+00:00 [info] <0.554.0> Setting up a table for channel tracking on this node: tracked_channel 2025-10-30 12:17:10.662301+00:00 [info] <0.554.0> Setting up a table for channel tracking on this node: tracked_channel_per_user 2025-10-30 12:17:10.662363+00:00 [info] <0.211.0> Running boot step networking_metadata_store defined by app rabbit 2025-10-30 12:17:10.662418+00:00 [info] <0.211.0> Running boot step feature_flags defined by app rabbit 2025-10-30 12:17:10.662627+00:00 [info] <0.211.0> Running boot step auth_backend_plugins_check defined by app rabbit 2025-10-30 12:17:10.662655+00:00 [info] <0.211.0> Running boot step codec_correctness_check defined by app rabbit 2025-10-30 12:17:10.662790+00:00 [info] <0.211.0> Running boot step external_infrastructure defined by app rabbit 2025-10-30 12:17:10.662899+00:00 [info] <0.211.0> Running boot step rabbit_event defined by app rabbit 2025-10-30 12:17:10.662979+00:00 [info] <0.211.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit 2025-10-30 12:17:10.663032+00:00 [info] <0.211.0> Running boot step rabbit_auth_mechanism_anonymous defined by app rabbit 2025-10-30 12:17:10.663070+00:00 [info] <0.211.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit 2025-10-30 12:17:10.663111+00:00 [info] <0.211.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit 2025-10-30 12:17:10.663147+00:00 [info] <0.211.0> Running boot step rabbit_exchange_type_direct defined by app rabbit 2025-10-30 12:17:10.663190+00:00 [info] <0.211.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit 2025-10-30 12:17:10.663229+00:00 [info] <0.211.0> Running boot step rabbit_exchange_type_headers defined by app rabbit 2025-10-30 12:17:10.663266+00:00 [info] <0.211.0> Running boot step rabbit_exchange_type_local_random defined by app rabbit 2025-10-30 12:17:10.663304+00:00 [info] <0.211.0> Running boot step rabbit_exchange_type_topic defined by app rabbit 2025-10-30 12:17:10.663340+00:00 [info] <0.211.0> Running boot step rabbit_priority_queue defined by app rabbit 2025-10-30 12:17:10.663360+00:00 [info] <0.211.0> Priority queues enabled, real BQ is rabbit_variable_queue 2025-10-30 12:17:10.663422+00:00 [info] <0.211.0> Running boot step kernel_ready defined by app rabbit 2025-10-30 12:17:10.663470+00:00 [info] <0.211.0> Running boot step pg_local_amqp_connection defined by app rabbit 2025-10-30 12:17:10.671199+00:00 [info] <0.211.0> Running boot step pg_local_amqp_session defined by app rabbit 2025-10-30 12:17:10.671270+00:00 [info] <0.211.0> Running boot step rabbit_sysmon_minder defined by app rabbit 2025-10-30 12:17:10.671380+00:00 [info] <0.211.0> Running boot step rabbit_epmd_monitor defined by app rabbit 2025-10-30 12:17:10.672209+00:00 [info] <0.563.0> epmd monitor knows us, inter-node communication (distribution) port: 25672 2025-10-30 12:17:10.672277+00:00 [info] <0.211.0> Running boot step guid_generator defined by app rabbit 2025-10-30 12:17:10.674310+00:00 [info] <0.211.0> Running boot step rabbit_node_monitor defined by app rabbit 2025-10-30 12:17:10.674384+00:00 [info] <0.567.0> Starting rabbit_node_monitor (partition handling strategy unapplicable with Khepri) 2025-10-30 12:17:10.674464+00:00 [info] <0.211.0> Running boot step delegate_sup defined by app rabbit 2025-10-30 12:17:10.674780+00:00 [info] <0.211.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit 2025-10-30 12:17:10.674827+00:00 [info] <0.211.0> Running boot step core_initialized defined by app rabbit 2025-10-30 12:17:10.674853+00:00 [info] <0.211.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit 2025-10-30 12:17:10.674888+00:00 [info] <0.211.0> Running boot step rabbit_classic_queue defined by app rabbit 2025-10-30 12:17:10.674942+00:00 [info] <0.211.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit 2025-10-30 12:17:10.675002+00:00 [info] <0.211.0> Running boot step rabbit_definitions_hashing defined by app rabbit 2025-10-30 12:17:10.675045+00:00 [info] <0.211.0> Running boot step rabbit_exchange_parameters defined by app rabbit 2025-10-30 12:17:10.676867+00:00 [info] <0.211.0> Running boot step rabbit_policies defined by app rabbit 2025-10-30 12:17:10.677124+00:00 [info] <0.211.0> Running boot step rabbit_policy defined by app rabbit 2025-10-30 12:17:10.677167+00:00 [info] <0.211.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit 2025-10-30 12:17:10.677234+00:00 [info] <0.211.0> Running boot step rabbit_quorum_queue defined by app rabbit 2025-10-30 12:17:10.677305+00:00 [info] <0.211.0> Running boot step rabbit_stream_coordinator defined by app rabbit 2025-10-30 12:17:10.677471+00:00 [info] <0.211.0> Running boot step rabbit_vhost_limit defined by app rabbit 2025-10-30 12:17:10.677521+00:00 [info] <0.211.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management 2025-10-30 12:17:10.677569+00:00 [info] <0.211.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent 2025-10-30 12:17:10.677655+00:00 [info] <0.211.0> Management plugin: using rates mode 'basic' 2025-10-30 12:17:10.677826+00:00 [info] <0.211.0> Running boot step recovery defined by app rabbit 2025-10-30 12:17:10.677998+00:00 [info] <0.211.0> Running boot step prevent_startup_if_node_was_reset defined by app rabbit 2025-10-30 12:17:10.678025+00:00 [info] <0.211.0> Running boot step empty_db_check defined by app rabbit 2025-10-30 12:17:10.678108+00:00 [info] <0.211.0> Will seed default virtual host and user... 2025-10-30 12:17:10.678198+00:00 [info] <0.211.0> Adding vhost '/' (description: 'Default virtual host', tags: []) 2025-10-30 12:17:10.696092+00:00 [info] <0.594.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@svqal1202/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists 2025-10-30 12:17:10.696661+00:00 [info] <0.594.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048' 2025-10-30 12:17:10.705660+00:00 [info] <0.594.0> Starting message stores for vhost '/' 2025-10-30 12:17:10.706842+00:00 [info] <0.594.0> Started message store of type transient for vhost '/' 2025-10-30 12:17:10.733774+00:00 [warning] <0.606.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch 2025-10-30 12:17:10.734607+00:00 [info] <0.594.0> Started message store of type persistent for vhost '/' 2025-10-30 12:17:10.734759+00:00 [info] <0.594.0> Recovering 0 queues of type rabbit_classic_queue took 37ms 2025-10-30 12:17:10.734783+00:00 [info] <0.594.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms 2025-10-30 12:17:10.734819+00:00 [info] <0.594.0> Recovering 0 queues of type rabbit_stream_queue took 0ms 2025-10-30 12:17:10.759496+00:00 [info] <0.211.0> Created user 'admin' 2025-10-30 12:17:10.795439+00:00 [info] <0.211.0> Successfully set user tags for user 'admin' to [administrator] 2025-10-30 12:17:10.829667+00:00 [info] <0.211.0> Successfully set permissions for user 'admin' in virtual host '/' to '.*', '.*', '.*' 2025-10-30 12:17:10.829735+00:00 [info] <0.211.0> Running boot step rabbit_observer_cli defined by app rabbit 2025-10-30 12:17:10.829889+00:00 [info] <0.211.0> Running boot step rabbit_core_metrics_gc defined by app rabbit 2025-10-30 12:17:10.830020+00:00 [info] <0.211.0> Running boot step background_gc defined by app rabbit 2025-10-30 12:17:10.830135+00:00 [info] <0.211.0> Running boot step routing_ready defined by app rabbit 2025-10-30 12:17:10.830152+00:00 [info] <0.211.0> Running boot step pre_flight defined by app rabbit 2025-10-30 12:17:10.830172+00:00 [info] <0.211.0> Running boot step notify_cluster defined by app rabbit 2025-10-30 12:17:10.830211+00:00 [info] <0.211.0> Running boot step networking defined by app rabbit 2025-10-30 12:17:10.830228+00:00 [info] <0.211.0> Running boot step rabbit_quorum_queue_periodic_membership_reconciliation defined by app rabbit 2025-10-30 12:17:10.830358+00:00 [info] <0.211.0> Running boot step definition_import_worker_pool defined by app rabbit 2025-10-30 12:17:10.830388+00:00 [info] <0.266.0> Starting worker pool 'definition_import_pool' with 2 processes in it 2025-10-30 12:17:10.830542+00:00 [info] <0.211.0> Running boot step cluster_name defined by app rabbit 2025-10-30 12:17:10.830590+00:00 [info] <0.211.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-uXS167DK0qDxQPOJLhwDmQ' 2025-10-30 12:17:10.833020+00:00 [info] <0.211.0> Running boot step virtual_host_reconciliation defined by app rabbit 2025-10-30 12:17:10.833132+00:00 [info] <0.211.0> Running boot step direct_client defined by app rabbit 2025-10-30 12:17:10.833186+00:00 [info] <0.211.0> Running boot step cluster_tags defined by app rabbit 2025-10-30 12:17:10.834805+00:00 [info] <0.211.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management 2025-10-30 12:17:10.834876+00:00 [info] <0.638.0> Resetting node maintenance status 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> Deprecated features: `management_metrics_collection`: Feature `management_metrics_collection` is deprecated. 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> By default, this feature can still be used for now. 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> Its use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined. 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> To continue using this feature when it is not permitted by default, set the following parameter in your configuration: 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> "deprecated_features.permit.management_metrics_collection = true" 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> To test RabbitMQ as if the feature was removed, set this in your configuration: 2025-10-30 12:17:11.035766+00:00 [warning] <0.660.0> "deprecated_features.permit.management_metrics_collection = false" 2025-10-30 12:17:11.100723+00:00 [info] <0.696.0> Management plugin: HTTP (non-TLS) listener started on port 15672 2025-10-30 12:17:11.100847+00:00 [info] <0.724.0> Statistics database started. 2025-10-30 12:17:11.100945+00:00 [info] <0.723.0> Starting worker pool 'management_worker_pool' with 3 processes in it 2025-10-30 12:17:11.101160+00:00 [info] <0.638.0> Ready to start client connection listeners 2025-10-30 12:17:11.110709+00:00 [info] <0.748.0> started TCP listener on 0.0.0.0:5672 2025-10-30 12:17:11.172037+00:00 [info] <0.638.0> Server startup complete; 3 plugins started. 2025-10-30 12:17:11.172037+00:00 [info] <0.638.0> * rabbitmq_management 2025-10-30 12:17:11.172037+00:00 [info] <0.638.0> * rabbitmq_management_agent 2025-10-30 12:17:11.172037+00:00 [info] <0.638.0> * rabbitmq_web_dispatch 2025-10-30 12:17:11.236387+00:00 [info] <0.10.0> Time to start RabbitMQ: 6406 ms

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

######################################## # 🐇 Configuration de base ########################################

########################################

🌐 Réseau

########################################

Port AMQP standard

listeners.tcp.default = 5672

########################################

🧩 Plugins

########################################

Activer le plugin de gestion

management.listener.port = 15672
management.listener.ip = 0.0.0.0

########################################

🧩 SSL

########################################

#listeners.ssl.default = 5671
#ssl_options.cacertfile = /etc/rabbitmq/ssl/cacert.pem
#ssl_options.certfile = /etc/rabbitmq/ssl/cert.pem
#ssl_options.keyfile = /etc/rabbitmq/ssl/key.pem
ssl_options.verify = verify_none
#ssl_options.fail_if_no_peer_cert = true

########################################

🔐 Sécurité

########################################

Utilisateur par défaut (à changer en prod)

default_user = admin
default_pass = admin

Steps to deploy RabbitMQ cluster

No cluster / standalone mode

Steps to reproduce the behavior in question

install rabbitmq from https://www.rabbitmq.com/docs/install-debian
install erlang from the same rabbitmq url
Activate khepri from features flag -> rabbitmqctl enable_feature_flag --experimental khepri_db (follow How to Enable Khepri | RabbitMQ)

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

[ {rabbit, [ {database, khepri} ]} ].

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

First I'm trying to active khepri on a fresh dev server but rabbitmqctl status show always the usage of mnesia metadata.
Change erlang conf in advanced.config with this one but no way :
[
{khepri, [
{default_ra_system, "/rabbitmq_data/khepri"},
{default_store_id, khepri}
]}
].

Use rabbitmq-env.conf also :
RABBITMQ_NODENAME=rabbit@svqal1202
RABBITMQ_LOGS=/rabbitmq_data/logs/[email protected]
RABBITMQ_FEATURE_FLAGS=khepri_db
#CONFIG_FILE=/etc/rabbitmq/advanced.config
RABBITMQ_ADVANCED_CONFIG_FILE=/etc/rabbitmq/advanced.config

Result of rabbitmqctl status :
Data directory
Node data directory: /var/lib/rabbitmq/mnesia/rabbit@svqal1202
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@svqal1202/quorum/rabbit@svqal1202

Result of diagnostic metadata_store
rabbitmq-diagnostics metadata_store_status
Status of metadata store on node rabbit@svqal1202 ...
lqqqqqqqqqqqqqqqqqqwqqqqqqqqqqqqwqqqqqqqqqqqqwqqqqqqqqqqqqqqqqwqqqqqqqqqqqqqqwqqqqqqqqqqqqqqwqqqqqqqqqqqqqqwqqqqqqqqqqqqqqqqwqqqqqqwqqqqqqqqqqqqqqqqqk
x Node Name x Raft State x Membership x Last Log Index x Last Written x Last Applied x Commit Index x Snapshot Index x Term x Machine Version x
tqqqqqqqqqqqqqqqqqqnqqqqqqqqqqqqnqqqqqqqqqqqqnqqqqqqqqqqqqqqqqnqqqqqqqqqqqqqqnqqqqqqqqqqqqqqnqqqqqqqqqqqqqqnqqqqqqqqqqqqqqqqnqqqqqqnqqqqqqqqqqqqqqqqqu
x rabbit@svqal1202 x leader x voter x 49 x 49 x 49 x 49 x -1 x 6 x 1 x
mqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqvqqqqqqqqqqqqvqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqvqqqqqqqqqqqqqqvqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqvqqqqqqvqqqqqqqqqqqqqqqqqj

In a second time, i'd like to move data folder to a second disk /rabbitmq_data/khepri, is this possible ?

Thanks for your help.

[mkuratczyk]

the formatting of your message is hard to read, but rabbit@svqal1202 leader means that Khepri is enabled. Side note: I'd highly recommend against using RABBITMQ_FEATURE_FLAGS unless you have a very good reason, especially in this non-relative manner. RABBITMQ_FEATURE_FLAGS=+khepri_db means "add khepri to the list of feature flags enabled by default". But RABBITMQ_FEATURE_FLAGS=khepri_db means "only enable the khepri feature flag" (don't enable flags that would be enabled by default).

[deeway]

Ok, this line has been commented. I do a service restart but still have mnesia referenced has node data directory, it must create khepri directory and migrate existing data ?

[mkuratczyk]

No, the data has been migrated already as part of enabling khepri. The folder is just a historical artefact. Even with Khepri enabled by default, the mnesia folder is created, because that's how things were done for 18 years before Khepri was introduced into RabbitMQ.

Anyway, you are all set. Enabling the feature flag is the only step necessary to migrate

[deeway]

@mkuratczyk Good i will go on now and try to migrate definitions from v3.9/3.11 to the newone.
The second question stay unanswer, how to move datastore directory to another disk ? You close this questions too quickly.

[deeway]

Found this tuto to help me, hope this will work.
https://myupbeat.wordpress.com/2014/02/07/chaging-rabbitmq-mnesia-database-location/

[deeway]

Seems to be good :
Data directory

Node data directory: /rabbitmq_data/khepri/rabbit@svqal1202
Raft data directory: /rabbitmq_data/khepri/rabbit@svqal1202/quorum/rabbit@svqal1202

Have a good day and a big thanks for your answers