[Suggestion] Consider self-routing dead-letters a cycle

[the-mikedavis]

RabbitMQ series

4.2.x

Operating system (distribution) used

Linux

How is RabbitMQ deployed?

Other

What would you like to suggest for a future version of RabbitMQ?

Currently, if you set up dead-lettering to route to an exchange which routes back to the source queue, you can infinitely 'print' messages with #'basic.nack'{requeue = false}.

1. make run-broker
2. Create a direct exchange dlx.
3. Create a queue one with x-dead-letter-exchange set to dlx and x-dead-letter-routing-key set to dl-key.
4. Create another queue two with any arguments.
5. Create bindings from exchange dlx to queues one and two with routing key dl-key
6. Publish a few messages to queue one.
7. Use basic.get and basic.reject with requeue set to false against queue one a few times.

In the shell, for convenience:

> rr(amqp_gen_connection).
> {ok, Conn} = amqp_connection:start(#amqp_params_direct{}), {ok, Ch} = amqp_connection:open_channel(Conn).
% repeat this one:
> (fun() -> {#'basic.get_ok'{delivery_tag = T}, _} = amqp_channel:call(Ch, #'basic.get'{queue = <<"one">>}), amqp_channel:cast(Ch, #'basic.nack'{delivery_tag = T, requeue = false}) end)().

With this topology, dead-lettered messages are routed to two as well as back into one. So nacking keeps one at the same number of messages while two grows. This is consequential because it can eventually cause disk exhaustion if you keep consuming and nacking.

We already have dead-lettering cycle detection to find loops where messages end up passed between multiple queues and eventually back to the source. Would it make sense to call the route from one to one a cycle as well and drop the message? Then the messages would only end up in two.

definitions.json...

{"rabbit_version":"4.2.0+beta.4.273.g0d7b3fd","rabbitmq_version":"4.2.0+beta.4.273.g0d7b3fd","product_name":"RabbitMQ","product_version":"4.2.0+beta.4.273.g0d7b3fd","rabbitmq_definition_format":"cluster","original_cluster_name":"localhost","explanation":"Definitions of cluster 'localhost'","users":[{"name":"guest","password_hash":"5BpEZQMHDhIaSH535+CnXZhV2zUealBe+lF7BVh76Tvs3Gvd","hashing_algorithm":"rabbit_password_hashing_sha256","tags":["administrator"],"limits":{}}],"vhosts":[{"name":"/","description":"Default virtual host","metadata":{"description":"Default virtual host","tags":[],"default_queue_type":"classic"},"tags":[]}],"permissions":[{"user":"guest","vhost":"/","configure":".*","write":".*","read":".*"}],"topic_permissions":[],"parameters":[],"global_parameters":[{"name":"cluster_tags","value":[]},{"name":"internal_cluster_id","value":"rabbitmq-cluster-id-a-6wNxUEWoBxAcNHauGz1w"},{"name":"cluster_name","value":"localhost"}],"policies":[],"queues":[{"name":"two","vhost":"/","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"quorum"}},{"name":"one","vhost":"/","durable":true,"auto_delete":false,"arguments":{"x-dead-letter-exchange":"dlx","x-dead-letter-routing-key":"dl-key","x-queue-type":"quorum"}}],"exchanges":[{"name":"dlx","vhost":"/","type":"direct","durable":true,"auto_delete":false,"internal":false,"arguments":{}}],"bindings":[{"source":"dlx","vhost":"/","destination":"one","destination_type":"queue","routing_key":"dl-key","arguments":{}},{"source":"dlx","vhost":"/","destination":"two","destination_type":"queue","routing_key":"dl-key","arguments":{}}]}

[ansd]

To me, everything works as expected.

The current cycle detection by design only drops messages if the cycle is "fully automatic", i.e. without a client explicitly rejecting the message somewhere in the cycle. The idea is that messages are only dropped if they cycle forever without explicit consumer action.

For example if you set up two queues A and B with both queues having message TTL configured and both queues dead lettering to each other, this will result in endless cycles, hence the cycle detection will drop those messages.

On the other hand, in your example, if the message cycles from queue one immediately back to queue one due to a consumer intentionally rejecting the message, that seems to be acceptable. It's a way to enqueue the message at the back (instead of head) of the queue. Although a weird use case, maybe some apps rely on this.

That said, I believe your concern is that messages might cycle forever due to unintentional cycles.
One idea I had in the past was to introduce some dead letter specific TTL or hop limit configuration and drop the message after that number of times. This value should have some high default value.

[the-mikedavis]

Ah ok, thanks for the context, that makes sense. I could imagine an application depending on the requeueing behavior.

Detecting/stopping unintentional routing setups like this is an interesting idea! Another thing we could consider is stopping dead lettering during alarms. At-least-once dead lettering would effectively pause and at-most-once dead letters would be dropped. That seems like a big behavior change though, maybe that's not a great idea.