Add Registry partition_by option for duplicate registries

[studzien]

Summary

This PR adds key-based partitioning for duplicate Registry entries, allowing optimization for workloads with many keys and few entries per key (e.g., many topics with few subscribers each).

Background

This feature addresses performance concerns first noticed in phoenixframework/phoenix_pubsub#198, where Registry's PID-based partitioning wasn't optimal for workloads with many topics and relatively few subscribers per topic.

Solution

Enhanced Keys API

• :unique: Traditional unique registry behavior
• :duplicate: Traditional duplicate registry with PID-based partitioning (default)
• {:duplicate, :pid}: Explicit PID-based partitioning for duplicate registries
• {:duplicate, :key}: New key-based partitioning for duplicate registries

Performance Optimization

• Key-based lookups with {:duplicate, :key} now only need to check a single partition
• Reduces lookup complexity from O(partitions) to O(1) for key-based operations

Changes

lib/elixir/lib/registry.ex:

• Updated keys typespec to include {:duplicate, :key} and {:duplicate, :pid}
• Enhanced all internal functions to handle key-based partitioning strategy
• Updated documentation with usage examples and performance guidance
• Modified lookup/2, values/3, match/3, count/1, count_match/3, select/2 and other functions to support both partitioning strategies

lib/elixir/test/elixir/registry_test.exs:

• Added comprehensive tests for all partitioning strategies
• Extended cleanup and functionality tests to cover key-based partitioning
• Added specific test cases for {:duplicate, :key} behavior

API Changes

New Keys Format

# Traditional duplicate registry (PID partitioning)
Registry.start_link(keys: :duplicate, name: MyRegistry)

# Explicit PID partitioning  
Registry.start_link(keys: {:duplicate, :pid}, name: MyRegistry)

# New key-based partitioning
Registry.start_link(keys: {:duplicate, :key}, name: MyRegistry)

Usage Guidelines

• Use :duplicate or {:duplicate, :pid}: When you have few keys with many entries (e.g., one topic with many subscribers)
• Use {:duplicate, :key}: When you have many keys with few entries each (e.g., many topics with few subscribers)

Backward Compatibility

This change is fully backward compatible:

• Default behavior remains unchanged (:duplicate still uses PID partitioning)
• Existing code continues to work without modification
• Only adds new functionality for those who opt into {:duplicate, :key} partitioning
• Key-based partitioning is only supported for duplicate registries (validated at startup)

Related

• Separate Registry pool size configuration phoenixframework/phoenix_pubsub#198 - Original performance issue identification

🤖 Generated with Claude Code

[josevalim]

Instead of adding a new element, I wonder if we should introduce a new internal type. So we have :unique, :duplicate and :duplicate_by_key.

[josevalim]

We could even have :duplicate_by_pid and :duplicate_by_key. Or, alternatively, have the type be {:duplicate, :pid | :key}. But basically there is no need to store more information that only applies to duplicates and not unique.

[studzien]

I think this makes sense, considering that the new key doesn't make sense for a unique registry. Just wondering if :duplicate should also have an explicit alias, something like :duplicate_by_pid

[josevalim]

Yeah, I think we commented at the same time. Let's go with either:duplicate_by_pid and :duplicate_by_key or {:duplicate, partition_by}. Your call. The second one will allow us to partially match when we don't care about the partition.

[studzien]

Sounds good. I'll also make sure that :duplicate keeps existing behaviour of :duplicate_by_pid (or {:duplicate, :pid}})

[studzien]

@josevalim I updated it to use {:duplicate, :key} or {:duplicate, :pid}.
I was not certain if the partition supervisor strategy should change, but I think it should stay the same (:one_for_one) since we partition both ets tables by the same key (:key or :pid).

Will update the PR description shortly.

[josevalim]

This claude needs to be reverted but we need to be careful with #{kind} below, that will no longer work.

[josevalim]

Reverting:

Suggested change

	arg =
	{kind, registry, i, partitions, key_partition, pid_partition, listeners, compressed}
	arg = {kind, registry, i, partitions, key_partition, pid_partition, listeners, compressed}

[josevalim]

I have dropped a few comments.

I also think we need to better parameterize the test:

1. Break the current RegistryTest into Registry.UniqueTest and Registry.DuplicateTest
2. Make sure the DuplicateTest is parameterized by partition count AND partition by

The new tests you added can be part of the new DuplicateTest.