[Bug] initialize-transaction-coordinator-metadata hangs indefinitely on ZK ConnectionLoss due to non-daemon executor thread leak in ZKMetadataStore constructor

[warren288]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

Environment where reproduced:

• Apache Pulsar 3.0.11
• JDK: OpenJDK 21.0.5+11 (Temurin)
• OS: Linux
• zookeeper.client.secure=true with ClientCnxnSocketNetty (TLS-enabled ZK connection)

This bug exists in Apache Pulsar 3.0.11 (and all 3.x/4.x versions, as the ZKMetadataStore constructor and AbstractMetadataStore executor pattern have not changed).

Issue Description

When PulsarTransactionCoordinatorMetadataSetup#main() fails to connect to ZooKeeper within the session timeout (default 30s), the process hangs indefinitely after the exception is printed instead of exiting with a non-zero status code.

Root cause

The bug is a two-layer resource leak in ZKMetadataStore construction:

Layer 1: Executor not cleaned up on construction failure

ZKMetadataStore constructor (ZKMetadataStore.java:87-117) calls super(metadataStoreConfig) first, which creates a ScheduledThreadPoolExecutor wit a non-daemon DefaultThreadFactory in AbstractMetadataStore.java:99-101:

this.executor = new ScheduledThreadPoolExecutor(1,
new DefaultThreadFactory(
StringUtils.isNotBlank(metadataStoreName) ? metadataStoreName : getClass().getSimpleName()));

Netty's DefaultThreadFactory(String) single-argument constructor sets daemon = false, so the worker thread configuration-metadata-store-1-1 is a non-daemon thread.

When PulsarZooKeeperClient.Builder.build() subsequently fails at waitForConnection() (PulsarZooKeeperClient.java:259), the exception propagates t the catch (Throwable t) block in ZKMetadataStore.java:114-116:

} catch (Throwable t) {
throw new MetadataStoreException(t); // executor not cleaned up
}

The catch block does not call super.close() or executor.shutdowor and its non-daemon worker thread are leaked.

Layer 2: try-with-resources does not call close() on failed ini

PulsarTransactionCoordinatorMetadataSetup.main() (PulsarTransacadataSetup.java:98-101) uses try-with-resources:

try (MetadataStoreExtended configStore = PulsarClusterMetadataSadataStore(...)) {
// ...
}

Per JLS §14.20.3, if the resource initialization expression thrclose() is not called. Since initConfigMetadataStore() → MetadataStoreExtended.create() → new ZKMetadataStore(...) throws, the configStore variable is never assigned, and AbstractMetadataStore.close() (which calls executor.shutdownNow()) is never invoked.

Result: The non-daemon thread configuration-metadata-store-1-1 ED_WAITING state. After main() throws and returns, the JVM creates the DestroyJavaVM thread, but cannot exit because a nonstill alive. The process hangs indefinitely.

Triggering condition

The issue is intermittent. It triggers whenever the ZooKeeper client fails to receive a SyncConnected event within the session timeout (ZooKeeperWatcherBase.waitForConnection(), default 30s). This can happen due to:

• Transient ZK server unavailability (leader election, GC pause reached)
• Network jitter or latency spikes
• TLS handshake timeout (when zookeeper.client.secure=true is set)
• System resource contention during JVM startup (e.g., javaagen

When the ZK connection succeeds within the timeout, the command completes normally and the process exits. When it fails, the process hangs.

Error messages

org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =   ConnectionLoss
      at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:114)
      at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:48)
      at org.apache.pulsar.PulsarClusterMetadataSetup.initConfigMetadataStore(PulsarClusterMetadataSetup.java:508)
      at org.apache.pulsar.PulsarTransactionCoordinatorMetadataSetup.main(PulsarTransactionCoordinatorMetadataSetup.java:98)
  Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
      at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:170)
      at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$Builder.build(PulsarZooKeeperClient.java:259)
      at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:101)
      ... 3 more

  After the exception is printed, the process never exits. kill -3 thread dumps show:

  "configuration-metadata-store-1-1" #43 prio=5 os_prio=0 cpu=1546.81ms elapsed=852.87s
     java.lang.Thread.State: TIMED_WAITING (parking)
          at jdk.internal.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(DelayedWorkQueue.java:1188)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1002)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

  "DestroyJavaVM" #50 prio=5 os_prio=0 cpu=0.26ms elapsed=852.87s
     java.lang.Thread.State: RUNNABLE

  Three consecutive kill -3 dumps (at elapsed 853s, 935s, 1113s) consistently show configuration-metadata-store-1-1 as the only non-daemon user thread keeping the JVM alive.

Reproducing the issue

1. Point to a ZK server that is temporarily unreachable or slow to respond

(e.g., stop ZK, add firewall rule, or saturate ZK connections)

pulsar initialize-transaction-coordinator-metadata
-cs <zk_address>:
-c <cluster_name>

2. Observe: ConnectionLoss exception is printed, but the process does not exit

3. Verify with thread dump:

kill -3

Look for:

- "DestroyJavaVM" thread in RUNNABLE state (main() has returned)

- "configuration-metadata-store-1-1" as the only non-daemon user thread

in TIMED_WAITING (parking) on ScheduledThreadPoolExecutor$DelayedWorkQueue.take

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

I checked the code and this looks like a real issue. The same failure path is still present in the current checkout.

One detail worth adding: creating the executor alone doesn’t start the thread. AbstractBatchedMetadataStore schedules the periodic batch flush before the ZooKeeper connection is established. I checked that this also happens in 3.0.11.

If the connection fails, the constructor throws without cleaning up the superclass resources. Try-with-resources is working as expected here: it cannot close a resource whose constructor failed. The constructor needs to handle that cleanup.

The executor layout has changed in the current code, so the leaked thread would now be named …-batch-flusher…, but the underlying issue remains.

This is based on source inspection; I haven’t reproduced the hang locally.