HADOOP-19624. [ABFS] Fixing Thread leak in AbfsClientThrottlingAnalyzer #7852

mattkduran · 2025-08-03T18:02:17Z

Description of PR

The ABFS driver's auto-throttling feature (fs.azure.enable.autothrottling=true) creates Timer threads in AbfsClientThrottlingAnalyzer that are never properly cleaned up, leading to a memory leak that eventually causes OutOfMemoryError in long-running applications like Hive Metastore.

Impact:

Thread count grows indefinitely (observed >100,000 timer threads)
Affects any long-running service that creates multiple ABFS filesystem instances

Root Cause:

AbfsClientThrottlingAnalyzer creates Timer objects in its constructor but provides no mechanism to cancel them. When AbfsClient instances are closed, the associated timer threads continue running indefinitely.

Solution

Implement proper resource cleanup by making the throttling components implement Closeable and ensuring timers are cancelled when ABFS clients are closed.

Changes Made

AbfsClientThrottlingAnalyzer.java

Added: implements Closeable
Added: close() method that calls timer.cancel() and timer.purge()
Purpose: Ensures timer threads are properly terminated when analyzer is no longer needed

AbfsThrottlingIntercept.java (Interface)

Added: extends Closeable
Added: close() method signature
Purpose: Establishes cleanup contract for all throttling intercept implementations

AbfsClientThrottlingIntercept.java

Added: close() method that closes both readThrottler and writeThrottler
Purpose: Coordinates cleanup of both read and write throttling analyzers

AbfsNoOpThrottlingIntercept.java

Added: No-op close() method
Purpose: Satisfies interface contract for no-op implementation

AbfsClient.java

Added: IOUtils.cleanupWithLogger(LOG, intercept) in existing close() method
Purpose: Integrates throttling cleanup into existing client resource management

https://github.com/mattkduran/ABFSleaktest
https://www.mail-archive.com/[email protected]/msg43483.html

How was this patch tested?

Standalone Validation Tool

This fix was validated using a standalone reproduction and testing tool that directly exercises the ABFS auto-throttling components outside of a full Hadoop deployment.
Repository: ABFSLeakTest

Testing Scope

Problem reproduction confirmed - demonstrates the timer thread leak
Fix validation confirmed - proves close() method resolves the leak
Resource cleanup verified - shows proper timer cancellation
Limited integration testing - standalone tool, not full Hadoop test suite

Test Results

Leak Reproduction Evidence

# Without fix: Timer threads accumulate over filesystem creation cycles
Cycle    Total Threads    ABFS Timer Threads    Status
1        50->52          0->2                   LEAK DETECTED
50       150->152        98->100               LEAK GROWING  
200      250->252        398->400              LEAK CONFIRMED

Final Analysis: 400 leaked timer threads named "abfs-timer-client-throttling-analyzer-*"
Memory Impact: ~90MB additional heap usage

# Direct analyzer testing:
🔴 Without close(): +3 timer threads (LEAKED)
✅ With close():    +0 timer threads (NO LEAK)

Test Environment

Java Version: OpenJDK 11.0.x
Hadoop Version: 3.3.6/3.4.1 (both affected)
Test Duration: 200 filesystem creation/destruction cycles
Thread Monitoring: JMX ThreadMXBean

For code changes:

[ X ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

This method closes the throttling analyzer and releases any associated resources.

anujmodi2021

Thanks for the patch and thorough testing of the issue @mattkduran
I have a few suggestions and comments. Please do take a look at them.

Also we need at least one test (unit or integration) to be inlcuded in this patch. Can you plan for one? Idea is to have coverage of code impacted here.

Also, I see a few PR checks failing, If you click on the link of each -1 commented by hadoop-yetus, you should be able to see the issue reported and fix them.

Once all of this is done, we can wait for a few more reviews and get this checked in.

Thanks again for all the efforts.

anujmodi2021 · 2025-08-26T09:21:22Z

...doop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsThrottlingIntercept.java

 @InterfaceAudience.Private
 @InterfaceStability.Unstable
-public interface AbfsThrottlingIntercept {
+public interface AbfsThrottlingIntercept extends Closable {


Shouldn't this be extends Closeable?

Fixed -- thank you for catching the typo!

anujmodi2021 · 2025-08-26T09:23:18Z

...-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsNoOpThrottlingIntercept.java

  }
+
+/**
+ * No-op implementation of close method.


Nit: javadoc to include @ throws

...zure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientThrottlingIntercept.java

...azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientThrottlingAnalyzer.java

anujmodi2021 · 2025-10-22T04:35:35Z

Spotbug issue might be due to this: https://issues.apache.org/jira/browse/HADOOP-19731
We can wait for the fix and merge with trunk once it gets resolved.

matt-duran-starburst · 2026-01-05T15:56:02Z

Spotbug issue might be due to this: https://issues.apache.org/jira/browse/HADOOP-19731 We can wait for the fix and merge with trunk once it gets resolved.

@anujmodi2021 I saw that this got merged, is there a way that I can manually trigger the CI job to confirm if this fixed the spotbugs issues?

anujmodi2021 · 2026-01-06T05:42:10Z

Spotbug issue might be due to this: https://issues.apache.org/jira/browse/HADOOP-19731 We can wait for the fix and merge with trunk once it gets resolved.

@anujmodi2021 I saw that this got merged, is there a way that I can manually trigger the CI job to confirm if this fixed the spotbugs issues?

A dummy commit will help.
Some minor javadoc change can be pushed.

hadoop-yetus · 2026-01-06T20:02:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 16s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 50s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 47s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	1m 36s	/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html	hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings.
+1 💚	shadedclient	34m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 44s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 26s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	1m 26s		the patch passed
+1 💚	shadedclient	32m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 25s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		133m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/5/artifact/out/Dockerfile
GITHUB PR	#7852
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 9cf785906c25 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd88518`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/5/testReport/
Max. process+thread count	586 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/5/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

matt-duran-starburst · 2026-01-06T20:33:12Z

Spotbug failed here -- not something introduced in this patch as far as I'm aware:

hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java

Line 200 in cd88518

acquireRetryCount++;


Bug type VO_VOLATILE_INCREMENT (click for details)In class org.apache.hadoop.fs.azurebfs.services.AbfsLease$1In method org.apache.hadoop.fs.azurebfs.services.AbfsLease$1.onFailure(Throwable)Field org.apache.hadoop.fs.azurebfs.services.AbfsLease.acquireRetryCountAt AbfsLease.java:[line 200]
--


[Bug type VO_VOLATILE_INCREMENT (click for details)](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7852/5/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html#VO_VOLATILE_INCREMENT)
In class org.apache.hadoop.fs.azurebfs.services.AbfsLease$1
In method org.apache.hadoop.fs.azurebfs.services.AbfsLease$1.onFailure(Throwable)
Field org.apache.hadoop.fs.azurebfs.services.AbfsLease.acquireRetryCount
At AbfsLease.java:[line 200]

anmolanmol1234 · 2026-01-08T06:59:04Z

...e/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClientThrottlingAnalyzer.java

+    }
+    return count;
+  }
 }


Extra line needed at end of file

anujmodi2021 · 2026-01-08T11:13:15Z

Thanks @matt-duran-starburst for the patch and taking care of everything.

anujmodi2021

+1

mattkduran added 8 commits July 16, 2025 13:02

Import closable

0492703

Class implements closable

792a86f

Add in close method

05ee721

This method closes the throttling analyzer and releases any associated resources.

Import in IOException

f894117

Add in implementation to call close method

5cf275c

Add in cleanup method

0a0f4e1

Adding in close method

25c6d89

Added no-op method

fee9861

github-actions bot added trunk TOOLS ABFS labels Aug 3, 2025

This comment was marked as outdated.

Sign in to view

anujmodi2021 requested changes Aug 26, 2025

View reviewed changes

matt-duran-starburst and others added 6 commits October 18, 2025 14:08

Fix typo

566f596

Add @throws in the javadoc

862e002

Add @throws to javadoc

7f1e3e8

Remove closeable from interface

5a8b78e

Add tests for AbfsClientThrottlingAnalyzer cleanup and idempotency

37baae3

Merge branch 'apache:trunk' into trunk

90abc25

This comment was marked as outdated.

Sign in to view

matt-duran-starburst added 2 commits October 18, 2025 18:13

Removing whitespaces

425eccb

Merge branch 'trunk' of https://github.com/mattkduran/hadoop into trunk

90b097b

This comment was marked as outdated.

Sign in to view

Dummy commit to force CI job

cd88518

anmolanmol1234 reviewed Jan 8, 2026

View reviewed changes

anmolanmol1234 approved these changes Jan 8, 2026

View reviewed changes

anujmodi2021 approved these changes Jan 8, 2026

View reviewed changes

anujmodi2021 changed the title ~~HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer~~ HADOOP-19624. Thread leak in ABFS AbfsClientThrottlingAnalyzer Jan 8, 2026

anujmodi2021 changed the title ~~HADOOP-19624. Thread leak in ABFS AbfsClientThrottlingAnalyzer~~ HADOOP-19624. [ABFS] Fixing Thread leak in AbfsClientThrottlingAnalyzer Jan 8, 2026

anujmodi2021 merged commit 4190e98 into apache:trunk Jan 8, 2026
2 of 5 checks passed

anmolanmol1234 mentioned this pull request Jan 8, 2026

Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria #8137

Open

HADOOP-19624. [ABFS] Fixing Thread leak in AbfsClientThrottlingAnalyzer #7852

HADOOP-19624. [ABFS] Fixing Thread leak in AbfsClientThrottlingAnalyzer #7852

Uh oh!

Conversation

mattkduran commented Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Impact:

Root Cause:

Solution

Changes Made

How was this patch tested?

Standalone Validation Tool

Testing Scope

Test Results

Test Environment

For code changes:

Uh oh!

This comment was marked as outdated.

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

matt-duran-starburst Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

matt-duran-starburst Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

anujmodi2021 commented Oct 22, 2025

Uh oh!

matt-duran-starburst commented Jan 5, 2026

Uh oh!

anujmodi2021 commented Jan 6, 2026

Uh oh!

hadoop-yetus commented Jan 6, 2026

Uh oh!

matt-duran-starburst commented Jan 6, 2026

Uh oh!

anmolanmol1234 Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 commented Jan 8, 2026

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mattkduran commented Aug 3, 2025 •

edited

Loading