Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread leaks (at least in tests) #23349

Open
findepi opened this issue Sep 10, 2024 · 4 comments
Open

Thread leaks (at least in tests) #23349

findepi opened this issue Sep 10, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@findepi
Copy link
Member

findepi commented Sep 10, 2024 

./mvnw -pl :trino-main test

To me this looks like an application leaking memory with leaking threads being likely the reason

image

eventually tests slow down progress probably due to retries

2024-09-10T09:53:19.188-0600	WARN	http-client-memoryManager-110913	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56590/v1/memory: Server refused connection: http://127.0.0.1:56590/v1/memory
2024-09-10T09:53:20.191-0600	WARN	http-client-memoryManager-110906	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56590/v1/memory: Server refused connection: http://127.0.0.1:56590/v1/memory
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.24 s -- in io.trino.sql.query.TestDistinctAggregationsNoMarkDistinct
2024-09-10T09:53:21.110-0600	WARN	http-client-memoryManager-107141	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:22.127-0600	WARN	http-client-memoryManager-107139	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:23.128-0600	WARN	http-client-memoryManager-107141	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:23.434-0600	WARN	http-client-node-manager-106280	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56422/v1/info/state: Server refused connection: http://127.0.0.1:56422/v1/info/state
2024-09-10T09:53:23.662-0600	WARN	http-client-node-manager-107123	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56450/v1/info/state: Server refused connection: http://127.0.0.1:56450/v1/info/state
2024-09-10T09:53:24.032-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:24.136-0600	WARN	http-client-memoryManager-107139	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.31 s -- in io.trino.operator.scalar.TestJoniRegexpFunctions
2024-09-10T09:53:25.034-0600	WARN	http-client-memoryManager-106301	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:26.038-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:27.045-0600	WARN	http-client-memoryManager-106301	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:28.049-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.73 s -- in io.trino.operator.scalar.TestVarbinaryFunctions
2024-09-10T09:53:35.822-0600	WARN	http-client-memoryManager-100157	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
2024-09-10T09:53:36.826-0600	WARN	http-client-memoryManager-100156	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
2024-09-10T09:53:37.155-0600	WARN	http-client-node-manager-100127	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56218/v1/info/state: Server refused connection: http://127.0.0.1:56218/v1/info/state
2024-09-10T09:53:37.834-0600	WARN	http-client-memoryManager-100157	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.6 s -- in io.trino.operator.scalar.TestStringFunctions

2024-09-10T09:53:59.571-0600	WARN	http-client-node-manager-111469	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56610/v1/info/state: Server refused connection: http://127.0.0.1:56610/v1/info/state
2024-09-10T09:53:59.939-0600	WARN	http-client-memoryManager-111482	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
2024-09-10T09:54:00.945-0600	WARN	http-client-memoryManager-111489	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
2024-09-10T09:54:01.952-0600	WARN	http-client-memoryManager-111482	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.28 s -- in io.trino.operator.scalar.timestamptz.TestOperators
^[[F2024-09-10T09:54:06.603-0600	WARN	http-client-memoryManager-104553	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:07.510-0600	WARN	http-client-node-manager-104528	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56377/v1/info/state: Server refused connection: http://127.0.0.1:56377/v1/info/state
2024-09-10T09:54:07.611-0600	WARN	http-client-memoryManager-104564	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:08.613-0600	WARN	http-client-memoryManager-104553	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:09.616-0600	WARN	http-client-memoryManager-104564	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 123.3 s -- in io.trino.operator.scalar.timestamptz.TestExtract

but continue nonetheless (awesome resilience!), although very slowly
@findepi findepi added the bug Something isn't working label Sep 10, 2024
@martint
Copy link
Member

martint commented Sep 10, 2024

Can you try disabling the thread-per-driver scheduler for the tests and rerun the analysis?

@findepi
Copy link
Member Author

findepi commented Sep 12, 2024

I tried that, and I did get similar results.

@wendigo
Copy link
Contributor

wendigo commented Sep 12, 2024 

➜  trino git:(master) ✗ grep http-client heap.dump | wc -l
     558
➜  trino git:(master) ✗ grep query-execution heap.dump | wc -l
    2661
➜  trino git:(master) ✗ grep query-execution heap.dump | wc -l
    2661

Most of the threads are in the query-execution- group

@findepi
Copy link
Member Author

findepi commented Sep 12, 2024

that's what i was seeing as well, yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

No branches or pull requests
3 participants
@martint @wendigo @findepi