Memory leaks

[emlozin]

Hi,
I experienced some serious memory leaks on Raspbian and Ubuntu 18.04 while publishing opencv images with current aiopubsub version. For me the fix was to change L.283 in init.py to:
callback = lambda key, message: queue.put((key, message))
which does not make any sense to me but somehow memory leaks are now gone.
Just thought it might need some investigation ;)

[vitawasalreadytaken]

Hi @emlozin,
Thanks for the report! Is it possible you mean line 274? (https://github.com/qntln/aiopubsub/blob/master/aiopubsub/__init__.py#L274). There's only a docstring on line 283 in the current __init__.py.

Would you be able to share a piece of your code so we can take a look at why memory leaks arise?

Thank you very much!

[emlozin]

Oh... Yes, sorry :) Typing too fast :O
Unfortunately, the code is not my property so I am not able to share. But I did some extensive testing and it was all good once I changed that line. I had multiple coroutines of asyncio running and communicating over aiopubsub (sending image frames and fps rate). I tested that on my Raspberry Pi and Lenovo and the results were same. Just Raspberry Pi froze sooner :)

[pbabics]

Hi @emlozin,

using callback = lambda key, message: queue.put((key, message)) may have broken your application, because asyncio.Queue.put is a coroutine (see https://docs.python.org/3/library/asyncio-queue.html#asyncio.Queue.put) whereas asyncio.Queue.put_nowait is not.

Method which is using this callback is https://github.com/qntln/aiopubsub/blob/master/aiopubsub/__init__.py#L116, if you look closer it is not using await callback(message), so no messages are put into queues.

[emlozin]

Hey @pbabics,
Yes, I know it is not a good fix. The funny thing is that it did not break anything functionality-wise. I still get the messages. But still works better for now than having huge memory leaks (where in 30 secs Raspberry Pi whole memory was gone)...

[edit] I did not create an issue to suggest a fix. I just wanted to mention that I faced this issue and where the problem occurred. Thought it might help others ;)

[emlozin]

Just to show the difference and give a hint of what's happening. Tracemalloc before my ugly fix:
[ Top 10 ]
/usr/lib/python3.6/multiprocessing/queues.py:113: size=45.7 MiB, count=212, average=221 KiB
file_1:82: size=45.7 MiB, count=105, average=446 KiB
file_2:44: size=45.7 MiB, count=104, average=450 KiB
file_1:30: size=2400 KiB, count=6, average=400 KiB
result_5
result_6
result_7
/usr/lib/python3.6/tracemalloc.py:65: size=147 KiB, count=2088, average=72 B
/usr/lib/python3.6/threading.py:846: size=99.3 KiB, count=88, average=1156 B
/usr/lib/python3.6/tracemalloc.py:113: size=91.5 KiB, count=1065, average=88 B

[ Top 10 differences ]
/usr/lib/python3.6/multiprocessing/queues.py:113: size=45.7 MiB (+900 KiB), count=212 (+4), average=221 KiB
file_1:82: size=45.7 MiB (+900 KiB), count=105 (+2), average=446 KiB
file_2:44 size=45.7 MiB (+900 KiB), count=104 (+2), average=450 KiB
/home/user/.local/lib/python3.6/site-packages/aiopubsub/init.py:20: size=18.2 KiB (+264 B), count=215 (+3), average=87 B
/usr/lib/python3.6/selectors.py:348: size=696 B (+168 B), count=7 (+1), average=99 B
result 6
/home/user/.local/lib/python3.6/site-packages/aiopubsub/init.py:21: size=9008 B (+120 B), count=218 (+3), average=41 B
file_3: size=4272 B (+80 B), count=54 (+1), average=79 B
/usr/lib/python3.6/tracemalloc.py:65: size=147 KiB (-72 B), count=2088 (-1), average=72 B
result 10