Skip to content

Conversation

@jamesoff
Copy link

@jamesoff jamesoff commented Jun 8, 2020

The main functional change here is to use the "ratelimiter" library, which makes use of time.sleep() to slow down execution to hit the requested rate limit on the loop. (The original code would busy-loop to slow itself down, causing high CPU usage.)

My testing of the two scripts before and after shows a considerable decrease in the "user" time:

# before
% time ./the_publisher.py
Publishing messages for 456976 symbols
2020-Jun-05 18:53:58: Pushed 21 objects for a total of 21 objects
2020-Jun-05 18:54:08: Pushed 21 objects for a total of 42 objects
2020-Jun-05 18:54:18: Pushed 21 objects for a total of 63 objects
^CPushed a total of 65 objects; exiting...
./the_publisher.py  12.02s user 0.20s system 39% cpu 31.214 total

# after
% time ./the_publisher.py
Publishing messages for 456976 symbols
2020-Jun-05 18:54:37: Pushed 20 objects for a total of 20 objects
2020-Jun-05 18:54:49: Pushed 20 objects for a total of 40 objects
2020-Jun-05 18:55:02: Pushed 20 objects for a total of 60 objects
^CPushed a total of 60 objects; exiting...
./the_publisher.py  0.94s user 0.13s system 2% cpu 39.150 total
# before
% time ./the_subscriber.py
2020-Jun-05 19:13:54: Retrieved 22 objects for a total of 22 objects
2020-Jun-05 19:14:04: Retrieved 22 objects for a total of 44 objects
2020-Jun-05 19:14:14: Retrieved 22 objects for a total of 66 objects
^CRetrieved a total of 66 objects; exiting...
./the_subscriber.py  17.65s user 0.16s system 56% cpu 31.428 total

# after
% time ./the_subscriber.py
2020-Jun-05 19:16:34: Retrieved 20 objects for a total of 20 objects
2020-Jun-05 19:16:42: Retrieved 20 objects for a total of 40 objects
2020-Jun-05 19:16:49: Retrieved 20 objects for a total of 60 objects
^CRetrieved a total of 60 objects; exiting...
./the_subscriber.py  0.63s user 0.12s system 3% cpu 22.098 total

The rate of execution is slightly different between the two versions but should be close enough as I don't believe that's critical to the workshop?

I have also tidied the Python code up to meet PEP8 a bit better, and added boto3 to the Pipfile to ensure it's included in the virtualenv created by the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant