|
8 | 8 |
|
9 | 9 | Job queues in python with asyncio, redis and msgpack.
|
10 | 10 |
|
11 |
| -rq meets asyncio. |
12 |
| - |
13 |
| -arq is not production ready yet, **use with caution.** |
14 |
| - |
15 |
| -arq is a tool for distributing tasks by first encoding a description of the job and adding it |
16 |
| -to a redis list, then pop the job description from the list and executing it somewhere else. |
17 |
| -The "somewhere else" can be another process or another computer. arq is inspired by |
18 |
| -[rq](https://github.com/nvie/rq) but takes a significantly different approach. |
19 |
| - |
20 |
| -It is: |
21 |
| - * **non-blocking** built using python's [asyncio](https://docs.python.org/3/library/asyncio.html) allowing non-blocking |
22 |
| - job enqueuing and job execution. |
23 |
| - * **pre-forked** In other works the worker starts two processes and uses the |
24 |
| - subprocess to execute all jobs, there's no overhead in forking a process for each job. |
25 |
| - * **fast** Asyncio, pre-forking and use of msgpack for job encoding make arq around 7x faster |
26 |
| - (see [benchmarks](/performance_benchmarks)) than rq for small jobs with no io, |
27 |
| - with io that might increase to around 40x faster. TODO |
28 |
| - * **elegant** arq uses a novel approach to variable scope with the `@concurrent` decorator being applied to bound |
29 |
| - methods of "Actor" classes which hold the connection pool. This works well with |
30 |
| - [aiohttp](http://aiohttp.readthedocs.io/en/stable/), avoids extended head scratching over how variables |
31 |
| - like connections are defined (is this attached to the request? or thread local? or truly global? |
32 |
| - where am I, what does global mean?) and allows for easier testing. See below. |
33 |
| - * **small** and easy to reason with - currently arq is only about 500 lines, that won't change significantly. |
34 |
| - |
35 |
| -## Install |
36 |
| - |
37 |
| -**Python >=3.5** and **redis** are required. After than: |
38 |
| - |
39 |
| - pip install arq |
40 |
| - |
41 |
| -Should install everything you need. |
42 |
| - |
43 |
| -## Usage |
44 |
| - |
45 |
| -Usage is best described with an example, `demo.py`: |
46 |
| - |
47 |
| -```python |
48 |
| -import asyncio |
49 |
| -from aiohttp import ClientSession |
50 |
| -from arq import Actor, BaseWorker, concurrent |
51 |
| - |
52 |
| - |
53 |
| -class Downloader(Actor): |
54 |
| - def __init__(self, **kwargs): |
55 |
| - super().__init__(**kwargs) |
56 |
| - self.session = ClientSession(loop=self.loop) |
57 |
| - |
58 |
| - @concurrent |
59 |
| - async def download_content(self, url): |
60 |
| - async with self.session.get(url) as response: |
61 |
| - assert response.status == 200 |
62 |
| - content = await response.read() |
63 |
| - print('{}: {:.80}...'.format(url, content.decode())) |
64 |
| - return len(content) |
65 |
| - |
66 |
| - async def close(self): |
67 |
| - await super().close() |
68 |
| - self.session.close() |
69 |
| - |
70 |
| - |
71 |
| -class Worker(BaseWorker): |
72 |
| - shadows = [Downloader] |
73 |
| - |
74 |
| - |
75 |
| -async def download_lots(loop): |
76 |
| - d = Downloader(loop=loop) |
77 |
| - for url in ('https://facebook.com', 'https://microsoft.com', 'https://github.com'): |
78 |
| - await d.download_content(url) |
79 |
| - await d.close() |
80 |
| - |
81 |
| -if __name__ == '__main__': |
82 |
| - loop = asyncio.get_event_loop() |
83 |
| - loop.run_until_complete(download_lots(loop)) |
84 |
| -``` |
85 |
| - |
86 |
| -You can then enqueue the jobs with just `python demo.py`, and run |
87 |
| -the worker to do the jobs with `arq demo.py`. |
88 |
| - |
89 |
| -`arq --help` for more help on how to run the worker. |
90 |
| - |
91 |
| -Still to be documented but working fine: |
92 |
| -* multiple queues |
93 |
| -* multiple actors |
94 |
| -* worker `max_concurrency` |
95 |
| -* worker job timeout |
96 |
| -* advanced worker logging |
97 |
| -* `.testing` py.test plugins. |
98 |
| - |
99 |
| -## Actors, Shadows and global variables |
100 |
| - |
101 |
| -TODO |
102 |
| - |
103 |
| -## TODO |
104 |
| - |
105 |
| -* jobs results |
106 |
| -* job uniqueness |
| 11 | +See [documentation](https://samuelcolvin.github.io/arq/index.html) for more details. |
0 commit comments