Skip to content

Commit 4ff84e2

Browse files
committed
uprev, update readme
1 parent c52266a commit 4ff84e2

File tree

4 files changed

+2
-100
lines changed

4 files changed

+2
-100
lines changed

README.md

+1-96
Original file line numberDiff line numberDiff line change
@@ -8,99 +8,4 @@ arq
88

99
Job queues in python with asyncio, redis and msgpack.
1010

11-
rq meets asyncio.
12-
13-
arq is not production ready yet, **use with caution.**
14-
15-
arq is a tool for distributing tasks by first encoding a description of the job and adding it
16-
to a redis list, then pop the job description from the list and executing it somewhere else.
17-
The "somewhere else" can be another process or another computer. arq is inspired by
18-
[rq](https://github.com/nvie/rq) but takes a significantly different approach.
19-
20-
It is:
21-
* **non-blocking** built using python's [asyncio](https://docs.python.org/3/library/asyncio.html) allowing non-blocking
22-
job enqueuing and job execution.
23-
* **pre-forked** In other works the worker starts two processes and uses the
24-
subprocess to execute all jobs, there's no overhead in forking a process for each job.
25-
* **fast** Asyncio, pre-forking and use of msgpack for job encoding make arq around 7x faster
26-
(see [benchmarks](/performance_benchmarks)) than rq for small jobs with no io,
27-
with io that might increase to around 40x faster. TODO
28-
* **elegant** arq uses a novel approach to variable scope with the `@concurrent` decorator being applied to bound
29-
methods of "Actor" classes which hold the connection pool. This works well with
30-
[aiohttp](http://aiohttp.readthedocs.io/en/stable/), avoids extended head scratching over how variables
31-
like connections are defined (is this attached to the request? or thread local? or truly global?
32-
where am I, what does global mean?) and allows for easier testing. See below.
33-
* **small** and easy to reason with - currently arq is only about 500 lines, that won't change significantly.
34-
35-
## Install
36-
37-
**Python >=3.5** and **redis** are required. After than:
38-
39-
pip install arq
40-
41-
Should install everything you need.
42-
43-
## Usage
44-
45-
Usage is best described with an example, `demo.py`:
46-
47-
```python
48-
import asyncio
49-
from aiohttp import ClientSession
50-
from arq import Actor, BaseWorker, concurrent
51-
52-
53-
class Downloader(Actor):
54-
def __init__(self, **kwargs):
55-
super().__init__(**kwargs)
56-
self.session = ClientSession(loop=self.loop)
57-
58-
@concurrent
59-
async def download_content(self, url):
60-
async with self.session.get(url) as response:
61-
assert response.status == 200
62-
content = await response.read()
63-
print('{}: {:.80}...'.format(url, content.decode()))
64-
return len(content)
65-
66-
async def close(self):
67-
await super().close()
68-
self.session.close()
69-
70-
71-
class Worker(BaseWorker):
72-
shadows = [Downloader]
73-
74-
75-
async def download_lots(loop):
76-
d = Downloader(loop=loop)
77-
for url in ('https://facebook.com', 'https://microsoft.com', 'https://github.com'):
78-
await d.download_content(url)
79-
await d.close()
80-
81-
if __name__ == '__main__':
82-
loop = asyncio.get_event_loop()
83-
loop.run_until_complete(download_lots(loop))
84-
```
85-
86-
You can then enqueue the jobs with just `python demo.py`, and run
87-
the worker to do the jobs with `arq demo.py`.
88-
89-
`arq --help` for more help on how to run the worker.
90-
91-
Still to be documented but working fine:
92-
* multiple queues
93-
* multiple actors
94-
* worker `max_concurrency`
95-
* worker job timeout
96-
* advanced worker logging
97-
* `.testing` py.test plugins.
98-
99-
## Actors, Shadows and global variables
100-
101-
TODO
102-
103-
## TODO
104-
105-
* jobs results
106-
* job uniqueness
11+
See [documentation](https://samuelcolvin.github.io/arq/index.html) for more details.

arq/version.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22

33
__all__ = ['VERSION']
44

5-
VERSION = StrictVersion('0.0.5')
5+
VERSION = StrictVersion('0.0.6')

docs/conf.py

-1
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,6 @@
141141
'codecov_button': True,
142142
'page_width': '1200px',
143143
'github_banner': True,
144-
145144
'github_type': 'star',
146145
}
147146

setup.py

-2
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
77
Job queues in python with asyncio, redis and msgpack.
88
9-
rq meets asyncio.
10-
119
See `github <https://github.com/samuelcolvin/arq>`__ for more details.
1210
1311
.. |Build Status| image:: https://travis-ci.org/samuelcolvin/arq.svg?branch=master

0 commit comments

Comments
 (0)