Skip to content
This repository was archived by the owner on Jan 8, 2026. It is now read-only.

Commit e3c4114

Browse files
committed
docs: Expand key differentiators section with detailed explanations and examples for FairQueue features
1 parent 15ce27a commit e3c4114

1 file changed

Lines changed: 122 additions & 122 deletions

File tree

README.md

Lines changed: 122 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,128 @@ Production-ready fair queue implementation using Redis with work stealing and pr
1818
-**Task Scheduling**: Cron-based task scheduling with distributed coordination
1919
- 📊 **Task States**: Comprehensive state management with 7 states (queued, started, deferred, finished, failed, canceled, scheduled)
2020

21+
## Key Differentiators vs. Existing Task Queues (RQ, Celery, etc.)
22+
23+
FairQueue goes beyond simple task queuing with production-ready features for complex multi-tenant applications:
24+
25+
### 🏢 **1. Multi-Tenancy with Fair Resource Allocation**
26+
**Problem**: Traditional queues like RQ use simple FIFO, allowing high-volume users to monopolize resources.
27+
**Solution**: FairQueue implements true fair scheduling with round-robin user selection.
28+
29+
```python
30+
# RQ: Tasks processed in order submitted - user1 can starve user2
31+
# user1 submits 1000 tasks, user2 submits 1 task → user2 waits behind all 1000
32+
33+
# FairQueue: Guaranteed fair access across users
34+
config = FairQueueConfig.create_default(
35+
assigned_users=["user:1", "user:2", "user:3"] # Equal processing opportunity
36+
)
37+
# Round-robin: user1 → user2 → user3 → user1, regardless of task volume
38+
```
39+
40+
### **2. Intelligent Resource Utilization with Work Stealing**
41+
**Problem**: Traditional workers are idle when their assigned queues are empty, even if other queues have work.
42+
**Solution**: FairQueue workers can steal tasks from other users when idle, maximizing throughput.
43+
44+
```python
45+
# Traditional: Worker assigned to user:1 sits idle if user:1 has no tasks
46+
# FairQueue: Worker processes assigned users first, then steals from targets
47+
config = FairQueueConfig.create_default(
48+
worker_id="worker-001",
49+
assigned_users=["user:1", "user:2"], # Primary responsibility
50+
steal_targets=["user:3", "user:4"] # Secondary work when idle
51+
)
52+
# Result: 100% worker utilization + maintains fairness guarantees
53+
```
54+
55+
### 🔗 **3. Built-in XCom for Cross-Task Communication**
56+
**Problem**: RQ has no native way for tasks to share data - requires external storage or manual Redis operations.
57+
**Solution**: FairQueue includes XCom system for seamless task-to-task data passing.
58+
59+
```python
60+
# RQ: Manual data passing via external storage
61+
def task1():
62+
result = process_data()
63+
redis.set("shared_data", json.dumps(result)) # Manual
64+
65+
def task2():
66+
data = json.loads(redis.get("shared_data")) # Manual
67+
68+
# FairQueue: Built-in XCom with automatic data management
69+
@xcom_task(push_key="processed_data", auto_xcom=True)
70+
def task1():
71+
return process_data() # Automatically stored
72+
73+
@task(depends_on=["task1"], enable_xcom=True)
74+
def task2():
75+
data = self.xcom_pull("processed_data") # Automatic retrieval
76+
```
77+
78+
### 🔄 **4. DAG Workflows Beyond Simple Task Queues**
79+
**Problem**: RQ is a simple job queue - complex workflows require external orchestration tools.
80+
**Solution**: FairQueue includes dependencies, pipelines, and DAG support built-in.
81+
82+
```python
83+
# RQ: No dependency management - manual orchestration required
84+
# Must manually track completion and trigger dependent tasks
85+
86+
# FairQueue: Full DAG support with pipeline operators
87+
@task(task_id="extract")
88+
def extract_data(): return {"records": 1000}
89+
90+
@task(task_id="transform", depends_on=["extract"])
91+
def transform_data(): return {"processed": 2000}
92+
93+
@task(task_id="load", depends_on=["transform"])
94+
def load_data(): return {"loaded": True}
95+
96+
# Airflow-style pipeline composition
97+
pipeline = extract_data() >> transform_data() >> load_data()
98+
parallel_pipeline = extract_data() >> (transform_data() | validate_data()) >> load_data()
99+
100+
queue.enqueue(pipeline) # Automatic dependency resolution
101+
```
102+
103+
### 🎯 **Multi-Tenant SaaS Applications**
104+
Perfect for applications where fair resource allocation is critical:
105+
- **E-commerce platforms**: Fair order processing across merchants
106+
- **Data processing services**: Equitable resource sharing across customers
107+
- **API gateways**: Preventing resource monopolization by high-volume clients
108+
- **Batch processing systems**: Balanced workload distribution
109+
110+
### 🚀 **Horizontal Scaling with Intelligence**
111+
Add workers dynamically with automatic work distribution:
112+
```python
113+
# Scale by adding workers with smart assignment
114+
workers = [
115+
{"id": "worker-001", "assigned_users": ["user:1", "user:3"], "steal_targets": ["user:2", "user:4"]},
116+
{"id": "worker-002", "assigned_users": ["user:2", "user:4"], "steal_targets": ["user:1", "user:3"]},
117+
{"id": "worker-003", "assigned_users": ["user:5", "user:6"], "steal_targets": ["user:1", "user:2"]}
118+
]
119+
# Result: Perfect load balancing + no resource waste + maintained fairness
120+
```
121+
122+
## Why "Fair"?
123+
124+
The name **FairQueue** reflects the core principle of **fairness** in task distribution and processing:
125+
126+
### 🎯 **Fair Task Distribution**
127+
- **Round-robin user selection**: Each user gets equal opportunity for their tasks to be processed
128+
- **No user starvation**: High-volume users cannot monopolize worker resources
129+
- **Balanced workload**: Tasks are distributed evenly across workers through work stealing
130+
131+
### ⚖️ **Fair Resource Allocation**
132+
- **Priority-aware fairness**: Critical tasks get immediate attention while maintaining fairness among normal priorities
133+
- **Worker equity**: All workers have equal opportunity to process tasks from their assigned users
134+
- **Dynamic load balancing**: Work stealing ensures optimal resource utilization without unfair advantage
135+
136+
### 🔄 **Fair Processing Order**
137+
- **Within user fairness**: Tasks from the same user are processed in priority order
138+
- **Cross-user fairness**: No single user can dominate the queue regardless of task volume
139+
- **Temporal fairness**: Tasks are processed in a predictable, fair manner based on submission time and priority
140+
141+
This fairness model makes FairQueue ideal for multi-tenant systems, SaaS platforms, and any application where equitable resource sharing is crucial for user experience and system stability.
142+
21143
## Quick Start
22144

23145
### Installation
@@ -294,128 +416,6 @@ config = FairQueueConfig.create_default(
294416
)
295417
```
296418

297-
## Key Differentiators vs. Existing Task Queues (RQ, Celery, etc.)
298-
299-
FairQueue goes beyond simple task queuing with production-ready features for complex multi-tenant applications:
300-
301-
### 🏢 **1. Multi-Tenancy with Fair Resource Allocation**
302-
**Problem**: Traditional queues like RQ use simple FIFO, allowing high-volume users to monopolize resources.
303-
**Solution**: FairQueue implements true fair scheduling with round-robin user selection.
304-
305-
```python
306-
# RQ: Tasks processed in order submitted - user1 can starve user2
307-
# user1 submits 1000 tasks, user2 submits 1 task → user2 waits behind all 1000
308-
309-
# FairQueue: Guaranteed fair access across users
310-
config = FairQueueConfig.create_default(
311-
assigned_users=["user:1", "user:2", "user:3"] # Equal processing opportunity
312-
)
313-
# Round-robin: user1 → user2 → user3 → user1, regardless of task volume
314-
```
315-
316-
### **2. Intelligent Resource Utilization with Work Stealing**
317-
**Problem**: Traditional workers are idle when their assigned queues are empty, even if other queues have work.
318-
**Solution**: FairQueue workers can steal tasks from other users when idle, maximizing throughput.
319-
320-
```python
321-
# Traditional: Worker assigned to user:1 sits idle if user:1 has no tasks
322-
# FairQueue: Worker processes assigned users first, then steals from targets
323-
config = FairQueueConfig.create_default(
324-
worker_id="worker-001",
325-
assigned_users=["user:1", "user:2"], # Primary responsibility
326-
steal_targets=["user:3", "user:4"] # Secondary work when idle
327-
)
328-
# Result: 100% worker utilization + maintains fairness guarantees
329-
```
330-
331-
### 🔗 **3. Built-in XCom for Cross-Task Communication**
332-
**Problem**: RQ has no native way for tasks to share data - requires external storage or manual Redis operations.
333-
**Solution**: FairQueue includes XCom system for seamless task-to-task data passing.
334-
335-
```python
336-
# RQ: Manual data passing via external storage
337-
def task1():
338-
result = process_data()
339-
redis.set("shared_data", json.dumps(result)) # Manual
340-
341-
def task2():
342-
data = json.loads(redis.get("shared_data")) # Manual
343-
344-
# FairQueue: Built-in XCom with automatic data management
345-
@xcom_task(push_key="processed_data", auto_xcom=True)
346-
def task1():
347-
return process_data() # Automatically stored
348-
349-
@task(depends_on=["task1"], enable_xcom=True)
350-
def task2():
351-
data = self.xcom_pull("processed_data") # Automatic retrieval
352-
```
353-
354-
### 🔄 **4. DAG Workflows Beyond Simple Task Queues**
355-
**Problem**: RQ is a simple job queue - complex workflows require external orchestration tools.
356-
**Solution**: FairQueue includes dependencies, pipelines, and DAG support built-in.
357-
358-
```python
359-
# RQ: No dependency management - manual orchestration required
360-
# Must manually track completion and trigger dependent tasks
361-
362-
# FairQueue: Full DAG support with pipeline operators
363-
@task(task_id="extract")
364-
def extract_data(): return {"records": 1000}
365-
366-
@task(task_id="transform", depends_on=["extract"])
367-
def transform_data(): return {"processed": 2000}
368-
369-
@task(task_id="load", depends_on=["transform"])
370-
def load_data(): return {"loaded": True}
371-
372-
# Airflow-style pipeline composition
373-
pipeline = extract_data() >> transform_data() >> load_data()
374-
parallel_pipeline = extract_data() >> (transform_data() | validate_data()) >> load_data()
375-
376-
queue.enqueue(pipeline) # Automatic dependency resolution
377-
```
378-
379-
### 🎯 **Multi-Tenant SaaS Applications**
380-
Perfect for applications where fair resource allocation is critical:
381-
- **E-commerce platforms**: Fair order processing across merchants
382-
- **Data processing services**: Equitable resource sharing across customers
383-
- **API gateways**: Preventing resource monopolization by high-volume clients
384-
- **Batch processing systems**: Balanced workload distribution
385-
386-
### 🚀 **Horizontal Scaling with Intelligence**
387-
Add workers dynamically with automatic work distribution:
388-
```python
389-
# Scale by adding workers with smart assignment
390-
workers = [
391-
{"id": "worker-001", "assigned_users": ["user:1", "user:3"], "steal_targets": ["user:2", "user:4"]},
392-
{"id": "worker-002", "assigned_users": ["user:2", "user:4"], "steal_targets": ["user:1", "user:3"]},
393-
{"id": "worker-003", "assigned_users": ["user:5", "user:6"], "steal_targets": ["user:1", "user:2"]}
394-
]
395-
# Result: Perfect load balancing + no resource waste + maintained fairness
396-
```
397-
398-
## Why "Fair"?
399-
400-
The name **FairQueue** reflects the core principle of **fairness** in task distribution and processing:
401-
402-
### 🎯 **Fair Task Distribution**
403-
- **Round-robin user selection**: Each user gets equal opportunity for their tasks to be processed
404-
- **No user starvation**: High-volume users cannot monopolize worker resources
405-
- **Balanced workload**: Tasks are distributed evenly across workers through work stealing
406-
407-
### ⚖️ **Fair Resource Allocation**
408-
- **Priority-aware fairness**: Critical tasks get immediate attention while maintaining fairness among normal priorities
409-
- **Worker equity**: All workers have equal opportunity to process tasks from their assigned users
410-
- **Dynamic load balancing**: Work stealing ensures optimal resource utilization without unfair advantage
411-
412-
### 🔄 **Fair Processing Order**
413-
- **Within user fairness**: Tasks from the same user are processed in priority order
414-
- **Cross-user fairness**: No single user can dominate the queue regardless of task volume
415-
- **Temporal fairness**: Tasks are processed in a predictable, fair manner based on submission time and priority
416-
417-
This fairness model makes FairQueue ideal for multi-tenant systems, SaaS platforms, and any application where equitable resource sharing is crucial for user experience and system stability.
418-
419419
## Architecture
420420

421421
### Priority System

0 commit comments

Comments
 (0)