Skip to content
This repository was archived by the owner on Jan 8, 2026. It is now read-only.

Commit 07436a1

Browse files
myuimyui
committed
Initial commit
0 parents  commit 07436a1

57 files changed

Lines changed: 14251 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Run CI
2+
3+
on:
4+
push:
5+
branches: [ "main" ]
6+
pull_request:
7+
branches:
8+
- "*"
9+
10+
jobs:
11+
pytest:
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- name: Install uv
18+
uses: astral-sh/setup-uv@v4
19+
with:
20+
enable-cache: true
21+
cache-dependency-glob: "pyproject.toml"
22+
23+
- name: Print the installed version
24+
run: echo "Installed uv version is ${{ steps.setup-uv.outputs.uv-version }}"
25+
26+
- name: "Set up Python"
27+
uses: actions/setup-python@v5
28+
with:
29+
python-version-file: "pyproject.toml"
30+
31+
- name: Install the project
32+
run: uv sync --all-extras --dev
33+
34+
- name: Run tests
35+
run: uv run pytest tests/

‎.github/workflows/publish.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# This workflow will upload a Python Package to PyPI when a version tag is pushed
2+
3+
name: "Publish Python Package to PyPI"
4+
5+
on:
6+
release:
7+
types: ["published"]
8+
push:
9+
tags:
10+
- 'v[0-9]+.[0-9]+*' # Push events to matching v1.0, v20.15.10 tags
11+
12+
jobs:
13+
run:
14+
name: "Build and publish release"
15+
runs-on: ubuntu-latest
16+
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Install uv
21+
uses: astral-sh/setup-uv@v4
22+
with:
23+
enable-cache: true
24+
cache-dependency-glob: "pyproject.toml"
25+
26+
- name: Print the installed version
27+
run: echo "Installed uv version is ${{ steps.setup-uv.outputs.uv-version }}"
28+
29+
- name: "Set up Python"
30+
uses: actions/setup-python@v5
31+
with:
32+
python-version-file: "pyproject.toml"
33+
34+
- name: Build
35+
run: uv build
36+
37+
- name: Publish
38+
run: uv publish --check-url https://pypi.org/simple/ --token ${{ secrets.PYPI_TOKEN }}

‎.gitignore‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
develop-eggs/
9+
dist/
10+
downloads/
11+
eggs/
12+
.eggs/
13+
lib/
14+
lib64/
15+
parts/
16+
sdist/
17+
var/
18+
wheels/
19+
*.egg-info/
20+
.installed.cfg
21+
*.egg
22+
MANIFEST
23+
.pytest_cache/
24+
.coverage
25+
htmlcov/
26+
.tox/
27+
.nox/
28+
.hypothesis/
29+
.mypy_cache/
30+
.ruff_cache/
31+
uv.lock
32+
33+
# Virtual Environment
34+
venv/
35+
env/
36+
ENV/
37+
.env
38+
.venv
39+
env.bak/
40+
venv.bak/
41+
42+
# Docker
43+
.docker/
44+
45+
# Logs
46+
logs/
47+
*.log
48+
49+
# IDE specific files
50+
.idea/
51+
.vscode/
52+
*.swp
53+
*.swo
54+
*~
55+
.project
56+
.classpath
57+
.settings/
58+
*.sublime-workspace
59+
60+
# OS specific
61+
.DS_Store
62+
.DS_Store?
63+
._*
64+
.Spotlight-V100
65+
.Trashes
66+
ehthumbs.db
67+
Thumbs.db
68+
69+
# Miscellaneous
70+
.secrets
71+
.env.local
72+
.env.development
73+
.env.production
74+
credentials/
75+
.cache/

‎LICENSE‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
BSD 3-Clause License
2+
3+
Copyright (c) 2024, Makoto YUI
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are met:
7+
8+
1. Redistributions of source code must retain the above copyright notice, this
9+
list of conditions and the following disclaimer.
10+
11+
2. Redistributions in binary form must reproduce the above copyright notice,
12+
this list of conditions and the following disclaimer in the documentation
13+
and/or other materials provided with the distribution.
14+
15+
3. Neither the name of the copyright holder nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Project.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# FairQueue Project
2+
3+
## Project Overview
4+
**fairque** is a production-ready fair queue implementation using Redis/Valkey with work stealing and priority scheduling.
5+
6+
## Key Features
7+
- Fair Scheduling: Round-robin user selection with work stealing
8+
- Priority Queues: Type-safe priority system (1-6) with critical/normal separation
9+
- Atomic Operations: Lua scripts ensure consistency and performance
10+
- Configuration-Based: No Redis state for user management, fully configurable
11+
- Pipeline Optimization: Batch operations for high throughput
12+
- Comprehensive Monitoring: Built-in statistics and alerting
13+
- Production Ready: Error handling, graceful shutdown, health checks
14+
- Cron-Based Scheduling: Task scheduling with croniter for recurring tasks
15+
## Architecture
16+
- **Redis/Valkey as persistent storage** (not in-memory cache)
17+
- **Lua scripts for server-side atomic operations** with integrated statistics
18+
- **Configuration-based user management** (no all_users Redis key)
19+
- **Dual implementation**: Synchronous and Asynchronous versions
20+
- **Type-safe Priority system** using IntEnum
21+
- **Work stealing strategy** for load balancing
22+
23+
## Technology Stack
24+
- Python 3.10+
25+
- Redis 7.2.5+ / Valkey 7.2.6+ / Amazon MemoryDB
26+
- Package Manager: uv
27+
- Type Annotations: Required (Full Typing)
28+
- Language: English only
29+
30+
## Project Structure
31+
```
32+
fairque/
33+
├── fairque/ # Main package
34+
│ ├── __init__.py
35+
│ ├── core/ # Core models and types
36+
│ │ ├── __init__.py
37+
│ │ ├── models.py # Task, Priority, DLQEntry
38+
│ │ ├── config.py # Configuration classes
39+
│ │ └── exceptions.py # Exception classes
40+
│ ├── queue/ # Queue implementation
41+
│ │ ├── __init__.py
42+
│ │ ├── fairqueue.py # Main FairQueue class
43+
│ │ └── async_fairqueue.py # Async implementation
44+
│ ├── worker/ # Worker implementation
45+
│ │ ├── __init__.py
46+
│ │ ├── worker.py # Sync worker
47+
│ │ └── async_worker.py # Async worker
48+
│ ├── scripts/ # Lua scripts
49+
│ ├── scheduler/ # Task scheduling
50+
│ │ ├── __init__.py
51+
│ │ ├── models.py # ScheduledTask model
52+
│ │ └── scheduler.py # TaskScheduler implementation
53+
│ ├── scripts/ # Lua scripts
54+
│ │ ├── common.lua # Shared functions
55+
│ │ ├── push.lua # Push operation
56+
│ │ ├── pop.lua # Pop operation
57+
│ │ └── stats.lua # Statistics operations
58+
│ └── utils/ # Utilities
59+
│ ├── __init__.py
60+
│ ├── stats.py # Statistics formatting
61+
│ └── monitoring.py # Monitoring and alerting
62+
├── tests/ # Test suite
63+
├── examples/ # Usage examples
64+
├── docs/ # Documentation
65+
├── pyproject.toml # Project configuration
66+
└── README.md # Project documentation
67+
```
68+
69+
## Implementation Status
70+
- [x] Project setup and structure
71+
- [x] Core models implementation (Priority, Task, DLQEntry)
72+
- [x] Configuration system (RedisConfig, WorkerConfig, QueueConfig, FairQueueConfig)
73+
- [x] Exception handling system
74+
- [x] Lua scripts implementation (common.lua, push.lua, pop.lua, stats.lua)
75+
- [x] FairQueue core implementation (FairQueue class, LuaScriptManager)
76+
- [x] Worker implementation (TaskHandler, Worker classes with work stealing)
77+
- [x] Basic testing suite (unit tests and integration tests)
78+
- [x] **Async implementation (AsyncTaskQueue, AsyncWorker, AsyncTaskHandler)**
79+
- [x] Task Scheduler implementation (Cron-based scheduling with distributed locking)
80+
- [x] **Performance testing suite** (throughput, worker, Redis operations, async comparison)
81+
- [ ] Complete documentation
82+
83+
## Current Phase
84+
**Phase 7: Performance Testing** - ✅ **PERFORMANCE TESTING COMPLETED**
85+
86+
**Status**: **🎉 COMPREHENSIVE PERFORMANCE TESTING SUITE COMPLETE 🎉**
87+
88+
### Performance Testing Details
89+
- **Queue Throughput Tests**: Single/batch operations, concurrent pushes, work stealing
90+
- **Worker Performance Tests**: Single/concurrent workers, processing efficiency
91+
- **Redis Operations Tests**: Lua script performance, pipeline optimization, memory efficiency
92+
- **Async vs Sync Comparison**: Throughput, concurrency, resource usage comparisons
93+
- **Benchmarking Utilities**: Reusable performance metrics and reporting
94+
- **Run Script**: `python tests/performance/run_benchmarks.py` for easy execution
95+
96+
### Async Implementation Details
97+
- **AsyncTaskQueue**: Full async version of TaskQueue using `redis.asyncio`
98+
- **AsyncWorker**: Async worker with `asyncio.Task` based concurrency
99+
- **AsyncTaskHandler**: Abstract base class for async task processing
100+
- **AsyncLuaScriptManager**: Async version of Lua script management
101+
- **Concurrent Operations**: `push_batch_concurrent()` for high-throughput scenarios
102+
- **Async Context Managers**: Full support for `async with` statements
103+
- **Health Checks**: Async health monitoring and statistics
104+
- **Graceful Shutdown**: Proper async task cleanup and resource management
105+
106+
### Dual Architecture Benefits
107+
- **Synchronous Version**: Thread-based concurrency, familiar patterns
108+
- **Asynchronous Version**: Event-loop based, higher throughput potential
109+
- **Shared Core**: Same Lua scripts, models, and configuration system
110+
- **API Compatibility**: Similar interfaces for easy migration
111+
- **Performance Options**: Choose based on use case and environment
112+
113+
## Current Phase
114+
**Phase 5: Testing and Documentation** - ✅ **CORE TESTING COMPLETED** - Comprehensive test suite covering all major functionality.
115+
116+
**Status**: **🎉 PRODUCTION-READY CORE SYSTEM COMPLETE 🎉**
117+
118+
## Next Steps
119+
1. Implement core models (Priority, Task, Configuration)
120+
2. Create Lua scripts with common functions
121+
3. Implement basic FairQueue functionality
122+
4. Add comprehensive error handling
123+
124+
## Development Guidelines
125+
- All code and documentation in English
126+
- Full type annotations required
127+
- Comprehensive docstrings
128+
- Follow PEP 8 style guidelines
129+
- Use uv for package management
130+
- Incremental implementation with checkpoints

0 commit comments

Comments
 (0)