Aether

A blazingly fast distributed task executor for Python scripts. Aether provides a robust cluster architecture to run your Python workloads across multiple machines with ease.

✨ Features

• ⚡ High Performance: Asynchronous Rust core for maximum throughput.
• 🐍 Python Focused: Execute Python scripts with proper isolation and resource management.
• 🔄 Priority Queuing: Support for high, medium, and low priority task scheduling.
• 🎯 Smart Matching: Automatic worker assignment based on GPU availability and CPU architecture.
• 🌐 HTTP API: RESTful interface for interacting with the broker.
• 🔗 JRPC Protocol: JSON-RPC communication between brokers and workers.
• 🛡️ Auto-reconnection logic: Don't worry if your broker or workers crash.
• 🔐 Authentication & Authorization: JWT authentication and permissions-based authorization.
• 🖥️ CLI: Intuitive command-line interface for all operations.

📋 Table of Contents

• Quick Start
• Architecture
• Installation
• Usage
• API
• Contributing
• License

🚀 Quick Start

Get Aether running in under 1 minute!

1. Start the Broker

aether broker start --api-port 8080 --jrpc-port 9090

2. Launch a Worker

In a new terminal:

# GPU and Architecture capabilities are set to `false` and `x86_64` by default.
aether worker start --worker-id worker1 --broker-ip 127.0.0.1 --broker-port 9090

3. Submit Your First Task

Create hello.py:

import time
print("Hello from Aether! 🚀")
time.sleep(1)
print("Task completed successfully!")

Aether authenticates all of their routes with JWT except the health and login endpoints.

Create a profile with the following command:

aether auth login --profile test --broker-ip 127.0.0.1 --broker-api-port 8080 --username admin --password admin

This sets the test profile as the active profile.

Submit the task:

aether task submit --task-file hello.py --name "Hello World"

4. Check Status

aether task check --task-id <task-uuid>

Output:

{
  "id": "59d5ca42-93e7-4b11-8927-e25c519db283",
  "name": "Hello World",
  "code_b64": "aW1wb3J0IHRpbWUKcHJpbnQoIkhlbGxvIGZyb20gQWV0aGVyISDwn5qAIikKdGltZS5zbGVlcCgxKQpwcmludCgiVGFzayBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5ISIpCg==",
  "result": {
    "exit_code": 0,
    "stderr": "",
    "stdout": "Hello from Aether! 🚀\nTask completed successfully!\n"
  },
  "status": "completed",
  "capabilities": {
    "gpu": false,
    "arch": "x86_64"
  }
}

🎉 Congratulations! You've just executed your first distributed Python task.

🏗️ Architecture

┌─────────────┐    HTTP     ┌─────────────┐
│   Client    │ ──────────► │   Broker    │
└─────────────┘             └─────────────┘
                                 │
                                 │ JRPC
                                 ▼
                         ┌─────────────┐
                         │   Worker    │
                         │  (Python)   │
                         └─────────────┘

• Broker: Central coordinator managing task queues and worker registry
• Workers: Execute Python scripts with resource matching
• Clients: Submit tasks via HTTP API and monitor progress

📦 Installation

From Source

Requires Rust 1.88+.

git clone https://github.com/NoOPeEKS/aether.git
cd aether
cargo build --release

From Crates.io (when available)

cargo install aether-cli

💻 Usage

Broker Commands

# Start broker
aether broker start --api-port 8080 --jrpc-port 9090 (optional --redis-ip 127.0.0.1 --redis-port 6379)

Worker Commands

# Start worker with GPU support
aether worker start --worker-id gpu-worker --broker-ip 10.0.0.1 --broker-port 9090 --gpu --arch aarch64

Task Management

Note

All commands make use of the active profile configuration by default. To bypass the active profile without switching you can inline the arguments --broker-ip, --broker-api-port and --token.

# Submit task
aether task submit --task-file script.py --name "Data Analysis"

# Check status
aether task check --task-id 550e8400-e29b-41d4-a716-446655440000

# List all tasks 
aether task list

# Stop a task
aether task stop --task-id 550e8400-e29b-41d4-a716-446655440000

Authentication

# Login and set as active profile (JWT token will be printed to stdout)
aether auth login --profile test --broker-ip 127.0.0.1 --broker-api-port 8080 --username admin --password admin

# Switch active profile
aether auth switch --profile prod

# Logout and delete profile
aether auth logout --profile test

🔌 API

Endpoints

Public

• GET /api/v1/health - Health check
• POST /api/v1/auth/login - Log in as a user and get a JWT

Authenticated

• POST /api/v1/tasks - Submit a new task
• GET /api/v1/tasks - List all tasks
• GET /api/v1/tasks/{id} - Get task status and result
• POST /api/v1/tasks/{id}/cancel - Cancel a given task execution
• POST /api/v1/users - Create a new user. Requires admin privileges.

🤝 Contributing

• 🐛 Found a bug? Open an issue
• 💡 Have a feature request? Start a discussion
• 🔧 Want to contribute code? Check out our development docs

📄 License

Licensed under the Apache 2.0 License. See LICENSE for details.