Skip to content

Commit 6783f4c

Browse files
psi29aCopilotCopilot
authored
Major Infra Restructure (#271)
* python 3.14 update * purge python2 * support python 3.9 to 3.14 * ruff things up; github actions; updated deps * clone worldengine-data for tests; update docker image to be used in github * clone worldengine-data for tests update * fix the windows building * skip some tests for windows * Update setup.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Initial plan * Initial plan * Replace wildcard imports with explicit imports Co-authored-by: psi29a <1122069+psi29a@users.noreply.github.com> * Remove unnecessary numpy import and data assignment in humidity.py Co-authored-by: psi29a <1122069+psi29a@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent 000a2ac commit 6783f4c

74 files changed

Lines changed: 5112 additions & 2888 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/copilot-instructions.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# WorldEngine Development Guide
2+
3+
## Project Overview
4+
5+
WorldEngine is a procedural world generator that simulates realistic terrain through plate tectonics, erosion, climate systems, and biomes. It generates both world data files (protobuf/hdf5) and visualization images (PNG). The project emphasizes scientific simulation over gaming shortcuts.
6+
7+
## Architecture
8+
9+
### Core Data Flow (Pipeline Pattern)
10+
11+
World generation follows a strict sequential pipeline in `worldengine/generation.py`:
12+
13+
1. **Plates** (`plates.py` + PyPlatec C extension) → generates elevation and plate boundaries
14+
2. **Temperature** → based on latitude and elevation
15+
3. **Precipitation** → considering rain shadow effects and ocean proximity
16+
4. **Erosion** → modifies elevation based on water flow
17+
5. **Hydrology** → watermap and river systems via recursive droplet simulation
18+
6. **Humidity** → combines precipitation, irrigation, and distance to water
19+
7. **Permeability** → soil absorption rates
20+
8. **Biome** → Holdridge life zones model classification
21+
9. **Icecap** → polar ice coverage
22+
23+
Each simulation in `worldengine/simulations/` is a class with `is_applicable()` and `execute(world, seed)` methods. Simulations modify the World object's layers in-place.
24+
25+
### World Model (`worldengine/model/world.py`)
26+
27+
The `World` class is the central data structure:
28+
- **Layers** stored as numpy arrays in `world.layers['name']` dict (elevation, ocean, precipitation, temperature, humidity, biome, etc.)
29+
- **LayerWithThresholds** for categorical data (e.g., elevation thresholds: sea/plain/hill/mountain)
30+
- **LayerWithQuantiles** for distribution-based data (e.g., humidity quantiles)
31+
- **Generation metadata**: seed, n_plates, ocean_level, step, temps/humids thresholds
32+
33+
Access patterns:
34+
- Direct: `world.layers['elevation'].data[y, x]` (numpy array indexing)
35+
- Helper: `world.elevation_at((x, y))` (tuple coordinates)
36+
- Boolean checks: `world.is_ocean((x, y))`, `world.has_biome()`
37+
38+
### Step System (`worldengine/step.py`)
39+
40+
Controls generation depth via `Step` enum:
41+
- `plates`: Only plate simulation
42+
- `precipitations`: Through precipitation/temperature
43+
- `full`: Complete pipeline including biomes
44+
45+
Check flags: `step.include_precipitations`, `step.include_erosion`, `step.include_biome`
46+
47+
### Biomes (`worldengine/biome.py`)
48+
49+
Metaclass-based registry pattern:
50+
- Each biome is a class inheriting from `Biome` (e.g., `class TropicalRainForest(Biome)`)
51+
- Auto-registration via `_BiomeMetaclass` converts CamelCase to "tropical rain forest"
52+
- Access: `Biome.by_name("boreal forest")`, `Biome.all_names()`
53+
- Stored as strings in world.layers['biome'], converted to indices for protobuf serialization
54+
55+
## Key Development Patterns
56+
57+
### NumPy-First Operations
58+
59+
Always prefer vectorized NumPy operations over Python loops:
60+
61+
```python
62+
# Good: Vectorized ocean detection
63+
ocean = numpy.zeros(elevation.shape, dtype=bool)
64+
65+
# Avoid: Cell-by-cell iteration unless simulating physical processes
66+
for y in range(height):
67+
for x in range(width): # Only when simulating droplets, adjacency, etc.
68+
```
69+
70+
### Seed Management
71+
72+
Deterministic generation requires careful seed handling:
73+
- Main seed → numpy RNG → 100 sub-seeds (one per simulation)
74+
- See `seed_dict` in `generate_world()` for allocation
75+
- Never use global random state in simulations
76+
77+
### Coordinate Systems
78+
79+
Two conventions coexist:
80+
- **Tuple style**: `(x, y)` for method parameters (e.g., `world.elevation_at((x, y))`)
81+
- **NumPy style**: `[y, x]` for array indexing (e.g., `world.layers['elevation'].data[y, x]`)
82+
83+
### Serialization
84+
85+
Two formats supported:
86+
- **Protobuf** (default): `World.proto` → `World_pb2.py` (regenerate with `protoc`)
87+
- **HDF5**: `worldengine/hdf5_serialization.py` (requires h5py, optional dependency)
88+
89+
Loading worlds: `world = World.from_pickle_file(filename)` or `load_world_from_hdf5()`
90+
91+
## Testing
92+
93+
### Running Tests
94+
95+
```bash
96+
# All tests
97+
nosetests tests -v
98+
99+
# Specific test module
100+
nosetests tests/biome_test.py -v
101+
102+
# With coverage
103+
coverage run --source worldengine --branch $(which nosetests) tests -v
104+
coverage report --omit=worldengine/tests/* --show-missing
105+
```
106+
107+
### Test Structure
108+
109+
- Tests in `tests/*_test.py` use unittest framework
110+
- `TestBase` in `tests/draw_test.py` provides common fixtures
111+
- Visual regression via `tests/blessed_images/` (compare generated images)
112+
- No mocking of NumPy/PyPlatec; tests use actual generation
113+
114+
## CLI Entry Point
115+
116+
`worldengine/cli/main.py` provides commands:
117+
- `worldengine world -s SEED -n NAME` → generate world
118+
- `worldengine ancient_map -w FILE` → render ancient-style map
119+
- `worldengine info -w FILE` → display world metadata
120+
121+
All CLI commands route through functions that call `world_gen()` or draw operations.
122+
123+
## Critical Dependencies
124+
125+
- **PyPlatec**: C extension for plate tectonics (fails gracefully if unavailable)
126+
- **NumPy**: All data is numpy arrays; version pinned for reproducibility
127+
- **protobuf 3.0.0a3**: Exact version required for compatibility
128+
- **pypng**: Image output (not PIL/Pillow)
129+
- **noise**: Perlin/Simplex noise (package `noise`, function `snoise2`)
130+
131+
## Common Pitfalls
132+
133+
1. **Don't modify world.layers['X'].data shape** - all layers must stay (height, width)
134+
2. **Ocean detection**: Use `world.is_ocean()` not `elevation <= sea_level` (accounts for threshold)
135+
3. **Verbose output**: Check `get_verbose()` before expensive debug operations
136+
4. **Border effects**: `place_oceans_at_map_borders()` intentionally lowers edges
137+
5. **Anti-aliasing**: Applied to some layers (watermap) via `anti_alias()` to smooth artifacts

‎.github/workflows/ci.yml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [ master ]
6+
pull_request:
7+
branches: [ master ]
8+
workflow_dispatch:
9+
10+
jobs:
11+
test:
12+
name: Test Python ${{ matrix.python-version }} on ${{ matrix.os }}
13+
runs-on: ${{ matrix.os }}
14+
strategy:
15+
fail-fast: false
16+
matrix:
17+
os: [ubuntu-latest, macos-latest, windows-latest]
18+
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
19+
20+
steps:
21+
- uses: actions/checkout@v4
22+
with:
23+
path: worldengine
24+
25+
- name: Checkout worldengine-data
26+
uses: actions/checkout@v4
27+
with:
28+
repository: Mindwerks/worldengine-data
29+
path: worldengine-data
30+
31+
- name: Set up Python ${{ matrix.python-version }}
32+
uses: actions/setup-python@v5
33+
with:
34+
python-version: ${{ matrix.python-version }}
35+
allow-prereleases: true
36+
37+
- name: Install uv
38+
uses: astral-sh/setup-uv@v4
39+
with:
40+
enable-cache: true
41+
42+
- name: Install dependencies
43+
working-directory: worldengine
44+
run: |
45+
uv pip install --system -e ".[hdf5,dev]"
46+
47+
- name: Run tests
48+
working-directory: worldengine
49+
run: |
50+
pytest -v tests --tb=short
51+
52+
- name: Test worldengine command
53+
if: runner.os != 'Windows'
54+
working-directory: worldengine
55+
run: |
56+
worldengine --help
57+
worldengine world -s 42 -n test_world -x 512 -y 512
58+
59+
- name: Test worldengine command (Windows - no world generation)
60+
if: runner.os == 'Windows'
61+
working-directory: worldengine
62+
run: |
63+
worldengine --help
64+
65+
lint:
66+
name: Lint and Type Check
67+
runs-on: ubuntu-latest
68+
69+
steps:
70+
- uses: actions/checkout@v4
71+
with:
72+
path: worldengine
73+
74+
- name: Set up Python
75+
uses: actions/setup-python@v5
76+
with:
77+
python-version: '3.11'
78+
79+
- name: Install uv
80+
uses: astral-sh/setup-uv@v4
81+
82+
- name: Install dependencies
83+
working-directory: worldengine
84+
run: |
85+
uv pip install --system -e ".[dev]"
86+
87+
- name: Run pre-commit
88+
working-directory: worldengine
89+
run: |
90+
pre-commit run --all-files
91+
92+
- name: Run ruff check
93+
working-directory: worldengine
94+
run: |
95+
ruff check worldengine/
96+
97+
- name: Run mypy
98+
working-directory: worldengine
99+
run: |
100+
mypy worldengine/ --ignore-missing-imports
101+
continue-on-error: true
102+
103+
build-wheels:
104+
name: Build wheels on ${{ matrix.os }}
105+
runs-on: ${{ matrix.os }}
106+
strategy:
107+
matrix:
108+
os: [ubuntu-latest, macos-latest, windows-latest]
109+
110+
steps:
111+
- uses: actions/checkout@v4
112+
with:
113+
path: worldengine
114+
115+
- name: Set up Python
116+
uses: actions/setup-python@v5
117+
with:
118+
python-version: '3.11'
119+
120+
- name: Install build tools
121+
run: |
122+
python -m pip install --upgrade pip build
123+
124+
- name: Build sdist (Ubuntu only)
125+
if: matrix.os == 'ubuntu-latest'
126+
working-directory: worldengine
127+
run: |
128+
python -m build --sdist
129+
130+
- name: Build wheel
131+
working-directory: worldengine
132+
run: |
133+
python -m build --wheel
134+
135+
- name: Upload artifacts
136+
uses: actions/upload-artifact@v4
137+
with:
138+
name: wheels-${{ matrix.os }}
139+
path: worldengine/dist/*
140+
141+
docker:
142+
name: Test Docker build
143+
runs-on: ubuntu-latest
144+
145+
steps:
146+
- uses: actions/checkout@v4
147+
with:
148+
path: worldengine
149+
150+
- name: Set up Docker Buildx
151+
uses: docker/setup-buildx-action@v3
152+
153+
- name: Build Docker image
154+
uses: docker/build-push-action@v6
155+
with:
156+
context: ./worldengine
157+
push: false
158+
tags: worldengine:test
159+
cache-from: type=gha
160+
cache-to: type=gha,mode=max
161+
load: true
162+
163+
- name: Test Docker image
164+
run: |
165+
docker run worldengine:test worldengine --help
166+
docker run worldengine:test worldengine world -s 123 -n docker_test -x 256 -y 256

0 commit comments

Comments
 (0)