feat: Sprint 3 — observability, concurrency tests, and production docs

- Add Prometheus Summary metrics to PrometheusMetricsRecorder with
  configurable quantiles (p50/p95/p99) via enable_summaries flag
- Export DEFAULT_QUANTILES constant from pharox public API
- Add threading stress tests for concurrent acquire/release scenarios
- Add geo utility tests for haversine_distance_km
- Add three ADRs: callbacks over events, sync IStorage, SQLAlchemy Core
- Add OpenTelemetry integration guide and production deployment checklist
- Set pytest coverage threshold to 85% (postgres adapter excluded)

Author: fzaca

docs/adr/001-callbacks-over-events.md

@@ -0,0 +1,40 @@
+# ADR 001 — Callbacks over an event system
+
+**Status:** Accepted
+**Date:** 2025-11-02
+
+## Context
+
+`ProxyManager` needs to let callers plug in observability (logging, metrics) and
+custom business logic (notifications, auditing) without hard-coding those concerns
+into the core. Two common approaches were considered:
+
+1. **Event bus / pub-sub** — a central dispatcher; handlers subscribe by event type.
+2. **Callback lists** — callers register plain callables; the manager invokes them
+   at defined lifecycle points (after acquire, after release).
+
+## Decision
+
+Use typed callback lists (`List[Callable[[Payload], None]]`) registered via
+`register_acquire_callback` and `register_release_callback`.
+
+## Rationale
+
+- **Zero extra dependencies.** A callback list needs no third-party event library.
+- **Explicit typing.** The payload types (`AcquireEventPayload`,
+  `ReleaseEventPayload`) are Pydantic models — mypy can verify handler signatures.
+- **Simple mental model.** Callbacks fire in registration order; there is no
+  routing, priority, or async dispatch complexity.
+- **Sufficient for the use cases on record** — metrics recording and structured
+  logging only need the two lifecycle hooks.
+
+An event bus would add value if many unrelated subsystems need to observe the
+same events. For the current scope (one manager, a handful of optional callbacks),
+the overhead is not justified.
+
+## Consequences
+
+- Adding a new lifecycle hook requires a new callback list and a new `register_*`
+  method on `ProxyManager` — a small but explicit change.
+- Callers that need async callbacks must wrap the synchronous call in
+  `asyncio.to_thread` themselves; the manager remains sync-only.

docs/adr/002-sync-storage-interface.md

@@ -0,0 +1,44 @@
+# ADR 002 — Synchronous `IStorage` interface
+
+**Status:** Accepted
+**Date:** 2025-11-02
+
+## Context
+
+`IStorage` defines the persistence contract used by `ProxyManager` and
+`HealthCheckOrchestrator`. Python has two concurrency models relevant here:
+
+- **Sync / threading** — blocking I/O, standard with WSGI servers and
+  `concurrent.futures`.
+- **Async / asyncio** — non-blocking I/O, standard with ASGI servers and
+  `asyncio`-based workers.
+
+The choice of sync vs. async propagates through the entire call chain.
+
+## Decision
+
+`IStorage` is a synchronous (blocking) interface. All abstract methods return
+values directly; none are coroutines.
+
+## Rationale
+
+- **Broader compatibility.** Sync adapters work inside both threaded and async
+  runtimes (via `asyncio.to_thread`). An async-only interface would exclude
+  threaded hosts without a wrapper.
+- **Simpler adapter authoring.** SQLAlchemy Core with a synchronous engine is the
+  most common production setup. Sync adapters avoid the complexity of
+  `async_sessionmaker` and connection pool lifecycle management.
+- **ProxyManager usage pattern.** Lease acquisition is typically short (sub-ms for
+  in-memory, low-ms for Postgres). Blocking a thread for that duration is
+  acceptable in most real-world scenarios.
+- **Async helpers already exist.** `async_helpers.py` wraps `ProxyManager`
+  operations in `asyncio.to_thread`, giving async callers a clean interface
+  without requiring an async `IStorage`.
+
+## Consequences
+
+- High-throughput async services that want non-blocking storage access should
+  use the async wrappers or implement `IAsyncStorage` separately (tracked in
+  the backlog as a Sprint 4 item).
+- SQLAlchemy async engine users will need a custom adapter; the provided
+  `PostgresStorage` reference implementation uses a synchronous engine.

docs/adr/003-sqlalchemy-core-not-orm.md

@@ -0,0 +1,41 @@
+# ADR 003 — SQLAlchemy Core over ORM for the Postgres adapter
+
+**Status:** Accepted
+**Date:** 2025-11-08
+
+## Context
+
+The reference `PostgresStorage` adapter needs to issue SQL queries for proxy
+lookup, lease creation, health-check updates, and pool statistics. SQLAlchemy
+offers two layers:
+
+- **ORM** — declares Python classes mapped to tables; rich relationship loading;
+  session-based identity map.
+- **Core** — table/column objects and composable SQL expression language; thin
+  wrapper over the DBAPI.
+
+## Decision
+
+Use **SQLAlchemy Core** with explicit `Table` definitions (`tables.py`) and
+parameterised `select`/`insert`/`update` statements.
+
+## Rationale
+
+- **No identity map overhead.** Proxy and lease objects are value objects — they
+  are re-created from database rows on every read. An ORM session tracking
+  identity is unnecessary and would add memory and lock-management overhead.
+- **Explicit SQL.** Core statements are close to the generated SQL, making
+  performance characteristics easy to reason about. `SKIP LOCKED`, window
+  functions, and advisory locks are straightforward to express.
+- **Pydantic owns validation.** Domain models (`Proxy`, `Lease`, etc.) are
+  Pydantic models. Having a parallel set of ORM-mapped Python classes would
+  create two representations of the same domain with synchronisation risk.
+- **Alembic compatibility.** `MetaData` + `Table` definitions integrate cleanly
+  with Alembic migrations regardless of whether ORM is used.
+
+## Consequences
+
+- Developers writing custom adapters for other databases follow the same
+  Core pattern; there is no ORM base class to inherit from.
+- Relationship traversal (e.g., joining pool → proxies → leases) must be
+  written as explicit joins rather than relying on ORM `relationship()`.

docs/how-to/opentelemetry.md

@@ -0,0 +1,95 @@
+---
+title: OpenTelemetry Integration
+description: Emit traces and spans from Pharox via the OpenTelemetry SDK.
+---
+# OpenTelemetry Integration
+
+Pharox ships with Prometheus metrics out of the box, but you can layer
+OpenTelemetry traces on top by hooking into the acquire/release callbacks.
+
+## Prerequisites
+
+```bash
+pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
+```
+
+## Basic Setup
+
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+
+provider = TracerProvider()
+provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
+trace.set_tracer_provider(provider)
+
+tracer = trace.get_tracer("pharox")
+```
+
+## Acquire Callback
+
+Register a callback that opens a span for every proxy acquisition:
+
+```python
+from pharox import ProxyManager
+from pharox.models import AcquireEventPayload
+
+def otel_acquire_callback(payload: AcquireEventPayload) -> None:
+    status = "hit" if payload.lease else "miss"
+    with tracer.start_as_current_span("pharox.acquire") as span:
+        span.set_attribute("pharox.pool", payload.pool_name)
+        span.set_attribute("pharox.consumer", payload.consumer_name or "")
+        span.set_attribute("pharox.status", status)
+        span.set_attribute("pharox.duration_ms", payload.duration_ms)
+
+manager = ProxyManager(storage=...)
+manager.register_acquire_callback(otel_acquire_callback)
+```
+
+## Release Callback
+
+```python
+from pharox.models import ReleaseEventPayload
+
+def otel_release_callback(payload: ReleaseEventPayload) -> None:
+    with tracer.start_as_current_span("pharox.release") as span:
+        span.set_attribute("pharox.pool", payload.pool_name)
+        span.set_attribute("pharox.lease_duration_ms", payload.lease_duration_ms)
+
+manager.register_release_callback(otel_release_callback)
+```
+
+## Propagating an Existing Span Context
+
+If the calling code already holds an active span, the callbacks run inside that
+context automatically — the `with tracer.start_as_current_span(...)` call will
+create a child span of whatever is current in the calling thread.
+
+For explicit propagation across thread boundaries (e.g. when using
+`acquire_proxy_with_retry_async` via `asyncio.to_thread`):
+
+```python
+from opentelemetry.context import attach, detach
+from opentelemetry.propagate import extract
+
+def otel_acquire_callback(payload: AcquireEventPayload) -> None:
+    # carrier is set by your async caller before handing off to the thread
+    token = attach(extract(carrier={}))
+    try:
+        with tracer.start_as_current_span("pharox.acquire"):
+            ...
+    finally:
+        detach(token)
+```
+
+## Notes
+
+- Pharox callbacks are **synchronous**. If your OTLP exporter is async-only,
+  wrap the export in `asyncio.run_coroutine_threadsafe` or use a synchronous
+  exporter.
+- For high-throughput workers, prefer the `BatchSpanProcessor` over
+  `SimpleSpanProcessor` to avoid blocking the acquire path.
+- Combine with `PrometheusMetricsRecorder` — both can be registered as callbacks
+  on the same manager instance.

docs/how-to/production-deployment.md

@@ -0,0 +1,79 @@
+---
+title: Production Deployment Checklist
+description: Steps to harden a Pharox-based service before going live.
+---
+# Production Deployment Checklist
+
+Work through this list before exposing a Pharox-backed service to production
+traffic.
+
+## Storage
+
+- [ ] **Use `PostgresStorage`** — `InMemoryStorage` does not survive process
+  restarts and is not safe across multiple workers.
+- [ ] **Run Alembic migrations** before deploying a new version:
+  ```bash
+  alembic upgrade head
+  ```
+- [ ] **Connection pool sizing** — set `pool_size` and `max_overflow` on the
+  SQLAlchemy engine to match your expected concurrency.  A safe starting point:
+  `pool_size = <worker_threads>`, `max_overflow = 2`.
+- [ ] **SSL/TLS** — pass `connect_args={"sslmode": "require"}` (or `verify-full`)
+  to the engine when the database is on a separate host.
+
+## Proxy Data
+
+- [ ] Seed proxies via `add_proxies_bulk` for large initial loads — it uses a
+  single transaction and avoids per-row round-trips.
+- [ ] Set realistic `max_concurrency` on each pool; leaving it at `None`
+  (unlimited) is only appropriate for dev/testing.
+- [ ] Configure `expires_at` on leases so stale holds are cleaned up
+  automatically by `cleanup_expired_leases`.
+
+## Health Checks
+
+- [ ] Run `HealthCheckOrchestrator` in a dedicated thread or process — it is
+  blocking and CPU-light but I/O-heavy.
+- [ ] Tune `HealthCheckOptions.interval_seconds` and `timeout_seconds` to your
+  proxy SLA.  A common production setting: 60 s interval, 10 s timeout.
+- [ ] Archive or truncate old health records periodically — see
+  `docs/how-to/health-archival.md`.
+
+## Observability
+
+- [ ] Register `PrometheusMetricsRecorder` as acquire/release callbacks and
+  expose the `/metrics` endpoint to Prometheus.
+- [ ] Set alert rules on `pharox_acquire_total{status="miss"}` to detect pool
+  exhaustion.
+- [ ] Enable summaries (`enable_summaries=True`) and configure quantiles if
+  p95/p99 latency SLOs are required.
+
+## Retry / Resilience
+
+- [ ] Pass an explicit `RetryConfig` to `acquire_proxy_with_retry` rather than
+  relying on defaults — defaults are conservative and may not match your SLA.
+- [ ] Cap `RetryConfig.max_backoff_seconds` to avoid unbounded blocking in
+  synchronous workers.
+
+## Security
+
+- [ ] Store proxy credentials in a secrets manager (e.g. HashiCorp Vault, AWS
+  Secrets Manager) — do not hard-code them or commit them to source control.
+- [ ] Restrict database user to `SELECT`, `INSERT`, `UPDATE`, `DELETE` on
+  Pharox tables only — no DDL grants in production.
+- [ ] Rotate credentials via `update_proxy` rather than deleting and re-inserting
+  to preserve lease history.
+
+## Testing Before Go-Live
+
+- [ ] Run `poetry run pytest` locally with the Postgres extra to confirm the
+  adapter contract tests pass against a staging database.
+- [ ] Run `poetry run mypy src/pharox` — zero errors expected.
+- [ ] Run the production-template stack (`docs/how-to/production-template.md`)
+  and verify metrics appear in Grafana before switching production traffic.
+
+## Dependency Versions
+
+- [ ] Pin `pharox` to a specific minor version in your service's lockfile —
+  minor versions may add new abstract methods to `IStorage`.
+- [ ] Review `CHANGELOG.md` on every upgrade for breaking changes.

pyproject.toml

@@ -62,7 +62,8 @@ known-first-party = ["pharox"]
 
 [tool.pytest.ini_options]
 minversion = "7.0"
-addopts = "-ra -q --cov=src/pharox --cov-report=term-missing"
+addopts = "-ra -q --cov=src/pharox --cov-report=term-missing --cov-fail-under=85"
+filterwarnings = ["ignore::DeprecationWarning"]
 testpaths = ["tests"]
 markers = ["contract: marks adapter contract tests that may require external services"]
 
@@ -74,6 +75,9 @@ disallow_untyped_defs = true
 requires = ["poetry-core>=2.0.0,<3.0.0"]
 build-backend = "poetry.core.masonry.api"
 
+[tool.coverage.run]
+omit = ["src/pharox/storage/postgres/*", "src/pharox/benchmarks.py"]
+
 [tool.commitizen]
 name = "cz_conventional_commits"
 version_provider = "poetry"

src/pharox/__init__.py

@@ -42,6 +42,7 @@
 )
 from .observability import (
     DEFAULT_LATENCY_BUCKETS,
+    DEFAULT_QUANTILES,
     PrometheusMetricsRecorder,
     StructuredLogger,
     register_prometheus_metrics,
@@ -97,4 +98,5 @@
     "StructuredLogger",
     "register_prometheus_metrics",
     "DEFAULT_LATENCY_BUCKETS",
+    "DEFAULT_QUANTILES",
 ]

src/pharox/observability/__init__.py

@@ -3,12 +3,14 @@
 from .logging import StructuredLogger
 from .metrics import (
     DEFAULT_LATENCY_BUCKETS,
+    DEFAULT_QUANTILES,
     PrometheusMetricsRecorder,
     register_prometheus_metrics,
 )
 
 __all__ = [
     "DEFAULT_LATENCY_BUCKETS",
+    "DEFAULT_QUANTILES",
     "PrometheusMetricsRecorder",
     "StructuredLogger",
     "register_prometheus_metrics",