Phase 3

[fzheng]

No description provided.

[fzheng]

High-level summary

Feature: Adds “Alpha Pool” support across services (hl-stream, hl-scout, hl-sage, ts-lib/hyperliquid). New endpoints for alpha-pool data, backfill/holdings/last-activity, and UI work in the dashboard.
Risk/ops: Introduces a weight-aware rate limiter and retry wrapper for Hyperliquid SDK calls (packages/ts-lib/hyperliquid.ts) — good for robustness but needs careful logging and observability tuning.
Tests: Adds unit and E2E tests (pytest in hl-decide and Playwright e2e tests) and test deps to requirements.
UX: Dashboard HTML/CSS updated to surface Alpha Pool refresh status, holdings, last-activity toggle, and new tab separation (Legacy vs Alpha).
Backwards compatibility: Several endpoints are split into explicit Legacy endpoints (e.g. /legacy/fills/) and new /alpha-pool/ endpoints.
High-priority issues and suggestions (actionable)

Missing request body when proxying non-GET requests (likely break for POST proxy)
Where you changed: fetch(target, { method: req.method, headers: { 'x-owner-key': OWNER_TOKEN }, });
Problem: If req.method is POST/PUT the handler does not forward the request body, content-type, or other headers (and probably not the original req stream). That will break proxied POST endpoints (e.g. adding pinned accounts or alpha-pool refresh if proxySage/proxyScout use the same pattern).
Fix: For proxying, forward method, headers (including Content-Type), and the request body (or pipe the stream). Example (Express + node-fetch): include body: JSON.stringify(req.body) for JSON, or use req.pipe when streaming. Also copy other headers that matter (authorization, content-type). If using fetch with RequestInit, for large bodies prefer streaming.
Authorization when proxying to hl-sage
Observation: New hl-stream alpha-pool endpoints call sageUrl directly (fetch(${sageUrl}/alpha-pool/addresses?...)) without the OWNER_TOKEN header. If hl-sage requires authentication/owner-key, these calls will fail or leak access.
Suggestion: Either use the same proxy function that includes OWNER_TOKEN or ensure proxySage forwards appropriate credentials. Audit all fetch() calls to internal services and ensure owner-key is forwarded where required.
SQL address case-sensitivity bug in getLastActivityPerAddress
Where: packages/ts-lib/src/persist.ts -> getLastActivityPerAddress
Issue: You normalize addresses to lowercase in JS, but the SQL uses WHERE address = ANY($1) (no LOWER(address)). If the DB stored addresses in mixed-case, the match may fail.
Fix: Change WHERE to use LOWER(address) = ANY($1) OR pass canonical-cased addresses that match DB storage format. Example: WHERE type = 'trade' AND LOWER(address) = ANY($1) and query with normalizedAddresses.
Missing import: math used in consensus.calculate_vote_weight
Where: services/hl-decide/app/consensus.py
Issue: calculate_vote_weight uses math.sqrt and math.log but I don’t see math imported in the diff. This will raise NameError at runtime.
Fix: Add "import math" at the top.
Inconsistent logging / print usage
Where: consensus.py now includes print() calls for gating warnings.
Suggestion: Replace print() with your structured logger (logger.xxx) to keep consistency and observability.
ATR gating: strict mode and fallback semantics
Observation: You introduced ATR-based gating and an "is_atr_valid_for_gating" check. You also kept a legacy fallback constant CONSENSUS_MAX_PRICE_BAND_BPS and a new CONSENSUS_MAX_PRICE_DRIFT_R. This is good, but:
Ensure ATR provider updates are robust and ATR_STRICT_MODE behavior is documented/configurable in staging vs prod.
In strict mode (ATR_STRICT_MODE=true) you block signals if ATR data is missing; consider an explicit admin/ops fallback metric or safe-mode flag to avoid accidental full stoppage.
Suggest adding clear logging and metrics around "atr validity" to detect when signals are blocked because ATR unavailable.
eff_k_from_corr fallback_counter_callback semantics
Issue: The code tracks fallback_count for unknown correlation pairs and calls fallback_counter_callback() if fallback_count > 0, but it doesn’t pass the fallback_count nor adjust for double-counting; comment mentions dividing by 2 but callback not given a value.
Suggestion: Either pass the fallback_count (or unique-pair count) to the callback or adjust the logic so it reports the correct number. Example: compute fallback_count_pairs = fallback_count / 2 then call callback(fallback_count_pairs) or increment a metric with that value.
withRateLimitAndRetry: swallowing non-429 errors
Where: packages/ts-lib/src/hyperliquid.ts
Issue: On any non-rate-limit error, withRateLimitAndRetry returns the fallback silently. That can mask real errors and make debugging harder.
Suggestion: At a minimum log the error with operationName and attempt count. Consider rethrowing non-rate-limit errors in dev/CI, or make a configurable behavior: either return fallback or rethrow/log. E.g. log error stack and operation details.
SQL casting of payload->>'at' to timestamptz may raise on bad format
Where: multiple queries using COALESCE((payload->>'at')::timestamptz, at)
Issue: If payload->>'at' is present but in an unexpected format, the cast (::timestamptz) will throw and the query will error (COALESCE won’t rescue cast exceptions).
Suggestion: Use try_cast patterns or safe conversion in SQL (e.g., to_timestamp(...) or check with to_timestamp only if matches a known format), or ensure all payload timestamps are normalized before insert. Alternatively, use an SQL function that returns NULL on parse failure.
Adding many new endpoints — auth, rate-limiting, and monitoring
Recommendation: New endpoints that trigger heavy DB or cross-service work (alpha-pool/refresh, holdings, fetch-history) should be:
Protected (owner tokens or admin-only)
Rate-limited
Instrumented with metrics and structured logs to track usage and failures
For example, alpha-pool/fetch-history may cause a big DB scan — ensure it’s guarded.
Proxy fetch change (method-only) — cross-site headers & CORS
The change to fetch with method but no headers beyond owner-key may drop other important headers (e.g., X-Forwarded-For, content-type). Ensure required headers are forwarded and CORS considerations covered if requests come from the browser.
Tests: heavy new E2E tests may be flaky
The Playwright tests are comprehensive, but many use page.waitForTimeout and rely on default timeouts. Consider replacing fixed timeouts with explicit waits for relevant network requests / DOM states to reduce flakiness.
Also ensure e2e tests are runnable in CI (they changed baseURL to localhost, which is appropriate for dev; confirm CI's network / container routing matches).
Minor/UX/Style notes

In dashboard.css there are duplicate @Keyframes pulse definitions added; consolidate to single definition to avoid confusion.
Some CSS rules use animations and spinner sizes changed — verify visual regression on small screens.
In packages/ts-lib listLiveFillsForAddresses, whereClause building is OK but ensure you sanitize placeholders properly and guard against empty address arrays (you already handle that).
Consider adding more metrics/logs for the new alpha-pool refresh endpoint: start/finish durations, per-step progress.
Suggested follow-ups (I can do these if you want)

I can produce patch snippets for:
forwarding request body in proxy fetch (Express code)
fix persist.getLastActivityPerAddress SQL to use LOWER(address)
add missing import math to consensus.py and replace print with logger
improve withRateLimitAndRetry to log non-429 errors
Run a quick code search for other places where request proxying was changed to ensure all proxies forward bodies/headers.
Run a quick grep for uses of "OWNER_TOKEN" / proxy patterns to ensure authentication is consistently forwarded.