Skip to content

Latest commit

 

History

History
310 lines (246 loc) · 13 KB

File metadata and controls

310 lines (246 loc) · 13 KB

GlobeNewsWire Signal Pipeline — LangGraph

What this project does

Scrapes press releases from GlobeNewswire, classifies each one into a 7-point price-movement signal using a three-step LLM + market-data pipeline, and writes a single annotated JSON file. Steps:

  1. Build a GlobeNewswire search URL from configured filters
  2. Scrape paginated search results → article stubs (title, date, source, link)
  3. Fetch full article bodies (SQLite-cached to avoid re-fetching on re-runs)
  4. Save raw articles to export/articles_<timestamp>_<mode>.json
  5. For each article: Extract (LLM → structured facts) → Enrich (yfinance → market data) → Verdict (LLM → signal)
  6. Merge signal data back into the same JSON file

Project structure

release_evaluation_langgraph/
│
├── CONFIG.py                             ← THE ONLY FILE YOU EDIT for a new run
├── run.py                                ← entry point: python run.py
├── graph.py                              ← main LangGraph StateGraph
├── state.py                              ← PipelineState + ArticleSignalState TypedDicts
│
├── nodes/                                ← one file per graph node
│   ├── build_url.py                      ← resolves date range, builds search URL
│   ├── scrape_pages.py                   ← paginates search results (internal while loop)
│   ├── fetch_bodies.py                   ← body fetching with article cache
│   ├── save_articles.py                  ← saves raw articles JSON
│   ├── run_signals.py                    ← signal orchestrator (cache check + strategy loop)
│   └── merge_output.py                   ← merges records into JSON, overwrites file
│
├── signal_strategies/
│   ├── base.py                           ← BaseSignalStrategy ABC
│   ├── __init__.py                       ← registry: get_strategy(), register_strategy()
│   └── default/
│       ├── strategy.py                   ← DefaultSignalStrategy (LangGraph subgraph internally)
│       ├── STRATEGY.md                   ← taxonomy, schema, signal logic, articles log
│       ├── extractor.py                  ← Step 1: LLM extraction
│       ├── enricher.py                   ← Step 2: yfinance market data
│       └── verdict.py                    ← Step 3: LLM verdict
│
├── utils/                                ← infrastructure only — NOT strategy-specific
│   ├── url_builder.py
│   ├── article_scraper.py
│   ├── article_fetcher.py
│   ├── article_cache.py                  ← SQLite cache for article bodies
│   ├── signal_cache.py                   ← SQLite cache for signal records
│   ├── llm_client.py                     ← unified LLM client (QGenie / OpenRouter)
│   └── utils.py                          ← save_articles(), build_filename()
│
├── filter_mapping.json                   ← maps human labels → GlobeNewswire URL codes
├── README.md
└── export/                               ← output files (gitignored, created at runtime)

GlobeNewsWire/db/
└── articles_cache.db                     ← SHARED SQLite DB (not inside this folder)

LangGraph architecture

Main pipeline graph (graph.py)

State type: PipelineState (see state.py)

START
  │
  ▼
build_url_node          Reads filter fields from state (populated from CONFIG).
                        Resolves date_range: today if live, CONFIG.DATE_RANGE if backtest.
                        Writes: state.base_url, state.date_range
  │
  ▼
scrape_pages_node       Internal while-loop: page=1,2,3… until scrape_articles() returns [].
                        Writes: state.article_stubs
  │
  ├─[no stubs]──► END   Conditional edge — graceful exit if no articles found
  │
  ▼
fetch_bodies_node       Per-stub: get_cached(link) → use; else fetch_article() + upsert_cache().
                        Logs [CACHED] or [FETCHED] per article.
                        Writes: state.articles
  │
  ▼
save_articles_node      save_articles(articles, base_url, mode) → JSON file.
                        Writes: state.export_path
  │
  ▼
run_signals_node        Sequential loop. Per article:
                          1. get_signal(link) → if cached: log [CACHED], use record, continue
                          2. if skipped flag: log [SKIP], continue
                          3. strategy.process_article(article) → log [DONE] or [SKIP]
                        Writes: state.signal_records  {link → record}
  │
  ▼
merge_output_node       Merges signal_records into articles list.
                        Reads export_path JSON, adds signals_at + strategy fields, overwrites.
                        Writes: state.merged_articles, state.run_summary
  │
  ▼
END

Signal strategy subgraph (signal_strategies/default/strategy.py)

State type: ArticleSignalState (see state.py)

Called via strategy.process_article(article) from run_signals_node. Only invoked on cache missesrun_signals_node handles the cache check and short-circuits before calling the strategy.

START
  │
  ▼
extract_node            LLM reads article → ticker, event_category, event_subtype,
                        deal_size_usd, offering_price_usd, shares_issued, is_binding, etc.
                        Writes: state.extracted
  │
  ├─[no ticker]──► mark_skipped_node ──► END   (returns None to run_signals_node)
  │
  ▼
enrich_node             yfinance: 370-day history window, all anchored to T-1.
                        Fetches prev_close, 52w_high/low, avg_volume_30d, shares_outstanding.
                        compute_ratios() fills deal_pct_market_cap, offering_discount_pct, dilution_pct.
                        Writes: state.enriched
  │
  ▼
verdict_node            LLM receives extracted + enriched → 7-point signal, confidence, goods, bads.
                        RULE: never receives T+1 data.
                        Writes: state.signal_record
  │
  ▼
save_signal_node        upsert_signal(link, record) → SQLite signals table.
  │
  ▼
END  (returns state.signal_record to run_signals_node)

State definitions (state.py)

PipelineState (main graph):
  Filters:         run_mode, date_range, exchanges, languages, subjects, ...
  Computed:        base_url, article_stubs, articles, export_path
  Signal results:  signal_records  {link → {extracted, enriched, signal, evaluation}}
  Final output:    merged_articles, run_summary

ArticleSignalState (strategy subgraph):
  Input:    articlefull article dict {title, published_date, source, link, body}
  Step 1:   extractedLLM-extracted facts
  Step 2:   enrichedyfinance market data + ratios
  Output:   signal_record — {link, extracted, enriched, signal, evaluation}
  Error:    skipped, skip_reason  ("no_ticker" | "error:<msg>")

The signal scale

speculative_bearish → bearish → mildly_bearish → neutral → mildly_bullish → bullish → speculative_bullish

Full taxonomy, decision rules, and worked examples are in signal_strategies/default/STRATEGY.md.


SQLite database

Shared DB at GlobeNewsWire/db/articles_cache.db — path defined as CONFIG.DB_PATH. Both tables are keyed on article link.

Table Contents
articles Cached article bodies. Prevents re-fetching on re-runs.
signals Full signal records (extracted + enriched + signal + evaluation). skipped=1 means a previous run failed for this link — no LLM call is made on re-run. upsert_signal() clears the flag if a later run succeeds.

Output JSON format

{
  "fetched_at":     "2026-04-20 14:30:22",
  "total_articles": 42,
  "url":            "https://www.globenewswire.com/en/search/...",
  "signals_at":     "2026-04-20 16:45:30",
  "strategy":       "default",
  "articles": [
    {
      "title":          "...",
      "published_date": "April 20, 2026 16:05 ET",
      "source":         "...",
      "link":           "...",
      "body":           "...",
      "extracted":      { "ticker": "TRVI", "event_category": "dilutive_equity", ... },
      "enriched":       { "prev_close": 15.4, "deal_pct_market_cap": 8.46, ... },
      "signal":         { "signal": "bearish", "confidence": "high", "goods": [...], "bads": [...], "reasoning": "..." },
      "evaluation":     null
    }
  ]
}

Articles where no US ticker was found: extracted, enriched, signal, evaluation are all null.


CONFIG.py — the only file you edit

RUN_MODE        = "live"       # "live" = today; "backtest" = DATE_RANGE below
DATE_RANGE      = ("2026-02-28", "2026-04-20")   # backtest only
EXCHANGES       = ["NYSE", "Nasdaq"]
LANGUAGES       = ["English"]
SUBJECTS        = ["Financing Agreements"]
INDUSTRIES      = None         # None = all
LLM_PROVIDER    = "qgenie"     # "qgenie" | "openrouter"
SIGNAL_STRATEGY = "default"    # must be registered in signal_strategies/__init__.py
DB_PATH         = ...          # auto-set to GlobeNewsWire/db/articles_cache.db

API keys go in .env, never in CONFIG.py:

QGENIE_API_KEY=...
OPENROUTER_API_KEY=...

Running the pipeline

cd release_evaluation_langgraph
python run.py

Output: export/articles_YYYY-MM-DD_HHMMSS_<mode>.json


Adding a new signal strategy

The main graph is strategy-agnostic. To add a new strategy:

  1. Create signal_strategies/my_strategy/ with two files:

    • strategy.py — implements BaseSignalStrategy
    • STRATEGY.md — documents the logic, taxonomy, decision rules for this strategy
  2. strategy.py minimal shape:

    from signal_strategies.base import BaseSignalStrategy
    
    class MyStrategy(BaseSignalStrategy):
        def process_article(self, article: dict) -> dict | None:
            # article: {title, published_date, source, link, body}
            # return: {link, extracted, enriched, signal, evaluation}
            # return None to skip (signal_cache.mark_skipped is called by run_signals_node)
            ...
        def name(self) -> str:
            return "my_strategy"
  3. Register in signal_strategies/__init__.py:

    from signal_strategies.my_strategy.strategy import MyStrategy
    _REGISTRY["my_strategy"] = MyStrategy
  4. Set SIGNAL_STRATEGY = "my_strategy" in CONFIG.py.

Zero changes to graph.py, nodes/, state.py, or utils/.


Conventions — follow these exactly

CONFIG.py is the single source of truth for all settings. No CLI arguments, no environment-level overrides. If something should be configurable, add it to CONFIG.py.

Cache ownership: run_signals_node owns the signal cache check. It checks get_signal(link) before calling the strategy and handles [CACHED]/[SKIP] short-circuits. The strategy's process_article() should assume it is only called on cache misses. Do not add cache checks inside a strategy.

upsert_signal() is called by the strategy's save_signal_node. mark_skipped() is called either by mark_skipped_node inside the strategy subgraph (no ticker) or directly in run_signals_node's exception handler (unexpected error). Do not call these from merge_output_node or anywhere else.

No next-day data in signals. The enrich_node only fetches data observable at T-1. verdict_node never receives T+1 or later data. Next-day closes are reserved for the evaluation block (Step 4, not yet built).

Signal records carry evaluation: null. The evaluation block is filled offline by a future evaluator (Step 4) via upsert_evaluation(link, evaluation). Never populate it in Steps 1–3.

Strategy-specific LLM logic lives inside the strategy folder. extractor.py, enricher.py, verdict.py are part of the default strategy — not shared infrastructure. If a new strategy needs different extraction logic, it gets its own copies. utils/ contains only infrastructure (HTTP, SQLite, LLM client abstraction).

utils/llm_client.py is the only LLM import point. extractor.py and verdict.py import from utils.llm_client import chat only. Never import openai or qgenie directly in strategy code.

merge_output_node strips article_title and article_link from extracted before writing to JSON (those fields are LLM-internal, not part of the public record schema). Do not add them back.

Output filename convention: export/articles_<YYYY-MM-DD_HHMMSS>_<mode>.json


What is NOT here yet (Step 4)

evaluation is null in all current records. Step 4 (evaluator) will:

  • Fetch T+1, T+2, T+5 closes from yfinance after the market has moved
  • Compute actual_return_1d/2d/5d, volume_ratio_1d, signal_correct, magnitude_error
  • Write via utils.signal_cache.upsert_evaluation(link, evaluation_dict)
  • Update the evaluation field in the articles JSON

This runs offline (never during the live pipeline) to prevent any look-ahead bias.