Skip to content

feat: add soft_5xx mode - #575

Merged
luthfifahlevi merged 3 commits into
mainfrom
feat/soft-5xx-mode-airflow
Jul 8, 2026
Merged

feat: add soft_5xx mode#575
luthfifahlevi merged 3 commits into
mainfrom
feat/soft-5xx-mode-airflow

Conversation

@luthfifahlevi

@luthfifahlevi luthfifahlevi commented Jul 6, 2026

Copy link
Copy Markdown

Add SOFT_5XX third-party sensor mode and improve dex error visibility

Summary

  • Adds THIRD_PARTY_SENSOR_TOGGLE_SOFT_5XX, a new sensor mode between ON and SOFT: behaves like ON (hard failure — keeps retrying until the sensor eventually times out) by default, but falls back to SOFT's bypass-after-THIRD_PARTY_SENSOR_MAX_TIME behavior specifically when dex itself returns a 5xx — since a dex outage is a dex-side problem, not genuine "data not ready yet," and shouldn't silently block the pipeline forever the way real unavailability should.
  • Restructures SuperExternal3rdPartyTaskSensor.poke() from nested nested conditionals into an explicit if/elif dispatch over the toggle value, with shared _hard_check/_apply_soft_grace_period helpers so the three active modes (ON, SOFT, SOFT_5XX) share logic instead of duplicating it.
  • is_upstream_data_available now returns (is_available, is_5xx_error) instead of a bare bool, via a new DexSensorAPIError exception that carries the HTTP status code so callers can distinguish a confirmed dex 5xx from other failure types (network error, 4xx, bad payload).
  • Both the Dex client (getCompletenessStats in ext/dex/dex.go) and the Airflow sensor (execute_third_party_sensor in __lib.py) now log a curl-equivalent of the failing request (method + URL, auth header intentionally omitted) on error, so a failure is reproducible without cross-referencing a separate debug log line.

Test plan

  • go build ./..., go vet ./ext/dex/...
  • python3 -m py_compile ext/scheduler/airflow/__lib.py

@luthfifahlevi luthfifahlevi self-assigned this Jul 6, 2026
@luthfifahlevi
luthfifahlevi force-pushed the feat/soft-5xx-mode-airflow branch from 3bb316c to 27551c7 Compare July 7, 2026 11:22
Comment thread ext/scheduler/airflow/__lib.py Outdated
Comment thread ext/scheduler/airflow/__lib.py Outdated
@Mryashbhardwaj
Mryashbhardwaj self-requested a review July 8, 2026 04:34
@luthfifahlevi
luthfifahlevi merged commit 55f3e78 into main Jul 8, 2026
3 checks passed
@luthfifahlevi
luthfifahlevi deleted the feat/soft-5xx-mode-airflow branch July 8, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants