feat: add soft_5xx mode - #575
Merged
Merged
Conversation
added 2 commits
July 7, 2026 18:21
luthfifahlevi
force-pushed
the
feat/soft-5xx-mode-airflow
branch
from
July 7, 2026 11:22
3bb316c to
27551c7
Compare
Mryashbhardwaj
self-requested a review
July 8, 2026 04:34
Mryashbhardwaj
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add SOFT_5XX third-party sensor mode and improve dex error visibility
Summary
THIRD_PARTY_SENSOR_TOGGLE_SOFT_5XX, a new sensor mode betweenONandSOFT: behaves likeON(hard failure — keeps retrying until the sensor eventually times out) by default, but falls back toSOFT's bypass-after-THIRD_PARTY_SENSOR_MAX_TIMEbehavior specifically when dex itself returns a 5xx — since a dex outage is a dex-side problem, not genuine "data not ready yet," and shouldn't silently block the pipeline forever the way real unavailability should.SuperExternal3rdPartyTaskSensor.poke()from nested nested conditionals into an explicit if/elif dispatch over the toggle value, with shared_hard_check/_apply_soft_grace_periodhelpers so the three active modes (ON,SOFT,SOFT_5XX) share logic instead of duplicating it.is_upstream_data_availablenow returns(is_available, is_5xx_error)instead of a bare bool, via a newDexSensorAPIErrorexception that carries the HTTP status code so callers can distinguish a confirmed dex 5xx from other failure types (network error, 4xx, bad payload).getCompletenessStatsinext/dex/dex.go) and the Airflow sensor (execute_third_party_sensorin__lib.py) now log a curl-equivalent of the failing request (method + URL, auth header intentionally omitted) on error, so a failure is reproducible without cross-referencing a separate debug log line.Test plan
go build ./...,go vet ./ext/dex/...python3 -m py_compile ext/scheduler/airflow/__lib.py