Skip to content

feat: potential SLA breach batch - #571

Merged
luthfifahlevi merged 6 commits into
mainfrom
feat/potential-sla-brench-batch
Jul 6, 2026
Merged

feat: potential SLA breach batch#571
luthfifahlevi merged 6 commits into
mainfrom
feat/potential-sla-brench-batch

Conversation

@luthfifahlevi

@luthfifahlevi luthfifahlevi commented Jul 1, 2026

Copy link
Copy Markdown

Batch and consolidate potential SLA breach alerts per team

Summary

  • IdentifyPotentialSLABreach now accepts repeated project_names and label_groups in one request instead of requiring a fan-out of multiple separate per-project/per-label calls, each of which triggered its own alert — teams previously got fragmented, noisy notifications instead of one clear summary.
  • Server-side logic is split into pure detection (computeBreaches) and batch consolidation (IdentifySLABreachesBatch), which aggregates every project/label-group combo into exactly one alert per team, structured as project → SLA group → target job → causing jobs, tolerant of individual combo failures.
  • Alert payload/model reworked to carry this structure end-to-end (proto, internal PotentialSLABreachAttrs, AlertManager adapter, Siren template); per-group severity was considered but dropped after team discussion in favor of a single request-level severity, keeping the request/response shape simpler.
  • Adds a full local end-to-end test harness (colima + local Kubernetes + real Airflow + Optimus built from source) and documents it in dev/README.md, including several pre-existing local dev-setup bugs found and fixed along the way (missing required server config causing crash loops, a hardcoded gRPC port mismatch, a plugin-discovery path bug, and an Airflow Helm chart version pin that had gone stale).

Test plan

  • go build ./...
  • go vet ./core/scheduler/... ./ext/notify/...
  • Unit tests: core/scheduler/service, core/scheduler/handler/v1beta1, ext/notify/alertmanager
  • Verified end-to-end against a from-scratch local Kubernetes + real Airflow + Optimus environment (5 real jobs, actual KubernetesPodOperator execution, real dependency chain) — confirmed correct multi-level lineage attribution and real alert delivery to Siren rendering through the updated template
  • Staging validation (pending)

@luthfifahlevi luthfifahlevi self-assigned this Jul 1, 2026
Comment thread core/scheduler/handler/v1beta1/job_run.go Outdated
Comment thread core/scheduler/handler/v1beta1/job_run.go
Comment thread core/scheduler/service/job_sla_predictor_service.go Outdated
Comment thread core/scheduler/service/job_sla_predictor_service.go Outdated

@ahmadnaufal ahmadnaufal left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few more comments, other than that LGTM

Comment thread core/scheduler/handler/v1beta1/job_run.go Outdated
Comment thread core/scheduler/service/job_sla_predictor_service.go Outdated
Comment thread core/scheduler/service/job_sla_predictor_service.go
@luthfifahlevi
luthfifahlevi merged commit e8c90f0 into main Jul 6, 2026
3 checks passed
@luthfifahlevi
luthfifahlevi deleted the feat/potential-sla-brench-batch branch July 6, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants