Fix/ash oban tenant scheduler broken

[mfreeman451]

User description

IMPORTANT: Please sign the Developer Certificate of Origin

Thank you for your contribution to ServiceRadar. Please note, when contributing, the developer must include
a DCO sign-off statement indicating the DCO acceptance in one commit message. Here
is an example DCO Signed-off-by line in a commit message:

Signed-off-by: J. Doe <[email protected]>

Describe your changes

Issue ticket number and link

Code checklist before requesting a review

• I have signed the DCO?
• The build completes without errors?
• All tests are passing when running make test?

---

PR Type

Enhancement, Bug fix

---

Description

• Major architectural refactor from poller to gateway model: Comprehensive rename and restructuring of the entire codebase to replace the poller-based architecture with a new gateway-based push-first communication model

• Multi-tenant support: Implemented full multi-tenant architecture with tenant ID and slug propagation throughout the system

• Gateway monitoring and status management: New comprehensive gateway health checking, offline/recovery detection, and alert handling with periodic monitoring and streaming status report support

• NATS bootstrap and account management: Added new NATS bootstrap functionality with JWT and credentials file management, supporting multi-tenant routing and operator/account generation

• Push-based result delivery: Refactored sync service to use gateway client for push-based result delivery instead of pull-based KV storage

• Dynamic configuration updates: Implemented gateway enrollment, config polling, and heartbeat loops for dynamic configuration management

• Protobuf message updates: Refactored protobuf messages from poller to gateway architecture with new agent-gateway communication types

• Removed legacy components: Deleted poller package, poller-related CLI commands, and associated infrastructure code

• Updated terminology: Systematic updates across API documentation, SNMP checker, edge onboarding, and all related services

---

Diagram Walkthrough

flowchart LR
  A["Poller Architecture<br/>Pull-based KV storage"] -->|Refactor| B["Gateway Architecture<br/>Push-based delivery"]
  B --> C["Multi-tenant<br/>Support"]
  B --> D["NATS Bootstrap<br/>& Account Mgmt"]
  B --> E["Gateway Monitoring<br/>& Status Mgmt"]
  C --> F["Tenant ID/Slug<br/>Propagation"]
  D --> G["JWT & Credentials<br/>Management"]
  E --> H["Health Checks<br/>& Recovery"]

Loading

File Walkthrough

	Relevant files
Enhancement	6 files monitoring.pb.go Refactor protobuf messages from poller to gateway architecture proto/monitoring.pb.go Renamed PollerId field to GatewayId across multiple message types (StatusRequest, ResultsRequest, StatusResponse, ResultsResponse) Removed deprecated PollerStatusRequest, PollerStatusResponse, and ServiceStatus message types Replaced PollerStatusChunk with new GatewayStatusRequest, GatewayStatusResponse, and GatewayStatusChunk types Added new GatewayServiceStatus message type with tenant-related fields (TenantId, TenantSlug) Added new agent-gateway communication messages: AgentHelloRequest, AgentHelloResponse, AgentConfigRequest, AgentConfigResponse, AgentCheckConfig Updated proto descriptor indices to reflect removed message types +1039/-404 nats_bootstrap.go Add comprehensive NATS bootstrap and configuration management pkg/cli/nats_bootstrap.go New file implementing NATS bootstrap functionality with handlers for nats-bootstrap and admin nats subcommands Provides local and remote NATS operator/account generation with JWT and credentials file management Implements bootstrap token generation, status checking, and tenant account listing via Core API Includes configuration validation, file I/O operations, and both text and JSON output formats Supports multi-tenant routing with tenant ID and slug propagation +961/-0  service.go Multi-tenant gateway push architecture with dynamic config pkg/sync/service.go Refactored sync service to support multi-tenant architecture with gateway push-first communication model Removed KV client and gRPC client dependencies; added gateway client for push-based result delivery Implemented per-source discovery scheduling with interval tracking and per-tenant source configuration management Added gateway enrollment, config polling, and heartbeat loops for dynamic configuration updates Replaced streaming results storage with device/source count tracking; deprecated pull-based GetResults API +1260/-316 gateways.go Complete gateway monitoring and status management system  pkg/core/gateways.go New comprehensive gateway monitoring and status management implementation with 1540 lines of core functionality Implements gateway health checks, offline/recovery detection, and alert handling with periodic monitoring Adds streaming status report support for large datasets with chunk reassembly and service message handling Includes gateway registration, device inventory updates, and integration with service registry and event publishing +1540/-0 edge_onboarding.go Rename poller to gateway terminology in edge onboarding    pkg/core/edge_onboarding.go Systematic rename of all poller references to gateway throughout the edge onboarding service (200+ occurrences) Updates type names, function names, variable names, and comments to reflect gateway terminology Adds support for EdgeOnboardingComponentTypeSync component type in package creation and validation Updates KV config paths from config/pollers/ to config/gateways/ and related metadata field names +208/-206 main.go Update SNMP checker to use gateway terminology                      cmd/checkers/snmp/main.go Updates SNMP checker service instantiation from NewSNMPPollerService to NewSNMPGatewayService Changes struct type from snmp.Poller to snmp.Gateway to align with gateway terminology +1/-1
Documentation	1 files main.go Update API documentation terminology                                          cmd/core/main.go Updated API description from "service pollers" to "service gateways" in Swagger documentation +1/-1
Configuration changes	2 files prod.exs Add Elixir production configuration                                            elixir/serviceradar_agent_gateway/config/prod.exs New production configuration file for Elixir agent-gateway service Sets logger level to info for production environment +4/-0      dev.exs Development configuration for Elixir gateway service          elixir/serviceradar_agent_gateway/config/dev.exs Added new development configuration file for Elixir service Sets logger level to debug for development environment +4/-0
Dependencies	2 files nats_account.pb.go NATS account management protobuf definitions                          proto/nats_account.pb.go Generated protobuf code for NATS account management service with 15 message types and 1 enum Defines account limits, subject mappings, user credentials, and operator bootstrap functionality Includes request/response types for tenant account creation, user credential generation, and JWT signing +1266/-0 nats_account_grpc.pb.go Generated gRPC code for NATS account service                          proto/nats_account_grpc.pb.go Auto-generated gRPC service code for NATS account management operations Implements client and server interfaces for JWT signing, account creation, and credential generation Provides stateless operations for NATS operator bootstrap and tenant account management +376/-0
Miscellaneous	1 files mock_armis.go Remove KVWriter from Armis mock interfaces                              pkg/sync/integrations/armis/mock_armis.go Updated mock generation command to remove KVWriter interface from mocked interfaces Reflects removal of KV writing functionality from Armis integration +2/-40
Additional files	101 files .bazelignore +4/-0      .bazelrc +5/-0      .env-sample +33/-0    .env.example +38/-0    main.yml +18/-0    AGENTS.md +177/-11 INSTALL.md +11/-11  MODULE.bazel +5/-0      Makefile +55/-14  README-Docker.md +17/-2    README.md +3/-3      ROADMAP.md +1/-1      BUILD.bazel +11/-6    BUILD.bazel +12/-0    mix_release.bzl +124/-49 BUILD.bazel +1/-0      README.md +4/-4      config.json +5/-6      main.go +174/-74 build.rs +0/-1      monitoring.proto +3/-26    server.rs +2/-0      README.md +2/-2      monitoring.proto +2/-26    server.rs +6/-6      main.go +16/-2    config.rs +26/-0    grpc_server.rs +2/-2      message_processor.rs +1/-0      nats.rs +4/-0      zen-consumer-with-otel.json +1/-0      zen-consumer.json +1/-0      app.go +1/-1      config.json +4/-4      config.json +4/-4      BUILD.bazel +1/-0      main.go +68/-0    README.md +3/-3      README.md +9/-12    flowgger.toml +1/-0      nats_output.rs +14/-0    otel.toml +1/-0      config.rs +13/-0    nats_output.rs +7/-0      setup.rs +1/-0      BUILD.bazel +0/-25    config.json +0/-111  main.go +0/-138  BUILD.bazel +0/-25    config.json +0/-77    main.go +0/-123  main.go +1/-1      config.rs +21/-0    main.rs +23/-3    docker-compose.elx.yml +109/-0  docker-compose.spiffe.yml +8/-157  docker-compose.yml +318/-269 Dockerfile.agent-gateway +94/-0    Dockerfile.core-elx +108/-0  Dockerfile.poller +0/-70    Dockerfile.sync +0/-95    Dockerfile.tools +1/-2      Dockerfile.web-ng +6/-0      agent-minimal.docker.json +6/-6      agent.docker.json +5/-20    agent.mtls.json +7/-10    bootstrap-nested-spire.sh +0/-80    .gitkeep +1/-0      datasvc.docker.json +3/-2      datasvc.mtls.json +14/-1    db-event-writer.docker.json +2/-2      db-event-writer.mtls.json +3/-2      FRICTION_POINTS.md +0/-355  README.md +0/-207  SETUP_GUIDE.md +0/-307  docker-compose.edge-e2e.yml +0/-27    manage-packages.sh +0/-211  setup-edge-e2e.sh +0/-198  edge-poller-restart.sh +0/-178  downstream-agent.conf +0/-32    env +0/-4      server.conf +0/-51    upstream-agent.conf +0/-32    entrypoint-certs.sh +13/-9    entrypoint-poller.sh +0/-274  entrypoint-sync.sh +0/-96    fix-cert-permissions.sh +2/-2      flowgger.docker.toml +2/-1      generate-certs.sh +214/-12 nats.docker.conf +16/-160 netflow-consumer.mtls.json +1/-0      otel.docker.toml +2/-0      pg_hba.conf +9/-0      pg_ident.conf +17/-0    poller-stack.compose.yml +0/-121  poller.docker.json +0/-128  poller.mtls.json +0/-135  poller.spiffe.json +0/-55    refresh-upstream-credentials.sh +0/-248  seed-poller-kv.sh +0/-83    Additional files not shown

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Unvalidated External IDs: New externally-supplied identifiers (gateway_id, tenant_id, tenant_slug) are introduced in request messages without any visible validation/sanitization in the diff, requiring verification that upstream handlers enforce expected formats and authorization boundaries. Referred Code // GatewayStatusRequest is sent by agents to push their status to the gateway. type GatewayStatusRequest struct { state protoimpl.MessageState `protogen:"open.v1"` Services []*GatewayServiceStatus `protobuf:"bytes,1,rep,name=services,proto3" json:"services,omitempty"` GatewayId string `protobuf:"bytes,2,opt,name=gateway_id,json=gatewayId,proto3" json:"gateway_id,omitempty"` // Gateway receiving this status AgentId string `protobuf:"bytes,3,opt,name=agent_id,json=agentId,proto3" json:"agent_id,omitempty"` // Agent sending this status Timestamp int64 `protobuf:"varint,4,opt,name=timestamp,proto3" json:"timestamp,omitempty"` Partition string `protobuf:"bytes,5,opt,name=partition,proto3" json:"partition,omitempty"` // Partition identifier SourceIp string `protobuf:"bytes,6,opt,name=source_ip,json=sourceIp,proto3" json:"source_ip,omitempty"` // Host IP where agent is running KvStoreId string `protobuf:"bytes,7,opt,name=kv_store_id,json=kvStoreId,proto3" json:"kv_store_id,omitempty"` // KV store identifier this agent is using TenantId string `protobuf:"bytes,8,opt,name=tenant_id,json=tenantId,proto3" json:"tenant_id,omitempty"` // Tenant UUID for multi-tenant routing TenantSlug string `protobuf:"bytes,9,opt,name=tenant_slug,json=tenantSlug,proto3" json:"tenant_slug,omitempty"` // Tenant slug for NATS subject prefixing unknownFields protoimpl.UnknownFields Learn more about managing compliance generic rules or creating your own custom rules
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Prevent buffer reuse bug in chunking In buildResultsChunks, fix a data corruption bug by ensuring a new backing array is allocated for each payload slice, preventing it from being overwritten by subsequent writes to the shared buffer. pkg/sync/service.go [1135-1169] func (s *SimpleSyncService) buildResultsChunks( allDeviceUpdates []*models.DeviceUpdate, sequence string, ) ([]*proto.ResultsChunk, error) { ... var payloads [][]byte var buf bytes.Buffer deviceCount := 0 flush := func() { if deviceCount == 0 { return } _ = buf.WriteByte(']') - payload := make([]byte, buf.Len()) - copy(payload, buf.Bytes()) + // Use append to a nil slice to force a new allocation and copy, + // preventing the underlying array from being overwritten by buf.Reset(). + payload := append([]byte(nil), buf.Bytes()...) payloads = append(payloads, payload) buf.Reset() deviceCount = 0 } ... [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical bug where buf.Reset() does not clear the underlying array, causing subsequent writes to corrupt previously created payload slices that share the same backing array. This would lead to data corruption in the generated chunks.	High
	✅ Use idiomatic error check for stream Suggestion Impact: Updated the stream receive error handling to use errors.Is(err, io.EOF) instead of comparing err.Error() to "EOF", and added the missing io import. code diff: @@ -20,6 +20,7 @@ "context" "errors" "fmt" + "io" "net" "os" "strings" @@ -999,7 +1000,7 @@ for { chunk, err := stream.Recv() if err != nil { - if err.Error() == "EOF" { + if errors.Is(err, io.EOF) { break } Replace the string comparison err.Error() == "EOF" with the idiomatic errors.Is(err, io.EOF) to reliably detect the end of a gRPC stream. pkg/core/gateways.go [1001-1007] if err != nil { - if err.Error() == "EOF" { + if errors.Is(err, io.EOF) { break } return nil, metadata, fmt.Errorf("error receiving stream chunk: %w", err) } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 6 __ Why: This suggestion is correct and improves code robustness by replacing a brittle string comparison (err.Error() == "EOF") with the idiomatic errors.Is(err, io.EOF) for checking the end of a stream.	Low
	Avoid potential deadlock in config update In UpdateConfig, move the creation of new integrations outside the configMu lock to reduce lock contention and minimize the critical section. pkg/sync/service.go [372-404] func (s *SimpleSyncService) UpdateConfig(newCfg *Config) { if newCfg == nil { return + } + + // Create new integrations outside the lock to avoid potential deadlocks + // and reduce lock contention. + newSources := make(map[string]Integration) + agentID := newCfg.AgentID + gatewayID := newCfg.GatewayID + for name, src := range newCfg.Sources { + if f, ok := s.registry[src.Type]; ok { + newSources[name] = s.createIntegration(s.ctx, src, f, agentID, gatewayID) + } } s.configMu.Lock() defer s.configMu.Unlock() // Check interval changes newDisc := time.Duration(newCfg.DiscoveryInterval) newUpd := time.Duration(newCfg.UpdateInterval) intervalsChanged := (newDisc != s.discoveryInterval) || (newUpd != s.armisUpdateInterval) s.discoveryInterval = newDisc s.armisUpdateInterval = newUpd - // Rebuild integrations even when sources are cleared. - // Use fallback values from the new config being applied. - agentID := newCfg.AgentID - gatewayID := newCfg.GatewayID - s.sources = make(map[string]Integration) - for name, src := range newCfg.Sources { - if f, ok := s.registry[src.Type]; ok { - s.sources[name] = s.createIntegration(s.ctx, src, f, agentID, gatewayID) - } - } + + // Atomically swap the integrations + s.sources = newSources + if intervalsChanged { select { case s.reloadChan <- struct{}{}: default: s.logger.Debug().Msg("Reload channel full; skipping reload trigger") } } s.config = *newCfg } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 5 __ Why: The suggestion's reasoning about a specific deadlock path is incorrect, as createIntegration does not call GetEffectiveDiscoveryInterval. However, the proposed change correctly applies a best practice of minimizing lock duration by performing expensive operations (integration creation) outside the critical section, which reduces lock contention and improves maintainability.	Low
	Use a valid base context Replace the call to context.WithoutCancel with context.Background() to create a valid base context for the background goroutine, preventing potential compilation errors. pkg/core/gateways.go [617-619] -// Run in a separate goroutine to not block the main status report flow. -// Create a detached context but preserve trace information -detachedCtx := context.WithoutCancel(ctx) +detachedCtx := context.Background() s.registerServiceOrCoreDevice(detachedCtx, req.GatewayId, req.Partition, normalizedIP, services, now) [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 5 __ Why: The suggestion correctly identifies that context.WithoutCancel is a new function and might not be available, proposing context.Background() as a fix, which is a valid approach to detach a background task's lifecycle.	Low
General	✅ Bubble enrollment-pending error Suggestion Impact: Removed the special-case block that returned nil when ensureGatewayEnrolled() returned errGatewayNotEnrolled, causing the error to be propagated upward (and added the explanatory comment). code diff: @@ -918,10 +918,8 @@ return err } + // Propagate enrollment-pending so Start() can defer bootstrap if err := s.ensureGatewayEnrolled(ctx); err != nil { - if errors.Is(err, errGatewayNotEnrolled) { - return nil - } return err } In bootstrapGatewayConfig, propagate the errGatewayNotEnrolled error instead of swallowing it, allowing the calling Start function to correctly handle the pending enrollment state. pkg/sync/service.go [912-929] func (s *SimpleSyncService) bootstrapGatewayConfig(ctx context.Context) error { if s.gatewayClient == nil { return nil } if err := s.ensureGatewayConnected(ctx); err != nil { return err } + // Propagate enrollment-pending so Start() can defer bootstrap if err := s.ensureGatewayEnrolled(ctx); err != nil { - if errors.Is(err, errGatewayNotEnrolled) { - return nil - } return err } return s.fetchAndApplyConfig(ctx) } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 8 __ Why: The suggestion correctly points out that swallowing errGatewayNotEnrolled breaks the intended logic in the Start function, which relies on this error to defer configuration bootstrapping. Propagating the error is critical for the correct startup behavior.	Medium
	Remove redundant tenant fields from message Remove the redundant TenantId and TenantSlug fields from the GatewayServiceStatus message, as they are already present in parent messages. proto/monitoring.pb.go [1113-1130] // GatewayServiceStatus represents a single service status pushed by an agent. type GatewayServiceStatus struct { state protoimpl.MessageState `protogen:"open.v1"` ServiceName string `protobuf:"bytes,1,opt,name=service_name,json=serviceName,proto3" json:"service_name,omitempty"` Available bool `protobuf:"varint,2,opt,name=available,proto3" json:"available,omitempty"` Message []byte `protobuf:"bytes,3,opt,name=message,proto3" json:"message,omitempty"` ServiceType string `protobuf:"bytes,4,opt,name=service_type,json=serviceType,proto3" json:"service_type,omitempty"` ResponseTime int64 `protobuf:"varint,5,opt,name=response_time,json=responseTime,proto3" json:"response_time,omitempty"` AgentId string `protobuf:"bytes,6,opt,name=agent_id,json=agentId,proto3" json:"agent_id,omitempty"` // Agent ID for traceability GatewayId string `protobuf:"bytes,7,opt,name=gateway_id,json=gatewayId,proto3" json:"gateway_id,omitempty"` // Gateway ID for traceability Partition string `protobuf:"bytes,8,opt,name=partition,proto3" json:"partition,omitempty"` // Partition identifier Source string `protobuf:"bytes,9,opt,name=source,proto3" json:"source,omitempty"` // Source of the message: "status" or "results" KvStoreId string `protobuf:"bytes,10,opt,name=kv_store_id,json=kvStoreId,proto3" json:"kv_store_id,omitempty"` // KV store identifier this service is using - TenantId string `protobuf:"bytes,11,opt,name=tenant_id,json=tenantId,proto3" json:"tenant_id,omitempty"` // Tenant UUID for multi-tenant routing - TenantSlug string `protobuf:"bytes,12,opt,name=tenant_slug,json=tenantSlug,proto3" json:"tenant_slug,omitempty"` // Tenant slug for NATS subject prefixing unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies data redundancy in the Protobuf message design, which impacts payload size and data consistency, and proposes a valid improvement.	Medium
	Regenerate Protobuf Go file Ensure the monitoring.pb.go file is not manually edited by regenerating it from the source .proto file. proto/monitoring.pb.go [827-840] -type GatewayStatusRequest struct { - state protoimpl.MessageState `protogen:"open.v1"` - Services []*GatewayServiceStatus `protobuf:"bytes,1,rep,name=services,proto3" json:"services,omitempty"` - GatewayId string `protobuf:"bytes,2,opt,name=gateway_id,json=gatewayId,proto3" json:"gateway_id,omitempty"` // Gateway receiving this status - AgentId string `protobuf:"bytes,3,opt,name=agent_id,json=agentId,proto3" json:"agent_id,omitempty"` // Agent sending this status - Timestamp int64 `protobuf:"varint,4,opt,name=timestamp,proto3" json:"timestamp,omitempty"` - Partition string `protobuf:"bytes,5,opt,name=partition,proto3" json:"partition,omitempty"` // Partition identifier - SourceIp string `protobuf:"bytes,6,opt,name=source_ip,json=sourceIp,proto3" json:"source_ip,omitempty"` // Host IP where agent is running - KvStoreId string `protobuf:"bytes,7,opt,name=kv_store_id,json=kvStoreId,proto3" json:"kv_store_id,omitempty"` // KV store identifier this agent is using - TenantId string `protobuf:"bytes,8,opt,name=tenant_id,json=tenantId,proto3" json:"tenant_id,omitempty"` // Tenant UUID for multi-tenant routing - TenantSlug string `protobuf:"bytes,9,opt,name=tenant_slug,json=tenantSlug,proto3" json:"tenant_slug,omitempty"` // Tenant slug for NATS subject prefixing - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache -} +# no manual edits; regenerate entire file via protoc [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 7 __ Why: The suggestion provides a critical best practice for working with generated code, correctly advising to modify the source .proto file rather than the output, which is what the PR author has already done.	Medium
Update

[qodo-code-review]

CI Feedback 🧐

(Feedback updated until commit 9b43388)

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: build

Failed stage: Configure SRQL fixture database for tests [❌]

Failed test name: ""

Failure summary: The action failed because a required secret for TLS verification was missing: - The job exited with code 1 after printing: SRQL_TEST_DATABASE_CA_CERT secret must be configured to verify SRQL fixture TLS. (log line 671). - The environment shows SRQL_TEST_DATABASE_CA_CERT: is empty, so the workflow cannot verify the SRQL test fixture database TLS connection and aborts.

Relevant error logs: 1: Runner name: 'arc-runner-set-2tp2m-runner-k4cvn' 2: Runner group name: 'Default' ... 174: �[36;1mif command -v apt-get >/dev/null 2>&1; then�[0m 175: �[36;1m sudo apt-get update�[0m 176: �[36;1m sudo apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler cmake flex bison�[0m 177: �[36;1melif command -v dnf >/dev/null 2>&1; then�[0m 178: �[36;1m sudo dnf install -y gcc gcc-c++ make openssl-devel protobuf-compiler cmake flex bison�[0m 179: �[36;1melif command -v yum >/dev/null 2>&1; then�[0m 180: �[36;1m sudo yum install -y gcc gcc-c++ make openssl-devel protobuf-compiler cmake flex bison�[0m 181: �[36;1melif command -v microdnf >/dev/null 2>&1; then�[0m 182: �[36;1m sudo microdnf install -y gcc gcc-c++ make openssl-devel protobuf-compiler cmake flex bison�[0m 183: �[36;1melse�[0m 184: �[36;1m echo "Unsupported package manager; please install gcc, g++ (or clang), make, OpenSSL headers, pkg-config, and protoc manually." >&2�[0m 185: �[36;1m exit 1�[0m 186: �[36;1mfi�[0m 187: �[36;1m�[0m 188: �[36;1mensure_pkg_config�[0m 189: �[36;1mprotoc --version || (echo "protoc installation failed" && exit 1)�[0m 190: shell: /usr/bin/bash -e {0} ... 351: shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 352: env: 353: BUILDBUDDY_ORG_API_KEY: *** 354: SRQL_TEST_DATABASE_URL: *** 355: SRQL_TEST_ADMIN_URL: *** 356: SRQL_TEST_DATABASE_CA_CERT: 357: DOCKERHUB_USERNAME: *** 358: DOCKERHUB_TOKEN: *** 359: TEST_CNPG_DATABASE: serviceradar_web_ng_test 360: INSTALL_DIR_FOR_OTP: /home/runner/_work/_temp/.setup-beam/otp 361: INSTALL_DIR_FOR_ELIXIR: /home/runner/_work/_temp/.setup-beam/elixir 362: ##[endgroup] 363: ##[group]Run : install rustup if needed 364: �[36;1m: install rustup if needed�[0m 365: �[36;1mif ! command -v rustup &>/dev/null; then�[0m 366: �[36;1m curl --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused --location --silent --show-error --fail https://sh.rustup.rs | sh -s -- --default-toolchain none -y�[0m 367: �[36;1m echo "$CARGO_HOME/bin" >> $GITHUB_PATH�[0m ... 507: shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 508: env: 509: BUILDBUDDY_ORG_API_KEY: *** 510: SRQL_TEST_DATABASE_URL: *** 511: SRQL_TEST_ADMIN_URL: *** 512: SRQL_TEST_DATABASE_CA_CERT: 513: DOCKERHUB_USERNAME: *** 514: DOCKERHUB_TOKEN: *** 515: TEST_CNPG_DATABASE: serviceradar_web_ng_test 516: INSTALL_DIR_FOR_OTP: /home/runner/_work/_temp/.setup-beam/otp 517: INSTALL_DIR_FOR_ELIXIR: /home/runner/_work/_temp/.setup-beam/elixir 518: CARGO_HOME: /home/runner/.cargo 519: CARGO_INCREMENTAL: 0 520: CARGO_TERM_COLOR: always 521: ##[endgroup] 522: ##[group]Run : work around spurious network errors in curl 8.0 523: �[36;1m: work around spurious network errors in curl 8.0�[0m 524: �[36;1m# https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/timeout.20investigation�[0m ... 575: SRQL_TEST_DATABASE_CA_CERT: 576: DOCKERHUB_USERNAME: *** 577: DOCKERHUB_TOKEN: *** 578: TEST_CNPG_DATABASE: serviceradar_web_ng_test 579: INSTALL_DIR_FOR_OTP: /home/runner/_work/_temp/.setup-beam/otp 580: INSTALL_DIR_FOR_ELIXIR: /home/runner/_work/_temp/.setup-beam/elixir 581: CARGO_HOME: /home/runner/.cargo 582: CARGO_INCREMENTAL: 0 583: CARGO_TERM_COLOR: always 584: ##[endgroup] 585: Attempting to download 1.x... 586: Acquiring v1.27.0 from https://github.com/bazelbuild/bazelisk/releases/download/v1.27.0/bazelisk-linux-amd64 587: Adding to the cache ... 588: Successfully cached bazelisk to /home/runner/_work/_tool/bazelisk/1.27.0/x64 589: Added bazelisk to the path 590: ##[warning]Failed to restore: Cache service responded with 400 591: Restored bazelisk cache dir @ /home/runner/.cache/bazelisk ... 657: env: 658: BUILDBUDDY_ORG_API_KEY: *** 659: SRQL_TEST_DATABASE_URL: *** 660: SRQL_TEST_ADMIN_URL: *** 661: SRQL_TEST_DATABASE_CA_CERT: 662: DOCKERHUB_USERNAME: *** 663: DOCKERHUB_TOKEN: *** 664: TEST_CNPG_DATABASE: serviceradar_web_ng_test 665: INSTALL_DIR_FOR_OTP: /home/runner/_work/_temp/.setup-beam/otp 666: INSTALL_DIR_FOR_ELIXIR: /home/runner/_work/_temp/.setup-beam/elixir 667: CARGO_HOME: /home/runner/.cargo 668: CARGO_INCREMENTAL: 0 669: CARGO_TERM_COLOR: always 670: ##[endgroup] 671: SRQL_TEST_DATABASE_CA_CERT secret must be configured to verify SRQL fixture TLS. 672: ##[error]Process completed with exit code 1. 673: Post job cleanup.