🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform with Datadog Observability

An enterprise-grade, multi-cluster continuous integration and progressive delivery platform purpose-built for TIBCO BusinessWorks™ Container Edition (BWCE 2.9.2 / 2.10.0) microservices on Red Hat OpenShift 4.20+. It integrates Jenkins Configuration as Code (JCasC), Dynamic Git Parameter Plugin Dropdowns, Multi-Remote SCM, ArgoCD 3.5 Multi-Cluster GitOps, and Datadog Full-Stack Observability (APM, DogStatsD, CI Visibility, Logs, and Argo Rollouts Metric Analysis).

Important

⚠️ Architectural Blueprint & Sovereign Environment Notice

Non-Production Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. It has not been tested or executed in a live production environment.
Sovereign & Air-Gapped Boilerplate: This project is engineered as a deterministic, version-controlled Infrastructure-as-Code (IaC) template that can be safely used as a standardized boilerplate in enterprise environments where Agentic AI / LLMs are prohibited due to data sovereignty, compliance regulations, intellectual property protection, or air-gapped security policies.
Enterprise Hardening: Platform engineers must review, adapt, and validate all cluster endpoints, TLS certificates, HashiCorp Vault configurations, Datadog API keys, and RBAC quotas prior to production adoption.
AI Model Attribution & Review: Originally generated using Gemini 3.7 Flash with Antigravity, subsequently audited, verified, and hardened using Gemini 3.8 Flash.
Origin & Real-World Heritage: This blueprint is based on nubenetes/jenkins-git-parameter and nubenetes/jenkins-git-parameter-global-vars, drawing directly from the author's personal hands-on experience designing and operating enterprise integration platforms with these exact requirements (engineered just before agentic AI became widespread).

Tip

🔗 Companion Repository (Single Source of Truth)

This orchestration platform works in tandem with its centralized configuration repository:
🌐 nubenetes/jenkins-git-parameter-bwce-global-vars
(Houses multi-cluster environment topologies, externalized .substvar BWCE profiles, Helm values, and Vault secret references)

🗺️ Quick Navigation Map

This enterprise platform decouples application CI pipelines from centralized multi-cluster GitOps configurations across two companion repositories. Use this map to navigate the architecture blueprint, deployment modules, and multimedia references:

🧭 Repository Architecture Blueprint

jenkins-git-parameter-bwce/              # 🚀 CI/CD & GitOps Orchestration Platform
├── 📁 jcasc/                            # Jenkins Configuration as Code (JCasC 2.492.2 LTS)
│   ├── 📄 jenkins.yaml                  # Master JCasC (security, credentials, agents, Datadog)
│   └── 📄 plugins.txt                   # Automated plugin inventory (git-parameter, datadog, etc.)
├── 📁 jobdsl/                           # Programmatic pipeline definitions (Job DSL API)
│   ├── 📄 00_seed_job.groovy            # Seed job orchestrator scanning for pipelines
│   ├── 📄 01_ci_build_bwce.groovy       # CI pipeline (build, test, containerize, cosign, sbom)
│   └── 📄 02_cd_release_bwce.groovy     # CD orchestrator triggering ArgoCD deployments
├── 📁 shared-library/                   # Jenkins pipeline shared library (declarative steps)
├── 📁 argocd-apps/                      # ArgoCD Application & ApplicationSet manifests
├── 📁 helm/                             # Helm charts for Jenkins, Vault, and BWCE apps
└── 📁 security/                         # Cosign keys, Trivy configs, and ESO Vault stores

jenkins-git-parameter-bwce-global-vars/  # 🌐 Centralized Configuration SSOT
├── 📁 environments/                     # Environment definitions (dev.yaml, staging.yaml, prod.yaml)
├── 📁 clusters/                         # OpenShift cluster topologies (DEV, STG, PROD)
├── 📁 apps/                             # Application catalog & Datadog unified service tags
├── 📁 helm-values/                      # Environment-specific Helm values & .substvar mappings
└── 📁 secrets/                          # External Secrets Operator (ESO) Vault sync definitions

🎬 AI-Generated Multimedia Series (YouTube)

This repository is accompanied by an educational video masterclass and technical shorts synthesized with Gemini NotebookLM based directly on the architectural patterns, multi-cluster GitOps pipelines, and enterprise security controls from this project. All videos are freely accessible on YouTube on the @nubenetes channel.

Note

Multilingual Learning Experience:
Content features native spoken audio in English 🇺🇸, with automated YouTube closed captions (CC) translated into Spanish 🇪🇸 and 20+ languages for global engineering teams.

📽️ Full-Length Technical Deep Dives (Architecture Masterclasses)

#	Video Guide Title	Engineering Domain & Core Architecture	Duration	Direct Link
01	TIBCO BWCE Modernization with jenkins git parameter	Cloud-Native Modernization & CFS Quota Fix 12-Factor .substvar profile externalization, removing CPU limits & Datadog APM	`9:57`	▶️ Watch
02	DevSecOps Fortress Blueprint	Zero-Trust DevSecOps & Supply Chain Security SLSA Level 3, Cosign image signing, CycloneDX SBOM & HashiCorp Vault secrets	`7:42`	▶️ Watch
03	Multi Repo CI CD Puzzle	Multi-Repo Decoupling & ArgoCD GitOps Solving the SCM render paradox, separating CI code from global-vars config SSOT	`7:32`	▶️ Watch

⚡ Video Shorts Matrix

#	Short Title	Architectural Domain & Focus	Duration	Action
01	How Canary Deployments Auto Rollback	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:28`	▶️ Watch
02	Why CPU Limits Freeze TIBCO BWCE	Linux CFS Kernel Quota Throttling Eliminating pod CPU limits on 64-thread JVMs & managing quota via ResourceQuotas	`1:32`	▶️ Watch

For complete technical summaries, topic breakdowns, and direct studio links, see Section: Video Walkthroughs & Architecture References.

📑 Table of Contents

Quick Navigation Map
AI-Generated Multimedia Series (YouTube)
Executive Summary & Architecture Overview
TIBCO BWCE Cloud-Native Best Practices
Datadog Full-Stack Observability Architecture
Multi-Repository Git Parameter CI/CD Patterns
- Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)
- Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator
Enterprise Security & Supply Chain Integrity
Repository Structure
Quick Start & Deployment
Decommissioning & Reinstallation
Video Walkthroughs & Architecture References (YouTube)
References & Standards

Executive Summary & Architecture Overview

This platform delivers an automated, governed, multi-cluster CI/CD and GitOps delivery pipeline for enterprise TIBCO BWCE workloads across three OpenShift clusters (DEV, STAGING, PROD).

🗺️ Click to expand: End-to-End Multi-Cluster Platform Topology Diagram

flowchart TB
    subgraph DevSystems ["1. Developer & Portals"]
        Dev["Developer / Ops"]
        Backstage["Backstage IDP<br/>Service Catalog"]
        ITSM["Jira Service Mgmt<br/>ServiceNow CMDB"]
    end

    subgraph OCP_DEV ["Cluster 1: OCP DEV"]
        direction TB
        subgraph JenkinsPlatform ["Jenkins Control Plane"]
            Master["Jenkins Master<br/>2.492.2 LTS"]
            Seed["00-Seed-Job<br/>Orchestrator"]
            CIJob["01-CI-Build<br/>BWCE App Git"]
            CDJob["02-CD-Release<br/>Global Vars Git"]
        end

        subgraph Agents ["Agent Pods"]
            BwceAgent["bwce-builder<br/>EAR & Tests"]
            GitOpsAgent["argocd-gitops<br/>Skopeo & Cosign"]
        end

        subgraph OCPDevRegistry ["DEV Registry"]
            DevReg["Internal Registry<br/>dev-bwce"]
        end

        subgraph ArgoCDMaster ["ArgoCD 3.5"]
            ArgoServer["ArgoCD Server<br/>ApplicationSets"]
        end

        subgraph DatadogAgentStack ["Datadog Stack"]
            DDAgent["Datadog Agent<br/>APM:8126 | DSD:8125"]
            DDCluster["Cluster Agent<br/>Metrics Provider"]
        end

        DevApps["BWCE DEV Apps<br/>dev-bwce"]
    end

    subgraph OCP_STG ["Cluster 2: OCP STAGING"]
        StgReg["STG Registry"]
        StgApps["BWCE STAGING Apps<br/>staging-bwce"]
    end

    subgraph OCP_PRD ["Cluster 3: OCP PROD"]
        PrdReg["PROD Registry"]
        PrdApps["BWCE PROD Apps<br/>prod-bwce"]
        RolloutCtrl["Argo Rollouts<br/>Canary Controller"]
    end

    subgraph DatadogCloud ["Datadog Cloud"]
        DDCIVis["CI Visibility<br/>Pipeline Spans"]
        DDAPM["APM Tracing<br/>Live Profiling"]
        DDDash["Dashboards<br/>& Monitors"]
    end

    Dev -->|"Selects Branch"| CIJob
    Dev -->|"Selects Config"| CDJob
    Backstage -->|"REST API"| CDJob
    ITSM -->|"Webhook"| CDJob

    CIJob -->|"Launches"| BwceAgent
    BwceAgent -->|"Pushes Image"| DevReg
    CIJob -->|"Triggers CD"| CDJob

    CDJob -->|"Launches"| GitOpsAgent
    GitOpsAgent -->|"Skopeo Copy"| StgReg
    GitOpsAgent -->|"Skopeo Copy"| PrdReg
    GitOpsAgent -->|"Sync & Health"| ArgoServer

    ArgoServer -->|"GitOps"| DevApps
    ArgoServer -->|"GitOps"| StgApps
    ArgoServer -->|"Sync Waves"| PrdApps

    Master -.->|"CI Spans"| DDAgent
    DevApps -.->|"APM & Metrics"| DDAgent
    StgApps -.->|"APM & Metrics"| DDAgent
    PrdApps -.->|"APM & Metrics"| DDAgent
    DDAgent -->|"HTTPS"| DatadogCloud
    RolloutCtrl -.->|"Query Metrics"| DatadogCloud

📋 Architectural Breakdown & Workflow Steps:

Developer & External Ingestion: Developers and ITSM platforms select TIBCO BWCE source branches and global configuration tags via Jenkins UI parameters.
Jenkins Controller (DEV Cluster): JCasC bootstraps the Seed Job, provisioning CI build pipelines and CD release orchestrators via Job DSL.
Pipeline 01 (CI Build): Ephemeral bwce-builder agent compiles .ear archives via bw6-maven-plugin, executes BWUnit tests, layers the EAR onto tibco/bwce:2.9.2, and triggers downstream CD.
Pipeline 02 (CD Release Orchestrator): Ephemeral agent promotes BWCE images via Skopeo, updates .substvar profile bindings in jenkins-git-parameter-bwce-global-vars, and calls argoAppSync.
Datadog Full-Stack Observability: Datadog Cluster Agent and APM tracing (dd-java-agent.jar) provide continuous performance monitoring across DEV, STAGING, and PROD.

💡 Architectural Summary & Conclusion: This topology modernizes enterprise TIBCO BWCE deployments by standardizing container compilation, separating CI from CD release promotion, and enabling multi-cluster OpenShift delivery with automated Datadog tracing.

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via `.substvar`

In enterprise TIBCO BWCE implementations, building environment-specific EAR files violates 12-Factor principles. Instead:

Build Once: A single immutable .ear artifact (tibco-bwce-order-service_2.1.0.ear) is compiled during CI and packaged into the container image (FROM tibco/bwce:2.9.2).
Externalize Profiles: Configuration tokens (DEV.substvar, STAGING.substvar, PROD.substvar) reside in the Single Source of Truth configuration repository: jenkins-git-parameter-bwce-global-vars.
Runtime Injection: The runtime profile is selected via BW_PROFILE environment variable or ConfigMap volume mount at container startup.

⚙️ Click to expand: TIBCO BWCE Profile Externalization Flow Diagram

flowchart LR
    subgraph BuildTime ["1. CI Packaging"]
        Source[".bwp Sources"] --> Maven["Maven Plugin"]
        Maven --> EAR["Single EAR"]
        EAR --> BaseImage["tibco/bwce:2.9.2"]
        BaseImage --> Image["Container Image<br/>(Immutable)"]
    end

    subgraph GitOpsSSOT ["2. Global Vars SSOT"]
        DevVars["DEV.substvar<br/>(dev.yaml)"]
        StgVars["STAGING.substvar<br/>(staging.yaml)"]
        PrdVars["PROD.substvar<br/>(prod.yaml)"]
    end

    subgraph RuntimeDeploy ["3. OpenShift Runtime"]
        Image --> OCPDev["DEV Pod<br/>Profile: DEV"]
        Image --> OCPStg["STG Pod<br/>Profile: STG"]
        Image --> OCPPrd["PROD Pod<br/>Profile: PROD"]

        DevVars -.->|"ConfigMap"| OCPDev
        StgVars -.->|"ConfigMap"| OCPStg
        PrdVars -.->|"ConfigMap"| OCPPrd
    end

📋 Profile Externalization Mechanics:

Build-Time EAR Immutability: The .ear archive contains all default process schemas, modules, and .substvar profile definitions.
Single Container Image: Layering the compiled EAR onto tibco/bwce:2.9.2 produces an immutable container image promoted across all environments.
Runtime Profile Binding: OpenShift deployment manifests inject BW_PROFILE: DEV.substvar, STAGING.substvar, or PROD.substvar at pod startup.

💡 Architectural Summary & Conclusion: Strict profile externalization achieves 12-Factor parity for TIBCO BWCE. Building the EAR once and externalizing environment variables into .substvar profiles prevents environment-specific re-compilation.

2. Engine Sizing & Performance Tuning

The BusinessWorks Container Edition runtime engine requires deliberate sizing parameters adapted to container cgroups:

Environment	Replicas	CPU Request (Burstable)	Memory Req / Limit	`BW_ENGINE_THREADCOUNT`	`BW_STEP_FLOWLIMIT`	`BW_LOGLEVEL`
DEV	2	`250m` (No CPU Limit)	512Mi / 1024Mi	`16`	`50`	`DEBUG`
STAGING	3	`500m` (No CPU Limit)	768Mi / 1536Mi	`32`	`100`	`INFO`
PROD	6	`1000m` (No CPU Limit)	1024Mi / 2048Mi	`64`	`250`	`WARN`

Tip

💡 Why CPU Limits are Omitted at the Pod Level (Preventing Linux CFS Bandwidth Throttling)

The CFS Quota Problem in Multi-Threaded Runtimes: Setting resources.limits.cpu instructs the Linux kernel Completely Fair Scheduler (CFS) to enforce hard bandwidth quotas (cfs_quota_us over 100ms periods). Because TIBCO BWCE and JVM garbage collection are highly concurrent (e.g. 16–64 engine threads), parallel thread execution can consume the quota in milliseconds, causing the container to be hard-throttled for the remainder of the period. This induces artificial p99 latency spikes and timeouts even when the OpenShift worker node has surplus CPU capacity.
Namespace-Level ResourceQuota Governance: Instead of per-pod CPU limits, capacity governance and noisy-neighbor protection are enforced at the OpenShift Project/Namespace boundary using security/openshift-namespace-resource-quota.yaml (ResourceQuota for aggregate requests.cpu, requests.memory, and limits.memory).
Deterministic Memory Limits: Unlike CPU (which is compressible), Memory is strictly capped with limits.memory to prevent out-of-memory cascading to neighbor pods.

BW_ENGINE_THREADCOUNT: Specifies the number of engine worker threads executing process instances concurrently.
BW_STEP_FLOWLIMIT: Prevents unbounded memory growth during traffic bursts by capping active in-memory process transitions.
BW_CONTAINER_SHUTDOWN_TIMEOUT_SECONDS: Set to 30s (DEV/STAGING) and 45s (PROD) for clean process draining upon SIGTERM.
JVM Options: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent ensuring GC predictability inside Linux container cgroups.

3. OpenShift Hardening & Health Probing

Security Context Constraints: Complies with OpenShift restricted-v2 SCC (non-root uid: 1001, dropped capabilities ALL, read-only root filesystems).
Probes:
- Liveness Probe: HTTP GET http://:8090/health (Initial delay: 45s, Period: 15s, Timeout: 5s).
- Readiness Probe: HTTP GET http://:8090/health (Initial delay: 20s, Period: 10s, Timeout: 3s).

Datadog Full-Stack Observability Architecture

Replaces OpenTelemetry and Grafana with an enterprise Datadog observability stack.

📊 Click to expand: Datadog Full-Stack Observability Architecture Diagram

flowchart TB
    subgraph Sources ["1. OpenShift Workloads"]
        direction TB
        Jenkins["Jenkins Master<br/>(Datadog Plugin)"]
        BWCEApp["TIBCO BWCE App<br/>(dd-java-agent)"]
        RolloutCtrl["Argo Rollouts<br/>(Canary Engine)"]
    end

    subgraph DaemonSetLayer ["2. Node Agent Layer"]
        direction TB
        DDAgent["Datadog Agent<br/>(DaemonSet)<br/>──────────────<br/>APM Port: 8126<br/>StatsD Port: 8125<br/>Metrics: 8090<br/>Logs: JSON"]
    end

    subgraph CloudPlatform ["3. Datadog Cloud Platform"]
        direction TB
        CIVisibility["CI Visibility<br/>Pipeline Spans<br/>Queue Metrics"]
        APMTrace["APM Tracing<br/>HTTP & JMS<br/>Distributed Spans"]
        OpenMetrics["Engine Metrics<br/>Thread Pools<br/>JVM GC Pauses"]
        Dashboards["Dashboards<br/>• Jenkins CI<br/>• BWCE Engine<br/>• ArgoCD Sync"]
        Monitors["Alert Monitors<br/>• 5xx > 1.0%<br/>• p99 > 500ms<br/>• Queue Spikes"]
    end

    Jenkins -->|"CI Spans"| DDAgent
    BWCEApp -->|"APM: 8126"| DDAgent
    BWCEApp -->|"Metrics: 8090"| DDAgent
    BWCEApp -->|"Logs"| DDAgent

    DDAgent -->|"HTTPS"| CIVisibility
    DDAgent -->|"HTTPS"| APMTrace
    DDAgent -->|"HTTPS"| OpenMetrics

    CIVisibility --> Dashboards
    APMTrace --> Dashboards
    OpenMetrics --> Dashboards

    CIVisibility --> Monitors
    APMTrace --> Monitors
    OpenMetrics --> Monitors

    RolloutCtrl -.->|"Query SLA"| APMTrace

📋 Datadog Telemetry Data Flow:

JVM APM Instrumentation: Java agent (dd-java-agent.jar) instruments BWCE runtime processes, capturing REST endpoints, JDBC queries, and JMS activities.
DogStatsD Metrics: Workload pods emit custom business metrics and engine throughput statistics over UDP to the Datadog Cluster Agent.
Unified Dashboards: SRE teams visualize distributed traces, service dependencies, and live log streams in unified Datadog APM dashboards.

💡 Architectural Summary & Conclusion: Full-stack Datadog instrumentation provides deep visibility into complex enterprise integration flows, enabling instant bottleneck diagnosis and automated SLA monitoring.

1. Jenkins CI/CD Visibility Plugin

The official Datadog Jenkins Plugin (datadog:5.9.0) is configured via JCasC (jcasc/jenkins-jcasc.yaml):

Automatically correlates pipeline builds, stage execution times, agent queue wait times, and test results.
Injects Datadog Trace IDs (x-datadog-trace-id) across pipeline steps.
Global tags: env:dev, team:integration-platform, tech:tibco-bwce, cluster:ocp-dev.

2. TIBCO BWCE Java APM Tracer & DogStatsD

Java Tracer: The Datadog Java APM agent (dd-java-agent.jar) is injected into the container image and attached to the BWCE engine via:

BW_JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -javaagent:/opt/datadog/dd-java-agent.jar -Ddd.service=tibco-bwce-order-service -Ddd.logs.injection=true -Ddd.profiling.enabled=true"

OpenMetrics Autodiscovery: Pod annotations configure the Datadog Agent to scrape BWCE management metrics from port 8090:

annotations:
  ad.datadoghq.com/tibco-bwce-order-service.logs: '[{"source": "tibco-bwce", "service": "tibco-bwce-order-service"}]'
  ad.datadoghq.com/tibco-bwce-order-service.check_names: '["openmetrics"]'
  ad.datadoghq.com/tibco-bwce-order-service.init_configs: '[{}]'
  ad.datadoghq.com/tibco-bwce-order-service.instances: |
    [{"openmetrics_endpoint": "http://%%host%%:8090/metrics", "namespace": "tibco_bwce", "metrics": [".*"]}]

3. Datadog Live Dashboards & Monitors

Preconfigured Datadog JSON dashboards and alert monitors located in observability/:

Jenkins CI Visibility: observability/dashboards/datadog-jenkins-ci-visibility.json
BWCE Engine Performance: observability/dashboards/datadog-bwce-runtime-performance.json
ArgoCD GitOps Sync: observability/dashboards/datadog-argocd-gitops-sync.json
Automated Monitors: observability/monitors/datadog-monitors-bwce.yaml (Monitors HTTP 5xx error rate > 1%, p99 latency > 500ms, and build agent queue bottlenecks).

4. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

During canary deployments, Argo Rollouts queries Datadog metrics directly using sample-apps/tibco-bwce-order-service/rollout/analysis-template-datadog.yaml:

Error Rate SLA: Evaluates that 5xx errors $\le 0.1%$ via sum:trace.tibco_bwce_order_service.request.errors{env:prod}.as_count() / sum:trace.tibco_bwce_order_service.request.hits{env:prod}.as_count().
Latency SLA: Evaluates that p99 latency $\le 250 ext{ms}$ via p99:trace.tibco_bwce_order_service.request.duration{env:prod} before traffic step promotion.

🐣 Click to expand: Argo Rollouts Canary Traffic Splitting & Datadog Metric Analysis Diagram

flowchart LR
    Ingress["OpenShift Route<br/>Ingress Traffic"]
    Canary["Canary Pods<br/>10% -> 25% -> 50%<br/>v2.1.0 (New EAR)"]
    Stable["Stable Pods<br/>90% -> 75% -> 50%<br/>v2.0.8 (Stable EAR)"]
    DatadogMetrics["Datadog APM API<br/>5xx <= 0.1%<br/>p99 <= 250ms"]
    RolloutCtrl["Argo Rollouts<br/>Controller"]

    Ingress -->|"Traffic Split"| Canary
    Ingress -->|"Traffic Split"| Stable
    Canary -.->|"APM Metrics"| DatadogMetrics
    DatadogMetrics -.->|"SLA Evaluation"| RolloutCtrl
    RolloutCtrl -->|"Pass: Promote<br/>Fail: Rollback"| Ingress

📋 Progressive Canary Delivery Steps:

Traffic Routing: Argo Rollouts splits OpenShift route traffic, routing 20% to canary BWCE pods and 80% to stable pods.
Datadog SLA Analysis: Automated AnalysisRun queries Datadog metrics (error rate $< 0.5%$ and p95 latency $< 200 ext{ms}$).
Automated Rollout / Rollback: If metrics satisfy SLAs, traffic scales to 100%; if SLAs breach, instant automated rollback redirects traffic to stable pods.

💡 Architectural Summary & Conclusion: Data-driven progressive delivery with Datadog metric analysis eliminates deployment risk for critical TIBCO integration services, ensuring zero downtime and automated fault protection.

Multi-Repository Git Parameter CI/CD Patterns

The Core Problem: Why Standard Pipelines Fail with Multiple Git Repositories

In cloud-native architectures, applications separate source code (tibco-bwce-order-service) from centralized configuration and Helm values (jenkins-git-parameter-bwce-global-vars).

When engineering teams attempt to introduce multiple gitParameter dropdowns into a single Jenkins Pipeline, they encounter a hard blocker where GitSCM appears to only support one repository, or secondary dropdowns fail.

⚠️

Click to expand: Jenkins SCM Lifecycle & Pre-Execution Blindspot Diagram

flowchart TB
    subgraph UI_Phase ["1. Pre-Execution (Master)"]
        direction TB
        User["User opens<br/>Build UI Form"]
        Master["Master reads<br/>Job XML SCM"]
        GitParam["git-parameter<br/>queries remote refs"]
        Dropdown["Renders Dropdown<br/>for Primary Repo"]

        User --> Master --> GitParam --> Dropdown
    end

    subgraph Runtime_Phase ["2. Runtime Phase (Agent)"]
        direction TB
        AllocAgent["Ephemeral Agent<br/>Pod Allocated"]
        RunStage["Pipeline Stage:<br/>checkout(repo-2)"]

        AllocAgent --> RunStage
    end

    Gap["SCM Blindspot:<br/>Dynamic checkouts<br/>run during build and are<br/>invisible at UI render"]

    Dropdown -.-> Gap
    Gap -.-> RunStage

📋 Why Multi-Repo Pipelines Fail for BWCE:

Pre-Execution Render Paradox: Jenkins Master parses gitParameter dropdowns before agent allocation, discovering only repositories statically declared in Job XML.
Dynamic SCM Blindspot: Secondary checkouts for .substvar files executed inside pipeline stages are invisible during initial UI rendering.
The Pattern Solution: Pattern 1 resolves this via multi-remote refspecs in Job DSL, while Pattern 2 separates pipelines into distinct build and release phases.

💡 Architectural Summary & Conclusion: Overcoming the SCM pre-execution blindspot ensures reliable parameter rendering when coordinating BWCE source code with central global configuration repositories.

Master Seed Job & JCasC Automation Architecture

The entire CI/CD platform is self-bootstrapping and self-healing using Jenkins Configuration as Code (JCasC) combined with the Job DSL Plugin.

⚙️ Click to expand: Seed Job & Job DSL Pipeline Provisioning Flow Diagram

flowchart TB
    subgraph JCasC_Phase ["1. Controller Bootstrap (JCasC)"]
        direction TB
        JCasC["jenkins-jcasc.yaml<br/>(Master Config)"] -->|"Registers"| SeedJob["00-Seed-Job<br/>(Platform Seed)"]
    end

    subgraph SCM_Phase ["2. SCM Synchronization"]
        direction TB
        GitRepo["Git Repository<br/>(Platform Code)"]
        SeedJob -->|"Polls H/15 * * * *"| GitRepo
        SeedJob -->|"Evaluates"| JobDSLFiles["Target Scripts<br/>(jobdsl/*.groovy)"]
    end

    subgraph Materialization ["3. Pipeline Materialization"]
        direction TB
        JobDSLFiles -->|"seed-job.groovy"| Folders["Creates Folders:<br/>• CI Pipelines<br/>• CD Orchestrators"]
        JobDSLFiles -->|"pipelines-ci.groovy"| CIPipelines["Generates CI:<br/>• order-service-ci<br/>• customer-api-ci"]
        JobDSLFiles -->|"pipelines-cd.groovy"| CDPipelines["Generates CD:<br/>• release-orchestrator<br/>• hotfix-deploy"]
    end

    Folders -.->|"Parent"| CIPipelines
    Folders -.->|"Parent"| CDPipelines

📋 Pipeline Provisioning Chain:

JCasC Controller Bootstrap: Jenkins initializes and configures 00-Seed-Job-Platform-Orchestrator.
Dynamic Polling: Polls the repository every 15 minutes for updates to Job DSL Groovy scripts.
Pipeline Generation: Materializes folder structures, BWCE CI pipelines (pipelines-ci.groovy), and CD orchestrators (pipelines-cd.groovy).
Automated Pruning: Deletes obsolete jobs automatically when removed from version control.

💡 Architectural Summary & Conclusion: Managing BWCE pipelines as version-controlled Job DSL code eliminates manual configuration drift and enables automated onboarding of enterprise integration services.

1. Why `jobdsl/seed-job.groovy` Only Declares Folders (Separation of Concerns)

In enterprise Jenkins-as-Code implementations, we deliberately apply the Separation of Concerns (SoC) principle:

Taxonomy & Hierarchy Initialization: jobdsl/seed-job.groovy acts as the root taxonomy initializer. It creates and configures the top-level folder containers (01-CI-Build-Pipelines and 02-CD-Release-Orchestrators) along with their descriptions and metadata.
Decoupled Pipeline Generation Logic: The application CI pipelines (jobdsl/pipelines-ci.groovy) and CD release orchestrators (jobdsl/pipelines-cd.groovy) contain iterative business logic (looping over the apps inventory, configuring multi-remote refspecs, Git Parameter dropdowns, log rotation, and Datadog CI visibility). Keeping folder declarations separate prevents monolithic scripts and enables modular additions of new business domains.

2. Managing Folders in Job DSL vs. Static JCasC

While folders can technically be defined directly in jenkins-jcasc.yaml, managing them through Job DSL is the recommended SRE practice for multi-tenant platforms:

Architectural Metric	Static JCasC Management (`jenkins-jcasc.yaml`)	Dynamic Job DSL Management (`jobdsl/seed-job.groovy`)
Creation Lifecycle	Executed strictly upon Jenkins Controller startup or manual JCasC reload.	Evaluated continuously on every Seed Job execution (automated SCM polling or manual trigger).
Orphaned Item Pruning	Does not automatically purge folders or child jobs if deleted from configuration.	Automated Pruning: `removedJobAction('DELETE')` and `removedViewAction('DELETE')` cleanly destroy removed jobs/folders.
Configuration Ownership	Belongs to Controller System Configuration (Platform Admin scope).	Belongs to Version-Controlled GitOps Code (Developer / Application Domain scope).
Zero-Downtime Evolution	Renaming or restructuring folders requires reloading master JCasC.	Restructuring folders is applied dynamically via Git commit without touching master configuration.

3. Step-by-Step Technical Chain & Execution Flow

JCasC Master Bootstrap:
When the Jenkins container starts, jcasc/jenkins-jcasc.yaml configures the master Seed Job:

jobs:
  - script: |
      job('00-Seed-Job-BWCE-Platform-Orchestrator') {
        description('Master Seed Job provisioning TIBCO BWCE Pipelines as Code.')
        scm {
          git {
            remote { url('https://github.com/nubenetes/jenkins-git-parameter-bwce.git') }
            branch('*/main')
          }
        }
        steps {
          jobDsl {
            targets('jobdsl/seed-job.groovy\njobdsl/pipelines-ci.groovy\njobdsl/pipelines-cd.groovy')
            removedJobAction('DELETE')
            removedViewAction('DELETE')
            lookupStrategy('JENKINS_ROOT')
          }
        }
        triggers { scm('H/15 * * * *') }
      }

lookupStrategy('JENKINS_ROOT') Context Resolution:
By setting lookupStrategy('JENKINS_ROOT'), Job DSL resolves all relative job paths from the root of Jenkins. This allows pipelines-ci.groovy to target "01-CI-Build-Pipelines/${app.name}-ci-build" and nest jobs into the folders previously created by seed-job.groovy.
Automated Drift Correction via SCM Polling:
The trigger scm('H/15 * * * *') polls this repository every 15 minutes. When an engineer adds a new microservice to the def apps = [...] inventory in pipelines-ci.groovy and pushes to main, the Seed Job automatically generates the new CI/CD pipelines in Jenkins with zero manual UI configuration.

Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)

Target: Developer Sandboxes & Feature Preview Environments

Dropdown 1: Queries application repository (APP_GIT_REVISION).
Dropdown 2: Queries global configuration repository (GLOBAL_VARS_REVISION).
Multi-remote refspecs configured via Job DSL in jobdsl/pipelines-ci.groovy.

Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator

Target: Enterprise Production Release Pipeline (Recommended)

🧪 Click to expand: Inner-Loop (Pattern 1) vs Outer-Loop (Pattern 2) Coexistence Diagram

flowchart TD
    subgraph InnerLoop ["Pattern 1: Dual-Dropdown"]
        Dev1["Feature Dev"]
        DualJob["Dual-Dropdown Job<br/>App & Config Refs"]
        DevSandbox["Dev Sandbox<br/>dev-bwce"]

        Dev1 -->|"Trigger"| DualJob
        DualJob -->|"Deploy"| DevSandbox
    end

    subgraph OuterLoop ["Pattern 2: Decoupled CI/CD"]
        Dev2["Release Lead"]
        CIJob["01-CI-Build<br/>App Repo"]
        Registry["DEV Registry"]
        CDOrchestrator["02-CD-Orchestrator<br/>Global Vars SSOT"]
        ArgoEngine["ArgoCD Engine"]
        Clusters["Promotion Chain<br/>DEV->STG->PROD"]

        Dev2 -->|"Trigger CI"| CIJob
        CIJob -->|"Push Image"| Registry
        CIJob -->|"Dispatch"| CDOrchestrator
        CDOrchestrator -->|"Sync"| ArgoEngine
        ArgoEngine -->|"Promote"| Clusters
    end

📋 Dual-Pattern Coexistence Mechanics:

Inner-Loop (Pattern 1: Dual-Dropdown): Single pipeline for rapid BWCE developer testing, allowing instant pairing of feature branches with custom .substvar configurations.
Outer-Loop (Pattern 2: Decoupled Hand-off): Standardized enterprise release pipeline with independent CI compilation and governed multi-cluster CD promotion.

💡 Architectural Summary & Conclusion: Supporting both inner-loop and outer-loop patterns provides the agility needed during integration development alongside the governance required for production releases.

🔄 Click to expand: End-to-End Hand-off Sequence Diagram

sequenceDiagram
    autonumber
    actor Dev as Developer / Operator
    participant CI as 01-CI-Build-Pipeline
    participant DevReg as OCP DEV Registry
    participant CD as 02-CD-Release-Orchestrator
    participant StgReg as OCP Staging Registry
    participant PrdReg as OCP Prod Registry
    participant Argo as ArgoCD 3.5 Control Plane
    participant DDog as Datadog Cloud APM

    Dev->>CI: Trigger Build (Select App Branch)
    activate CI
    CI->>CI: Execute BWUnit Mock Tests
    CI->>CI: Package EAR and Build Container
    CI->>CI: Scan with Trivy and Sign with Cosign
    CI->>DevReg: Push image (e.g. 2.1.0-42)
    CI->>DDog: Emit CI Span (ci.bwce.pipeline_completed)
    CI->>CD: Dispatch downstream CD Orchestrator
    deactivate CI

    activate CD
    CD->>Argo: Sync DEV Application (Wave 0 -> Wave 1)
    Argo-->>CD: DEV Health Check Passed (HTTP 200)
    CD->>StgReg: Skopeo copy (DEV -> STAGING)
    CD->>Argo: Sync STAGING Application
    Argo-->>CD: STAGING Integration Suite Passed (100%)
    
    CD->>Dev: Production Approval Gate (Interactive Input)
    Dev-->>CD: Approved by Release Engineer

    CD->>PrdReg: Skopeo copy (STAGING -> PROD)
    CD->>Argo: Sync PROD Application (Argo Rollouts Canary)
    Argo->>DDog: Query Datadog Metric Analysis (p99 < 250ms, 5xx < 0.1%)
    DDog-->>Argo: Metrics Passed (100% Promoted)
    CD->>DDog: Emit Release Metric (cd.bwce.release_completed)
    deactivate CD

📋 Step-by-Step CI/CD Hand-off Sequence:

1. CI Pipeline Trigger: Launches Pipeline 01 with APP_GIT_REVISION.
2. EAR Build & Test: Packages .ear via bw6-maven-plugin, runs BWUnit tests, layers onto tibco/bwce:2.9.2, and pushes to DEV registry.
3. Downstream Hand-off: Triggers Pipeline 02 with IMAGE_TAG and target environment parameters.
4. Promotion & Manifest Commit: Copies image via Skopeo and updates .substvar profile in jenkins-git-parameter-bwce-global-vars.
5. ArgoCD Sync: Invokes argoAppSync to reconcile OpenShift clusters and verify HTTP 200 health.

💡 Architectural Summary & Conclusion: The decoupled CI/CD sequence enforces artifact immutability and ensures that promoted BWCE images are tested and verified before reaching production clusters.

Automated Downstream Hand-off & Environment Targeting for BWCE

In Pattern 2, Pipeline 01 (BWCE CI Build) and Pipeline 02 (CD Release Orchestrator) are connected via an automated downstream hand-off trigger:

🔄 Click to expand: BWCE Downstream Hand-off & Multi-Environment Promotion Flow Diagram

flowchart LR
    subgraph CIStage["1. Pipeline 01: BWCE CI"]
        direction TB
        Dev1["👩‍💻 User"] -->|"Selects Branch & Env"| CI["🏗️ 01-CI-Build<br/>• Packages EAR<br/>• Signs Image<br/>• Pushes to DEV"]
        CI -->|"Trigger CD"| Dispatch["⚡ build job:<br/>02-CD-Orchestrator<br/>Passes: IMAGE_TAG"]
    end

    subgraph CDStage["2. Pipeline 02: CD Orchestrator"]
        direction TB
        Dispatch --> CD["🚀 02-CD-Orchestrator<br/>• Resolves .substvar<br/>• Routes by Env"]
        
        CD -->|"env == 'dev'"| DevTarget["☸️ OpenShift DEV<br/>• gitopsCommit(dev)<br/>• argoAppSync(dev)"]
        CD -->|"env == 'staging'"| StgTarget["☸️ OpenShift STAGING<br/>• skopeoPromote<br/>• argoAppSync(stg)"]
        CD -->|"env == 'prod'"| PrdTarget["☸️ OpenShift PROD<br/>• Approval Gate<br/>• skopeoPromote<br/>• argoAppSync(prd)"]
    end

📋 Downstream Parameter Passing & Execution Logic:

Configured in Job DSL (jobdsl/pipelines-ci.groovy):
- TRIGGER_CD_RELEASE (boolean, default: true): Automatically dispatches downstream release.
- GLOBAL_VARS_BRANCH (string, default: 'main'): Global configuration repository branch passed to Pipeline 02.
- TARGET_ENVIRONMENT (choice: dev, staging, prod, full-promotion-chain): Initial target environment.

Trigger Stage in Pipeline 01 (Jenkinsfile.app-bwce):

stage('Trigger CD Release Orchestrator') {
    when { expression { return params.TRIGGER_CD_RELEASE == true } }
    steps {
        build job: '02-CD-Release-Orchestrators/multi-cluster-release-orchestrator',
            parameters: [
                string(name: 'GLOBAL_VARS_REVISION', value: params.GLOBAL_VARS_BRANCH ?: 'main'),
                string(name: 'APP_NAME', value: env.APP_NAME),
                string(name: 'IMAGE_TAG', value: env.CALCULATED_TAG),
                string(name: 'TARGET_ENVIRONMENT', value: params.TARGET_ENVIRONMENT ?: 'dev'),
                booleanParam(name: 'AUTO_PROMOTE_TO_STAGING', value: true),
                booleanParam(name: 'REQUIRE_PROD_APPROVAL', value: true),
                string(name: 'TRIGGERED_BY', value: 'CI_PIPELINE'),
                string(name: 'CHANGE_REQUEST_ID', value: "CI-BWCE-AUTO-${BUILD_NUMBER}")
            ],
            wait: false
    }
}

Environment-Specific Routing in Pipeline 02 (Jenkinsfile.release-orchestrator):
- Target dev: Binds DEV.substvar profile and triggers argoAppSync on OpenShift DEV.
- Target staging: Promotes container image via Skopeo, binds STAGING.substvar, and synchronizes OpenShift STAGING.
- Target prod: Promotes image to PROD registry, pauses at the interactive Production Approval Gate (input step), binds PROD.substvar, and deploys via ArgoCD canary sync waves on OpenShift PROD.

💡 Architectural Summary & Conclusion: This automated linkage guarantees 'Build Once, Deploy Anywhere' artifact immutability for TIBCO BWCE. The EAR archive is compiled and packaged once in CI, while Pipeline 02 promotes the verified container image across environments by binding the appropriate .substvar profiles.

Enterprise Security & Supply Chain Integrity

1. Cosign Image Signing & SLSA Level 3 Attestation

All container images layered on the BWCE base runtime are cryptographically signed using Sigstore Cosign during CI execution, guaranteeing supply chain integrity before promotion into production.

🔏 Click to expand: Cosign Image Signing & SLSA Level 3 Attestation Flow Diagram

flowchart LR
    CI["CI Build<br/>(Maven BWCE)"]
    Trivy["Trivy Scan<br/>& Syft SBOM"]
    Cosign["Cosign Sign<br/>& Attestation"]
    Registry["OCP DEV<br/>Registry"]
    Policy["OpenShift Policy<br/>Block Unsigned"]

    CI --> Trivy
    Trivy --> Cosign
    Cosign --> Registry
    Registry -.->|"Enforced"| Policy

📋 Supply Chain Security & Signing Chain:

Artifact Packaging: Compiles EAR and builds container image.
Cosign Signing: Ephemeral agent signs the image digest using keys stored in HashiCorp Vault.
Attestation Generation: Attaches CycloneDX SBOM and SLSA provenance to the OpenShift image registry.
Admission Verification: OpenShift Image Signature Policy enforces signature verification at pod admission.

💡 Architectural Summary & Conclusion: SLSA Level 3 supply chain security ensures that only authenticated, signed BWCE container images can be deployed across enterprise clusters.

2. Zero-Trust Secrets with External Secrets Operator & Vault

Plaintext credentials (database passwords, JMS connection tokens, Datadog API keys) are never stored in Git. Instead, the External Secrets Operator (ESO) synchronizes encrypted secrets from HashiCorp Vault directly into OpenShift namespaces.

🔐 Click to expand: External Secrets Operator & Vault Synchronization Flow Diagram

flowchart LR
    Vault["HashiCorp Vault<br/>Secrets & Keys"]
    GlobalVars["Global Vars Repo<br/>ExternalSecret CRD"]
    ESO["External Secrets<br/>Operator (ESO)"]
    K8sSecret["Kubernetes Secret<br/>(In-Memory)"]
    BWCEPod["TIBCO BWCE Pod<br/>(Workload)"]

    Vault -->|"Token Fetch"| ESO
    GlobalVars -->|"Ref Spec"| ESO
    ESO -->|"Sync/Rotate"| K8sSecret
    K8sSecret -->|"Env Mount"| BWCEPod

📋 Secret Management & Dynamic Injection:

Centralized Storage: HashiCorp Vault securely stores backend database credentials, keystores, and API tokens.
External Secrets Operator (ESO): Synchronizes secrets into native Kubernetes Secret resources.
Dynamic Injection: BWCE runtime consumes secrets at startup without hardcoding sensitive data in .substvar files.

💡 Architectural Summary & Conclusion: ESO with HashiCorp Vault eliminates hardcoded passwords from BWCE property files, enabling centralized rotation and enterprise audit compliance.

3. Ephemeral PR Preview Environments via ArgoCD ApplicationSets

Pull Requests targeting application repositories automatically trigger the ArgoCD PR Generator ApplicationSet, creating ephemeral preview namespaces for end-to-end integration testing before merging.

⚡ Click to expand: Ephemeral PR Preview Provisioning Flow Diagram

flowchart LR
    Dev["Developer"]
    PR["GitHub PR #42"]
    AppSet["ArgoCD AppSet<br/>(PR Generator)"]
    PreviewNS["Preview Env<br/>pr-42-preview"]
    Datadog["Datadog<br/>PR Dashboard"]

    Dev -->|"Opens"| PR
    PR -->|"Discovers"| AppSet
    AppSet -->|"Deploys"| PreviewNS
    PreviewNS -->|"DD_ENV"| Datadog

📋 Dynamic Ephemeral Preview Provisioning Flow:

PR Trigger: Opening a pull request triggers Jenkins CI to package an ephemeral BWCE preview image.
ArgoCD Dynamic Sync: Provisions an isolated namespace pr-preview-<id>-bwce on DEV cluster.
Automated Cleanup: Closing or merging the pull request automatically deletes the namespace and temporary resources.

💡 Architectural Summary & Conclusion: Ephemeral preview environments provide isolated sandboxes for validating BWCE business logic and API contracts before merging changes to main branches.

📂 Repository Structure

.
├── config/
│   ├── environments.env              # Pinned versions, domain names, Datadog credentials
│   └── clusters.yaml                 # ConfigMap defining OCP DEV, STAGING, PROD topologies
├── helm/
│   ├── jenkins/                      # Official Jenkins chart values with Datadog plugin
│   ├── observability/                # Datadog Agent Helm values (OpenShift hardened)
│   └── argocd/                       # ArgoCD 3.5 Helm values
├── jcasc/
│   ├── jenkins-jcasc.yaml            # JCasC security matrix, Datadog CI visibility, seed job
│   ├── pod-templates.yaml            # Ephemeral Kubernetes agent pod templates (bwce-builder)
│   └── github-app-credentials.yaml   # GitHub App credential rotation
├── jobdsl/
│   ├── seed-job.groovy               # Master Seed Job definition
│   ├── pipelines-ci.groovy           # Pattern 1 and Pattern 2 CI pipelines
│   └── pipelines-cd.groovy           # CD Release Orchestrator & Hotfix pipelines
├── jenkinsfiles/
│   ├── ci/
│   │   ├── Jenkinsfile.app-bwce
│   │   └── Jenkinsfile.app-bwce-dual-dropdown
│   └── cd/
│       ├── Jenkinsfile.release-orchestrator
│       └── Jenkinsfile.hotfix-deploy
├── shared-library/
│   ├── vars/
│   │   ├── datadogLogEvent.groovy    # Emits Datadog CI spans & DogStatsD metrics
│   │   ├── bwceEarBuild.groovy       # Compiles and packages BWCE EAR
│   │   ├── bwceProfileOverride.groovy# Injects .substvar profile tokens
│   │   ├── argoAppSync.groovy        # ArgoCD 3.5 synchronization
│   │   ├── skopeoPromote.groovy      # Cross-cluster image promotion
│   │   ├── cosignSign.groovy         # Cryptographic image signing (SLSA Level 3)
│   │   ├── sbomGenerate.groovy       # CycloneDX SBOM generation
│   │   └── gitopsCommit.groovy       # Updates Global Vars SSOT repository
│   └── src/com/nubenetes/gitops/
├── sample-apps/
│   ├── tibco-bwce-order-service/     # Complete TIBCO BWCE project + Dockerfile + K8s + Rollout
│   └── tibco-bwce-customer-api/
├── observability/
│   ├── dashboards/                   # Datadog dashboards (Jenkins, BWCE Engine, ArgoCD)
│   └── monitors/                     # Datadog automated alert monitors
├── argocd-apps/                      # ArgoCD Root App-of-Apps & ApplicationSets
├── security/                         # Image signature policy & External Secrets Operator
├── scripts/                          # Automation & setup scripts
├── deploy.sh                         # 1-Click Platform Deployment Script
├── destroy.sh                        # Clean Decommission Script
├── reinstall.sh                      # Full Wipe & Reinstall Script
└── Makefile

🚀 Quick Start & Deployment

1. Prerequisites

Red Hat OpenShift 4.20+ cluster (or Kubernetes 1.31+)
kubectl, oc, and helm v3 CLI tools installed
Datadog Account & API/App Keys

2. Configure Credentials

Set your Datadog credentials in config/environments.env or export environment variables:

export DATADOG_API_KEY="your-datadog-api-key"
export DATADOG_APP_KEY="your-datadog-app-key"

3. Deploy Platform

./deploy.sh
# or
make deploy

4. Access Platform Services

Jenkins Controller: https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal (Default login: admin / admin123!)
ArgoCD UI: https://argocd-server.apps.ocp-dev.nubenetes.internal
Datadog Dashboard: https://app.datadoghq.eu

Decommissioning & Reinstallation

Clean Decommission

./destroy.sh
# or
make destroy

Full Reinstallation

./reinstall.sh
# or
make reinstall

Video Walkthroughs & Architecture References (YouTube)

Architectural deep dives, video walkthroughs, and technical shorts for jenkins-git-parameter-bwce, TIBCO BWCE modernization, and Datadog GitOps progressive delivery on OpenShift 4.20+ are hosted on the Nubenetes YouTube Channel (@nubenetes).

📂 Full-Length Technical Deep Dives (Architecture Masterclasses)

1. TIBCO BWCE Modernization with jenkins git parameter

🔗 Direct Link: https://www.youtube.com/watch?v=PaS5pkjBIXU
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 9:57
🏷️ Engineering Domain: Cloud-Native Modernization, 12-Factor Profile Externalization & Linux CFS Quota Fix
📝 Technical Overview: Architectural blueprint for modernizing legacy TIBCO BusinessWorks Container Edition (BWCE 2.9.2 / 2.10.0) microservices on Red Hat OpenShift 4.20+. Explains 12-factor configuration externalization via .substvar, eliminating pod CPU limits to prevent Linux CFS quota bandwidth throttling on 64-thread engines, OpenMetrics scraping on port 8090, and automated canary rollouts with Argo Rollouts and Datadog APM.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

2. DevSecOps Fortress Blueprint

🔗 Direct Link: https://www.youtube.com/watch?v=ofQfc-NgoUQ
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 7:42
🏷️ Engineering Domain: Zero-Trust DevSecOps, Supply Chain Security & Vault Secrets
📝 Technical Overview: Zero-trust security architecture for enterprise TIBCO BWCE microservices on OpenShift. Covers SLSA Level 3 supply chain provenance, cryptographic image signing with Sigstore Cosign, automated vulnerability scanning with Aqua Trivy, CycloneDX 1.5 SBOM generation, dynamic secret injection with HashiCorp Vault and External Secrets Operator (ESO), and running under OpenShift restricted-v2 SecurityContextConstraints.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

3. Multi Repo CI CD Puzzle

🔗 Direct Link: https://www.youtube.com/watch?v=kH7Hweim5oY
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 7:32
🏷️ Engineering Domain: Multi-Repository Decoupling & ArgoCD Multi-Cluster GitOps
📝 Technical Overview: Deconstructs the multi-repository decoupling challenge when separating application source code (jenkins-git-parameter-bwce) from global configuration topologies (jenkins-git-parameter-bwce-global-vars). Solves the SCM pre-execution render paradox inherent to Jenkins git-parameter dropdowns and demonstrates the shift to declarative, pull-based GitOps with ArgoCD 3.5 across multi-cluster OpenShift topologies.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

📂 Technical Shorts Matrix & Architecture Breakdowns

⚡ Video Shorts Matrix

#	Short Title	Architectural Domain & Focus	Duration	Action
01	How Canary Deployments Auto Rollback	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:28`	▶️ Watch
02	Why CPU Limits Freeze TIBCO BWCE	Linux CFS Kernel Quota Throttling Eliminating pod CPU limits on 64-thread JVMs & managing quota via ResourceQuotas	`1:32`	▶️ Watch

1. How Canary Deployments Auto Rollback

🔗 Direct Link: https://www.youtube.com/shorts/GUmnvmFmXxk
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:28
🏷️ Engineering Domain: Progressive Delivery, Argo Rollouts & Datadog SLA Analysis
📝 Technical Overview: Demonstrates how progressive canary delivery protects mission-critical enterprise integration services. Details traffic splitting with Argo Rollouts (routing 20% traffic to canary and 80% to stable) and continuous Datadog APM SLA evaluation (5xx error rate under 0.1%, P99 latency under 250ms), triggering instant automated rollbacks when thresholds are breached.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

2. Why CPU Limits Freeze TIBCO BWCE

🔗 Direct Link: https://www.youtube.com/shorts/MNl8eUYzkDQ
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:32
🏷️ Engineering Domain: Linux Kernel cgroups, CFS Quota Bandwidth Throttling & JVM Sizing
📝 Technical Overview: Unpacks the counterintuitive Linux Completely Fair Scheduler (CFS) bandwidth throttling trap on containerized multi-threaded JVMs like TIBCO BWCE. Explains why setting pod-level CPU limits divides CPU time into discrete 100ms budgetary windows that 64 engine threads exhaust immediately, causing kernel freezes despite idle node CPU. Demonstrates the production fix: removing container CPU limits and managing capacity at the namespace boundary via OpenShift ResourceQuotas.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

References & Standards

TIBCO BWCE Documentation: TIBCO BusinessWorks™ Container Edition Official Docs
Datadog Jenkins Integration: Datadog CI Visibility & Jenkins Plugin
Datadog Java Tracer APM: Datadog Java Agent Documentation
ArgoCD 3.5 & ApplicationSets: ArgoCD Documentation
Sigstore Cosign: Cosign Container Signing Documentation
External Secrets Operator: External Secrets Operator Docs
Jenkins Job DSL & Git Parameter: Jenkins Job DSL API

🔗 Related Repositories

Global Variables SSOT: jenkins-git-parameter-bwce-global-vars
Base Generic CI/CD Blueprint: nubenetes/jenkins-git-parameter
Base Generic Global Variables SSOT: nubenetes/jenkins-git-parameter-global-vars

Name		Name	Last commit message	Last commit date
argocd-apps		argocd-apps
config		config
helm		helm
jcasc		jcasc
jenkinsfiles		jenkinsfiles
jobdsl		jobdsl
observability		observability
sample-apps		sample-apps
scripts		scripts
security		security
shared-library		shared-library
CHANGELOG.md		CHANGELOG.md
Makefile		Makefile
README.md		README.md
deploy.sh		deploy.sh
destroy.sh		destroy.sh
reinstall.sh		reinstall.sh

Latest commit

History

Folders and files

Repository files navigation

🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform with Datadog Observability

⚠️ Architectural Blueprint & Sovereign Environment Notice

🔗 Companion Repository (Single Source of Truth)

🗺️ Quick Navigation Map

🧭 Repository Architecture Blueprint

🎬 AI-Generated Multimedia Series (YouTube)

📽️ Full-Length Technical Deep Dives (Architecture Masterclasses)

⚡ Video Shorts Matrix

📑 Table of Contents

Executive Summary & Architecture Overview

📋 Architectural Breakdown & Workflow Steps:

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via .substvar

📋 Profile Externalization Mechanics:

2. Engine Sizing & Performance Tuning

💡 Why CPU Limits are Omitted at the Pod Level (Preventing Linux CFS Bandwidth Throttling)

3. OpenShift Hardening & Health Probing

Datadog Full-Stack Observability Architecture

📋 Datadog Telemetry Data Flow:

1. Jenkins CI/CD Visibility Plugin

2. TIBCO BWCE Java APM Tracer & DogStatsD

3. Datadog Live Dashboards & Monitors

4. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

📋 Progressive Canary Delivery Steps:

Multi-Repository Git Parameter CI/CD Patterns

The Core Problem: Why Standard Pipelines Fail with Multiple Git Repositories

📋 Why Multi-Repo Pipelines Fail for BWCE:

Master Seed Job & JCasC Automation Architecture

📋 Pipeline Provisioning Chain:

1. Why jobdsl/seed-job.groovy Only Declares Folders (Separation of Concerns)

2. Managing Folders in Job DSL vs. Static JCasC

3. Step-by-Step Technical Chain & Execution Flow

Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)

Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator

📋 Dual-Pattern Coexistence Mechanics:

📋 Step-by-Step CI/CD Hand-off Sequence:

Automated Downstream Hand-off & Environment Targeting for BWCE

📋 Downstream Parameter Passing & Execution Logic:

Enterprise Security & Supply Chain Integrity

1. Cosign Image Signing & SLSA Level 3 Attestation

📋 Supply Chain Security & Signing Chain:

2. Zero-Trust Secrets with External Secrets Operator & Vault

📋 Secret Management & Dynamic Injection:

3. Ephemeral PR Preview Environments via ArgoCD ApplicationSets

📋 Dynamic Ephemeral Preview Provisioning Flow:

📂 Repository Structure

🚀 Quick Start & Deployment

1. Prerequisites

2. Configure Credentials

3. Deploy Platform

4. Access Platform Services

Decommissioning & Reinstallation

Clean Decommission

Full Reinstallation

Video Walkthroughs & Architecture References (YouTube)

1. TIBCO BWCE Modernization with jenkins git parameter

2. DevSecOps Fortress Blueprint

3. Multi Repo CI CD Puzzle

⚡ Video Shorts Matrix

1. How Canary Deployments Auto Rollback

2. Why CPU Limits Freeze TIBCO BWCE

References & Standards

🔗 Related Repositories

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

1. 12-Factor Profile Externalization via `.substvar`

1. Why `jobdsl/seed-job.groovy` Only Declares Folders (Separation of Concerns)