🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform on OpenShift 4.20+ (Pure GitOps)

Important

⚠️ AI Generation & Model Review Notice

Initial Generation: Originally generated by Gemini 3.7 Flash with Antigravity.
Review & Production Hardening: Audited, remediated, and verified for enterprise readiness using Gemini 3.8 Flash with Antigravity.
Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. Platform engineers must review and validate all cluster endpoints, secrets, and TLS certificates prior to live production deployment.

Important

🔗 Architecture Paradigm & Linked Repositories

This platform implements the Pure GitOps (Pull-based) Architecture for TIBCO BWCE microservices, serving as the modern cloud-native alternative to the push-based nubenetes/jenkins-git-parameter-bwce pattern:

🚀 Pure GitOps Platform Orchestrator (This Repository): nubenetes/jenkins-without-git-parameter-bwce — Infrastructure-as-Code, Lean Jenkins JCasC CI, ArgoCD 3.5 ApplicationSets, Native TargetRevision Selection, and Datadog APM Observability.
🌐 Reference Push-Based Repository: nubenetes/jenkins-git-parameter-bwce — Legacy/Push model with Jenkins UI gitParameter dropdowns and multi-remote SCM Job DSL.
📦 Embedded Workload Microservices: Located directly within sample-apps/tibco-bwce-order-service and sample-apps/tibco-bwce-customer-api — TIBCO BusinessWorks™ Container Edition cloud-native microservices with 12-Factor .substvar profile externalization and Datadog APM integration.

🗺️ Quick Navigation Map

This enterprise repository implements a Pure GitOps (Pull-based) delivery model for TIBCO BWCE microservices on Red Hat OpenShift 4.20+. By decoupling Jenkins into a lean, parameterless CI build engine and orchestrating multi-cluster deployments via Git and ArgoCD 3.5 ApplicationSets, this architecture eliminates the SCM pre-execution render paradox and external credential exposure. Use this map to navigate the platform blueprint, declarative manifests, and multimedia resources:

🧭 Repository Architecture Blueprint

jenkins-without-git-parameter-bwce/       # 🚀 Pure GitOps Platform Orchestrator (Monorepo)
├── 📁 argocd-apps/                      # ArgoCD Application & ApplicationSet Declarative Manifests
│   ├── 📄 applicationset-clusters.yaml  # Multi-cluster deployment generator (DEV, STG, PROD)
│   ├── 📄 applicationset-git-branches.yaml # Native Git branch parameterization generator
│   ├── 📄 applicationset-pull-request-preview.yaml # Dynamic ephemeral PR preview environments
│   ├── 📄 root-app-of-apps.yaml         # Declarative Root Application orchestrator
│   └── 📁 apps/                         # App manifests for Jenkins, Datadog & BWCE services
├── 📁 jcasc/                            # Jenkins Configuration as Code (JCasC 2.492.2 LTS)
│   ├── 📄 jenkins-jcasc.yaml            # Master JCasC (security, credentials, agents, Datadog)
│   ├── 📄 github-app-credentials.yaml   # GitHub App tokenless authentication
│   └── 📄 pod-templates.yaml            # Ephemeral Kubernetes builder agent templates
├── 📁 jobdsl/                           # Parameterless Programmatic Pipelines (Job DSL)
│   ├── 📄 seed-job.groovy               # Seed job automatically bootstrapping CI pipelines
│   └── 📄 pipelines-ci.groovy           # Parameterless CI build, test, containerize & sign
├── 📁 sample-apps/                      # Embedded Workload Microservices & GitOps Manifests
│   ├── 📁 tibco-bwce-order-service/     # TIBCO BWCE microservice with 12-Factor .substvar profiles
│   ├── 📁 tibco-bwce-customer-api/      # TIBCO BWCE customer integration microservice
│   └── 📁 gitops-manifests/             # Helm charts, Argo Rollouts canary, & overlays
├── 📁 observability/                    # Datadog Full-Stack Observability Suite
│   ├── 📁 dashboards/                   # Datadog JSON dashboards for BWCE JVM & CI Visibility
│   └── 📁 monitors/                     # Datadog APM alerts & Argo Rollouts SLA tripwires
├── 📁 security/                         # Zero-Trust & Supply Chain Security Hardening
│   ├── 📁 external-secrets-operator/    # ESO ClusterSecretStores syncing HashiCorp Vault secrets
│   ├── 📄 openshift-image-signature-policy.yaml # Cosign SLSA 3 signature verification
│   └── 📄 openshift-namespace-resource-quota.yaml # CPU quota governance replacing pod limits
├── 📁 helm/                             # Helm charts for ArgoCD, Jenkins, and BWCE apps
└── 📁 scripts/                          # Automated provisioning and promotion scripts
    ├── 📄 gitops-promote.sh             # Pull-request based semantic promotion helper
    ├── 📄 deploy-datadog-agent.sh       # Datadog Agent OCP deployment with OpenMetrics
    └── 📄 ocp-setup-scc.sh              # OpenShift restricted-v2 SCC configuration

🎬 AI-Generated Multimedia Series (YouTube)

This repository is accompanied by an educational video masterclass and technical shorts synthesized with Gemini NotebookLM based directly on the architectural patterns, multi-cluster Pure GitOps pipelines, and enterprise security controls from this project. All videos are freely accessible on YouTube on the @nubenetes channel.

Note

Multilingual Learning Experience:
Content features native spoken audio in English 🇺🇸, with automated YouTube closed captions (CC) translated into Spanish 🇪🇸 and 20+ languages for global engineering teams.

📽️ Full-Length Technical Deep Dives (Architecture Masterclasses)

#	Video Guide Title	Engineering Domain & Core Architecture	Duration	Direct Link
01	Modernize TIBCO BWCE on OpenShift	Cloud-Native Modernization & CFS Quota Fix 12-Factor .substvar profile externalization, removing CPU limits & Datadog APM	`8:12`	▶️ Watch
02	Datadog in GitOps	Full-Stack Observability & Automated Canary Rollouts Datadog APM, Jenkins CI Visibility, and Argo Rollouts SLA tripwire	`7:56`	▶️ Watch
03	Jenkins Pure GitOps	CI/CD Decoupling & Pure GitOps Solving the SCM blind spot, eliminating UI deploy buttons & ArgoCD pull model	`7:52`	▶️ Watch
04	TIBCO BWCE Pure GitOps	The Pure GitOps Paradigm Shift Evolving from legacy Jenkins push pipelines to declarative ArgoCD pull sync	`8:30`	▶️ Watch

⚡ Video Shorts Matrix

#	Short Title	Architectural Domain & Focus	Duration	Action
01	How CFS Throttling Freezes TIBCO BWCE	Linux CFS Kernel Quota Throttling Eliminating pod CPU limits on 64-thread JVMs & managing quota via ResourceQuotas	`1:13`	▶️ Watch
02	How Datadog Automates Microservice Canary Rollouts	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:27`	▶️ Watch
03	How Pure GitOps Reverses Deployments	Pure GitOps Architecture In-cluster ArgoCD pulling state vs fragile external push scripts	`1:13`	▶️ Watch
04	The Shift to Pure GitOps Deployments	Eliminating UI Deploy Buttons Replacing Jenkins UI dropdowns with Git pull requests & ArgoCD synchronization	`1:23`	▶️ Watch
05	How Pure GitOps Replaces Jenkins Push	Zero-Trust CI/CD Security Stripping cluster admin credentials from Jenkins & relying on in-cluster ArgoCD pull	`1:19`	▶️ Watch
06	Cómo funciona un despliegue Canary automatizado 🇪🇸	Despliegue Progresivo y SLA en Tiempo Real División de tráfico 20/80 con Argo Rollouts y rollbacks automáticos con Datadog APM	`1:13`	▶️ Watch
07	How Canary Rollouts Catch Bad Code	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:18`	▶️ Watch

For complete technical summaries, topic breakdowns, and direct studio links, see Section: Video Walkthroughs & Architecture References.

📑 Table of Contents

Quick Navigation Map
AI-Generated Multimedia Series (YouTube)
Executive Summary & Architectural Paradigm Shift
In-Depth Architectural Comparison: Push vs. Pull Model for TIBCO BWCE
TIBCO BWCE Cloud-Native Best Practices
Datadog Full-Stack Observability Architecture
Comprehensive Mermaid Architecture Diagrams & Workflows
Which Pattern is Easier, Recommended, and Why?
Repository Structure
Quick Start: 1-Click Automated Deployment
Decommissioning & Reinstallation
Video Walkthroughs & Architecture References (YouTube)
References & Standards

Executive Summary & Architectural Paradigm Shift

For enterprise integration platforms hosting TIBCO BusinessWorks™ Container Edition (BWCE 2.9.2 / 2.10.0) on Red Hat OpenShift 4.20+, organizations face a crucial architectural decision:

The Legacy Push Model (jenkins-git-parameter-bwce): Jenkins is used as a manual deployment gate. Developers log into the Jenkins Web UI, select the BWCE application branch and environment configuration tag via git-parameter dropdowns, and Jenkins compiles the EAR, builds the container image, commits to the configuration repo, and triggers ArgoCD sync.
The Declarative Pull Model (jenkins-without-git-parameter-bwce - Pure GitOps): Jenkins is decoupled and strictly scoped to Continuous Integration (CI) (compiling the BWCE project with bw6-maven-plugin, packaging the .ear, building the container image on top of tibco/bwce:2.9.2, scanning with Trivy, generating Syft SBOM, and signing with Cosign SLSA 3). Jenkins has zero deployment parameters and zero cluster credentials.

All parameter selections (branches, tags, commits, .substvar profiles) are executed natively via Git (the Single Source of Truth) and ArgoCD 3.5:
- Feature branch testing via Git Pull Requests (ArgoCD ApplicationSets automatically create and destroy ephemeral preview environments).
- Release selection via Git Semantic Versioning tags (v1.2.0, release-2026.09).
- Ad-hoc overrides via ArgoCD UI/CLI native targetRevision tracking.

In-Depth Architectural Comparison: Push vs. Pull Model for TIBCO BWCE

0. The Architectural Evolution: From Two-Pipeline CI/CD Hand-off to Pure GitOps for BWCE

Important

Context & Heritage: The Patterns in jenkins-git-parameter-bwce vs. Pure GitOps

In the baseline repository jenkins-git-parameter-bwce, we evaluated two core Jenkins push patterns for enterprise TIBCO BWCE environments:

Pattern 1: Dual Git Parameter Dropdowns in a Single Pipeline (Multi-Remote SCM):
Configured both the BWCE application repo and jenkins-git-parameter-bwce-global-vars inside a single Job DSL definition using custom refspecs (origin-app and origin-vars). While functional, it suffered from SCM namespace collisions, Jenkins master UI render latency, and workspace checkout quirks.
Pattern 2: Decoupled Two-Pipeline Architecture (CI ➔ CD Hand-off) [RECOMMENDED in jenkins-git-parameter-bwce]:
Separated EAR compilation from release promotion:
- Pipeline 01 (01-CI-Build-Pipelines/*-ci-build): Bound to the BWCE application repo with an APP_GIT_REVISION dropdown. Built the .ear via bw6-maven-plugin, layered it onto tibco/bwce:2.9.2, and triggered downstream.
- Pipeline 02 (02-CD-Release-Orchestrators/multi-cluster-release-orchestrator): Bound directly to jenkins-git-parameter-bwce-global-vars with its own GLOBAL_VARS_REVISION dropdown (DEV, STAGING, PROD). Orchestrated multi-cluster image promotion (via Skopeo), committed .substvar profile updates to GitOps manifests, and invoked argoAppSync.

While Pattern 2 was the best possible design within the Jenkins Push Paradigm (honoring Single Responsibility Principle and "Build Once, Deploy Anywhere"), it still kept Jenkins as the manual Parameter Proxy, release coordinator, and credential holder in Pipeline 02.

In jenkins-without-git-parameter-bwce (This Repository), we complete the cloud-native evolution:

Pipeline 02 is completely eliminated from Jenkins: ArgoCD 3.5 natively performs all continuous delivery, drift detection, and multi-cluster reconciliation directly from Git.
Pipeline 01 becomes a parameterless, webhook-driven CI engine: Developers never interact with Jenkins UI dropdowns; pushing code or opening PRs automatically compiles the EAR, scans (Trivy), generates SBOM (Syft), signs (Cosign SLSA 3), and commits the image tag to GitOps.
Release selection is native to Git & ArgoCD: Handled via Git PRs, Semantic Version tags (v1.2.0), or ArgoCD's native targetRevision UI/CLI with 12-Factor .substvar profiles managed declaratively in Kustomize overlays.

Click to expand: 🔄 Three-Way Architectural Evolution Diagram (Pattern 1 vs. Pattern 2 vs. Pure GitOps for BWCE)

flowchart TB
    subgraph P1["Pattern 1: Push"]
        direction TB
        Dev1["👩‍💻 User"] -->|"1. Select Dropdowns"| JJob1["📋 Single Pipeline<br/>• Multi-Remote SCM<br/>• origin-app<br/>• origin-vars"]
        JJob1 -->|"2. Builds & Deploys"| Agent1["⚙️ Jenkins Agent<br/>• Cluster Secrets"]
        Agent1 -->|"3. Imperative Push"| K8s1["☸️ OpenShift Clusters"]
    end

    subgraph P2["Pattern 2: Hand-off"]
        direction TB
        Dev2["👩‍💻 User"] -->|"1. Selects BWCE Branch"| CI2["🏗️ Pipeline 01: CI<br/>• App Git Parameter"]
        CI2 -->|"2. Builds EAR & Image"| Reg2["🐳 Container Registry"]
        CI2 -->|"3. Triggers Downstream"| CD2["🚀 Pipeline 02: CD<br/>• Global Vars Dropdown<br/>• Release Orchestrator"]
        CD2 -->|"4. Skopeo & Commit"| GitOps2["🌐 GitOps Repo<br/>(global-vars)"]
        CD2 -->|"5. Calls argoAppSync"| Argo2["🐙 ArgoCD Controller"]
        Argo2 -->|"6. Syncs Cluster"| K8s2["☸️ OpenShift Clusters"]
    end

    subgraph P3["Pattern 3: Pure GitOps"]
        direction TB
        Dev3["👩‍💻 Developer"] -->|"1. Git PR / Tag"| Git3["🐙 Git Repo (SSOT)<br/>• BWCE App Code<br/>• GitOps Overlays"]
        Git3 -.->|"2. Webhook Event"| CI3["🏗️ Lean Jenkins CI<br/>• Zero UI Params<br/>• Multibranch Webhook"]
        CI3 -->|"3. Package & Scan"| Reg3["🐳 Container Registry<br/>(Cosign SLSA 3)"]
        CI3 -->|"4. Auto-commit Tag"| Git3
        Git3 -->|"5. Continuous Sync"| Argo3["🐙 ArgoCD 3.5 Engine<br/>• targetRevision<br/>• ApplicationSets"]
        Argo3 -->|"6. Self-Healing"| K8s3["☸️ OpenShift Runtime<br/>• Zero Secrets in CI<br/>• Multi-Cluster"]
    end

📊 Three-Way Architecture Comparison Matrix for TIBCO BWCE

Architectural Feature	Pattern 1 (Dual Dropdown Push)	Pattern 2 (Decoupled CI ➔ CD Hand-off Push)	Pure GitOps (This Repository)
Jenkins Jobs Count	1 Monolithic Pipeline per BWCE app.	2 Pipelines (01-CI-Build + 02-CD-Orchestrator).	1 Parameterless CI Pipeline per app.
Parameter Interface	Jenkins UI (2 dropdowns in 1 job).	Jenkins UI (1 dropdown in CI, 1 dropdown in CD).	Git (PRs/Tags) & ArgoCD `targetRevision`.
`.substvar` Profile Binding	Managed via Jenkins UI dropdown.	Managed via Pipeline 02 `global-vars` dropdown.	Declarative Kustomize Overlays in Git.
SCM Complexity	High (Multi-remote refspec bindings).	Moderate (Isolated SCM bindings per pipeline).	Zero SCM Hacks (Standard webhooks/branches).
Release Trigger	Manual human click in Jenkins.	Manual click or downstream trigger to Pipeline 02.	100% Event-Driven via Git webhooks / PRs.
Jenkins Credentials	High (Cluster tokens & Git write).	High (Cluster tokens, Skopeo, ArgoCD API keys).	Zero Cluster Credentials (Least Privilege).
Deployment Engine	Jenkins Agent (Push).	Jenkins Agent ➔ invokes ArgoCD sync.	ArgoCD 3.5 Pull Controller (Continuous).
Ephemeral PR Envs	Complex Groovy scripts.	Complex Groovy scripts in Pipeline 02.	Native ArgoCD ApplicationSet PR Generator.
Configuration Drift	Blindspot (No self-healing).	Blindspot (ArgoCD only syncs when Jenkins runs).	Continuous Self-Healing (24/7 reconciliation).

💡 Architectural Summary & Conclusion: Applying Pure GitOps to enterprise TIBCO BWCE modernizes legacy integration workflows into cloud-native pipelines. By eliminating Pipeline 02 from Jenkins and relying on ArgoCD 3.5 for declarative .substvar profile synchronization, integration teams achieve zero-trust security, immutable EAR packaging, and 24/7 self-healing drift detection.

⚖️ Decoupled Push (Pattern 2) vs. Pure GitOps Pull Implementation Analysis for BWCE

A critical architectural distinction is: Why does Pure GitOps completely eliminate Pipeline 02 from Jenkins rather than chaining downstream pipelines for TIBCO BWCE?

Click to expand: 🔄 Pattern 2 Downstream Push vs. Pure GitOps Event-Driven Pull Diagram for BWCE

flowchart TB
    subgraph LegacyPush["1. Pattern 2: Push"]
        direction TB
        CI1["🏗️ Pipeline 01: CI<br/>• Packages EAR<br/>• Passes Tag"] -->|"build job: '02-CD...'<br/>(Passes: TARGET_ENV)"| CD1["🚀 Pipeline 02: CD<br/>• Release Lead<br/>• Cluster Secrets"]
        CD1 -->|"argoAppSync / Skopeo"| Cluster1["☸️ OpenShift (Push)<br/>(DEV / STG / PROD)"]
    end

    subgraph PureGitOps["2. Pure GitOps"]
        direction TB
        CI2["🏗️ Lean CI (Pull)<br/>• Compiles & Signs<br/>• Single Pipeline"] -->|"gitopsCommit (Git)"| GitSSOT["🐙 Git Repo (SSOT)<br/>• .substvar Overlays<br/>• Image Digests"]
        GitSSOT -->|"Continuous Sync"| ArgoCD["🐙 ArgoCD 3.5 Engine<br/>• Reconciles State<br/>• Zero CI Secrets"]
        ArgoCD -->|"Declarative Pull"| Cluster2["☸️ OpenShift (Pull)<br/>(DEV / STG / PROD)"]
    end

📋 Key Architectural & Implementation Differences:

Architectural Dimension	`jenkins-git-parameter-bwce` (Pattern 2)	`jenkins-without-git-parameter-bwce` (Pure GitOps)
Pipelines in Jenkins	2 Pipelines: `01-ci-build` + `02-release-orchestrator`.	1 Pipeline: `01-CI-Build-Pipelines` (Lean Multibranch).
Downstream Linkage	Jenkins `build job: '02-CD-Release-Orchestrators/...'`.	No downstream Jenkins job. CI commits image tag to Git and terminates.
Who Orchestrates CD?	Jenkins Pipeline 02 (promotes via Skopeo, binds `.substvar`, calls `argoAppSync`).	ArgoCD 3.5 Controller (tracks Git and reconciles clusters continuously).
Environment Targeting	Chosen in Jenkins parameter dropdown (`TARGET_ENVIRONMENT`).	Driven by Git Branch / PR: `main` ➔ DEV, PR/Tag ➔ STAGING/PROD.
Promotion Mechanism	Pipeline 02 promotes via Skopeo and runs approval gates (`input`).	Git Pull Request (PR): Merging a PR into environment manifests triggers ArgoCD.
Cluster Credentials	Stored inside Jenkins agents (`argocd-gitops`).	Zero cluster credentials in Jenkins. Only ArgoCD accesses OpenShift.

💻 Code Walkthrough in Pure GitOps BWCE CI Pipeline:

In jenkinsfiles/ci/Jenkinsfile.app-bwce, the pipeline ends immediately after compiling the EAR, signing the container image, and writing the digest to Git—completely removing downstream job invocations:

stage('GitOps Automatic Sync (Update Git Repository)') {
    steps {
        script {
            def targetEnv = (env.BRANCH_NAME == 'main') ? 'dev' : ((env.BRANCH_NAME == 'staging' || env.TAG_NAME) ? 'staging' : 'dev')
            echo "📝 Updating GitOps Repository for environment: ${targetEnv} with new image tag: ${env.IMAGE_TAG}"
            
            // Jenkins writes directly to Git, then terminates.
            gitopsCommit(
                appName: env.APP_NAME,
                imageTag: env.IMAGE_TAG,
                envName: targetEnv,
                commitMsg: "chore(gitops): promote ${env.APP_NAME} to ${env.IMAGE_TAG} for ${targetEnv} [skip ci]"
            )
        }
    }
}

💡 Architectural Summary & Conclusion: Retiring Pipeline 02 modernizes enterprise TIBCO BWCE delivery into a pure cloud-native workflow. Jenkins functions strictly as a high-speed EAR compiler and container packager, while ArgoCD manages multi-cluster .substvar profile synchronization, drift correction, and automated canary rollouts.

1. Comprehensive Comparison Matrix: Jenkins Git Parameter vs. Pure GitOps for BWCE

Architectural Dimension	`jenkins-git-parameter-bwce` (Push Model)	`jenkins-without-git-parameter-bwce` (Pure GitOps)	Advantage & Recommendation
Control Model	Imperative Push: Jenkins drives both EAR compilation and cluster deployment.	Declarative Pull: Jenkins handles CI only; ArgoCD continuously reconciles cluster state from Git.	🏆 GitOps: Cloud-native standard (CNCF / OpenGitOps).
Where Parameters are Selected	Jenkins Web UI: User selects branch/tag/commit via `git-parameter` dropdowns.	Git & ArgoCD: Selected via Git branches, tags, PRs, or ArgoCD `targetRevision` UI/CLI.	🏆 GitOps: Eliminates Jenkins UI lag and human form errors.
BWCE Profile (`.substvar`) Management	Split: Handled via Jenkins job parameters and external global-vars repo dropdowns.	Declarative Overlays: Managed cleanly via Kustomize overlays in the GitOps repository.	🏆 GitOps: Immutable, versioned configuration.
Single Source of Truth (SSOT)	Split: Jenkins build history & job parameters determine deployed state.	Pure Git: Git repository commit history is the single, cryptographically auditable source of truth.	🏆 GitOps: Immutable commit logs with GPG/Cosign attestation.
Jenkins Master Performance	High Overhead: Master dynamically queries remote Git refs before rendering build forms; SCM latency.	Zero Overhead: Jenkins uses lightweight webhooks and multibranch indexing; zero UI querying overhead.	🏆 GitOps: Controller remains fast, stable, and highly responsive.
Security & RBAC Boundary	High Exposure: Jenkins Master & agents require cluster admin tokens and ArgoCD API keys.	Zero-Trust: Jenkins only has write access to internal container registry & Git repo. Only ArgoCD has cluster access.	🏆 GitOps: Drastically reduced blast radius and attack surface.
Configuration Drift Handling	Blindspot: Out-of-band cluster changes (`oc edit`) cannot be detected until the next build.	Continuous Self-Healing: ArgoCD continuously monitors cluster state and automatically self-heals any drift.	🏆 GitOps: Zero drift guarantee across all OpenShift clusters.
Rollback Mechanism	Imperative Re-run: Finding prior build in Jenkins and re-running with previous tags.	Instant Git Revert: Standard `git revert <commit>` or 1-click rollback in ArgoCD UI/CLI.	🏆 GitOps: Deterministic, instant, and fully traceable.
Ephemeral Preview Environments	Script-Heavy: Complex custom Groovy stages in Jenkinsfile to create/destroy namespaces.	Declarative ApplicationSets: ArgoCD PR Generator automatically creates and destroys preview environments per GitHub PR.	🏆 GitOps: 100% automated lifecycle per PR.
Observability Integration	Datadog agent injected via Jenkins build arguments.	Native K8s Injection: Datadog APM injected via Pod annotations and standard OpenShift admission.	🏆 GitOps: Clean separation of telemetry from CI scripts.

2. The Root Cause of Jenkins SCM Friction in BWCE Builds

In the jenkins-git-parameter-bwce pattern, engineering teams hit three major architectural bottlenecks:

The Pre-Execution vs. Runtime Lifecycle Paradox: Jenkins renders build parameter dropdowns in the browser before launching an agent pod, before cloning source code, and before running pipeline stages. Therefore, git-parameter can only query Git repositories statically defined in the Jenkins Master's Job XML configuration.
The SCM Multi-Remote Binding Bottleneck: Declarative Pipelines (cpsScm) were designed for a single primary SCM. When a pipeline needs to select a branch from an application repository (app-repo) AND a configuration tag from an environment repository (global-vars), Job DSL must configure multi-remote Git refspecs (origin-app, origin-vars), making jobs brittle and complex.
Security Inversion: In push-based Jenkins, Jenkins agents must possess broad OpenShift cluster administrative privileges or ArgoCD admin tokens to deploy resources, violating the principle of least privilege.

3. How Git & ArgoCD Solve BWCE Parameterization Natively

┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                          Pure GitOps Parameter Selection for BWCE                        │
├──────────────────────────────────────┬───────────────────────────────────────────────────┤
│ Requirement                          │ How it is Handled in Pure GitOps                  │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 1. Deploying a Feature Branch        │ Open a Pull Request in GitHub ──>                 │
│                                      │ ArgoCD ApplicationSet PR Generator creates        │
│                                      │ preview-pr-<number>-bwce environment instantly.   │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 2. Deploying a Release Tag (v1.2.0)  │ Create a Git Release Tag `v1.2.0` in GitOps repo  │
│                                      │ (or update targetRevision in ArgoCD Application). │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 3. Ad-hoc Commit / Hotfix Override   │ Execute `argocd app set <app> --revision <sha>`   │
│                                      │ or select revision in the ArgoCD Web UI dialog.   │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 4. Promoting DEV -> STAGING -> PROD  │ Jenkins CI automatically commits image tag to Git │
│                                      │ or opens a Promotion PR across environment files. │
└──────────────────────────────────────┴───────────────────────────────────────────────────┘

4. Why There is No `jenkins-without-git-parameter-bwce-global-vars` Repository

In jenkins-git-parameter-bwce, a separate global-vars repository existed because Jenkins required a second UI dropdown (GLOBAL_VARS_REVISION) to let humans select environment configurations at build time.

In jenkins-without-git-parameter-bwce:

Jenkins has no dropdowns: CI runs 100% automated via webhooks.
ArgoCD owns the configuration: Environment definitions and .substvar profile mappings live in standard GitOps directories (config/clusters.yaml, sample-apps/gitops-manifests/environments/, and argocd-apps/).
No Jenkins-Specific Naming: In pure GitOps, configuration repositories belong to ArgoCD and Kubernetes, not Jenkins.

5. Decoupled Architecture: Docker Images vs. Environment Variables & Placeholders

A core design consideration for enterprise TIBCO BusinessWorks™ Container Edition (BWCE) is: How are BWCE container images decoupled from environment variables, .substvar profile placeholders, and secrets?

1. Lifecycle Decoupling (Artifact vs. Configuration in BWCE)

In Pure GitOps, the executable BWCE container image and environment-specific configurations are cleanly separated at the container boundary:

The BWCE Docker Image is 100% Environment-Agnostic:
- The image contains the compiled BusinessWorks Enterprise Archive (.ear) built via bw6-maven-plugin, layered onto tibco/bwce:2.9.2.
- It contains zero environment endpoints, zero credentials, and zero hardcoded database URLs.
- The exact same container image digest (sha256:def5678) promoted from DEV runs unchanged in STAGING and PROD.
Environment Variables & .substvar Profiles Live in GitOps Manifests:
- Environment-specific values (BW_PROFILE: DEV.substvar, STAGING.substvar, PROD.substvar, Vault secret placeholders) live declaratively in Kustomize overlays (sample-apps/tibco-bwce-order-service/k8s/overlays/{dev,staging,prod}/patch-env.yaml).
- At pod startup, Kubernetes/OpenShift injects the BW_PROFILE variable and ConfigMaps/Secrets reconciled by ArgoCD.

Click to expand: 📦 TIBCO BWCE Docker Image vs. .substvar Decoupling Lifecycle Diagram

flowchart LR
    subgraph BuildTime["1. CI Build (BWCE)"]
        direction TB
        Code["📦 BWCE Studio<br/>(bw6-maven-plugin)"] --> Build["🏗️ Lean Jenkins CI"]
        Build --> Image["🐳 Immutable BWCE Image<br/>(bwce:2.9.2 + EAR)<br/>• Zero env endpoints<br/>• Zero secrets"]
        Image --> Registry["OpenShift Registry"]
    end

    subgraph Runtime["2. CD GitOps (Config)"]
        direction TB
        Overlays["📁 GitOps Manifests<br/>• DEV.substvar<br/>• PROD.substvar<br/>• ConfigMaps / Vault"]
        ArgoCD["🐙 ArgoCD 3.5 Engine"]
        Cluster["☸️ OpenShift Cluster"]
        
        Overlays --> ArgoCD
        ArgoCD -->|"Injects Profile"| Cluster
    end

    Registry -.->|"Pulls by Digest"| Cluster

2. Why This Repository Uses a Unified Platform Monorepo

Rather than fragmenting into 3 separate Git repositories (bwce-app-repo, ci-platform-repo, and global-vars-repo), this project organizes them in a cohesive, self-contained Platform Blueprint:

1-Click Reproducibility & Portability: Platform engineers can clone a single repository and execute ./deploy.sh or make deploy to provision Jenkins JCasC, ArgoCD 3.5, ApplicationSets, Datadog APM, and sample BWCE microservices with zero broken links.
Elimination of Jenkins Cross-Repo Parameter Coordination: In the legacy push model, separate repositories were needed because Jenkins had a manual UI form requiring developers to pick combinations of (BWCE_App_Tag, Substvar_Tag). In Pure GitOps, Jenkins does not coordinate parameters—it is event-driven via webhooks.
Atomic Versioning & Traceability: Every commit represents a verified snapshot where Jenkins BWCE builder pod templates (jcasc/), pipeline steps (jenkinsfiles/), ArgoCD ApplicationSets (argocd-apps/), and workload overlays (sample-apps/) are guaranteed to be mutually compatible.
Prevention of Orphaned Remote Dependencies: Avoids dependency on external standalone repos that might experience breaking changes, access revocations, or deletion.

3. Platform Monorepo Blueprint vs. Enterprise Polyrepo GitOps Matrix

Architectural Dimension	Platform Monorepo (This Blueprint)	Enterprise Polyrepo GitOps
Repository Topology	1 Cohesive Repository containing IaC, CI, GitOps manifests, and BWCE sample apps.	2 to 3 Repositories (`bwce-app`, `ci-platform`, `central-gitops-manifests`).
Target Audience	Reference architectures, blueprints, platform teams, PoCs, and fast onboarding.	Large enterprises with strict organization boundaries (Integration Teams vs. Platform Ops).
Docker Image Decoupling	Fully Decoupled: Image is built environment-agnostic; overlays inject `BW_PROFILE`.	Fully Decoupled: Image is built environment-agnostic; overlays inject `BW_PROFILE`.
Jenkins CI Role	Parameterless Multibranch CI: Builds EAR, scans, signs, and commits tag to local GitOps folder.	Parameterless Multibranch CI: Builds EAR, scans, signs, and opens a PR to central GitOps repo.
Jenkins UI Parameters	Zero Parameters (Pure webhook / event-driven).	Zero Parameters (Pure webhook / event-driven).
ArgoCD GitOps Role	Reconciles manifests directly from `sample-apps/gitops-manifests/` or overlays.	Reconciles manifests directly from the central `gitops-manifests` repository.
Rollback & Auditability	Instant `git revert` or ArgoCD 1-click revision rollback.	Instant `git revert` in the GitOps repo or ArgoCD 1-click revision rollback.

💡 Architectural Summary & Conclusion: Decoupling BWCE container images from .substvar profile variables enables true 'Build Once, Deploy Anywhere' parity. The exact same container image layered over tibco/bwce:2.9.2 runs across all environments, while environment profiles (DEV.substvar, STAGING.substvar, PROD.substvar) are managed declaratively in Kustomize overlays.

6. Enterprise Developer Portals & Governance: Backstage IDP & ServiceNow / Jira ITSM Integration

A critical enterprise architectural consideration is: How do Internal Developer Portals (Backstage IDP) and ITSM Change Management platforms (ServiceNow, Jira Service Management) integrate with this Pure GitOps pattern?

1. The Paradigm Shift: From Jenkins API Trigger to GitOps Declarative Governance

In the legacy push architecture (jenkins-git-parameter Pattern 2), Backstage and ITSM platforms called Jenkins Pipeline 02 via REST API (POST /job/02-CD-Release-Orchestrators/job/multi-cluster-release-orchestrator/buildWithParameters), making Jenkins the central orchestrator and security vulnerability.

In Pure GitOps (jenkins-without-git-parameter), Backstage and ServiceNow interact directly with Git (the Single Source of Truth) and ArgoCD 3.5, while ArgoCD Notifications automatically updates tickets and developer catalogs:

Click to expand: 🔄 Side-by-Side ITSM & Backstage Architectural Flow (Push vs. Pure GitOps)

flowchart TB
    subgraph LegacyITSM["Pattern A: Push ITSM"]
        direction TB
        DevA["👩‍💻 Developer /<br/>Release Manager"] -->|"1. Opens Ticket"| ITSMA["📋 ServiceNow / Jira<br/>(CHG00123)"]
        ITSMA -->|"2. Approved: API"| JMasterA["⚙️ Jenkins Master<br/>(REST API Trigger)"]
        JMasterA -->|"3. Runs Pipeline 02"| JAgentA["🚀 Jenkins Agent Pod<br/>• Holds Cluster Keys<br/>• Skopeo Engine"]
        JAgentA -->|"4. Commits & Syncs"| ArgoA["🐙 ArgoCD Controller"]
        ArgoA -->|"5. Deploys"| K8sA["☸️ OpenShift PROD"]
        JAgentA -->|"6. Closes Ticket"| ITSMA
        BackstageA["🎭 Backstage IDP"] -->|"Trigger Job"| JMasterA
    end

    subgraph PureGitOpsITSM["Pattern B: GitOps ITSM"]
        direction TB
        DevB["👩‍💻 Developer /<br/>Release Manager"] -->|"1. Self-Service"| PortalB["🎭 Backstage / Jira<br/>(CHG00123)"]
        PortalB -->|"2. Merge PR (CHG00123)"| GitB["🐙 GitOps Repo (SSOT)<br/>(Protected branch)"]
        GitB -->|"3. Continuous Sync"| ArgoB["🐙 ArgoCD 3.5 Engine<br/>(AppSets & Rollouts)"]
        ArgoB -->|"4. Progressive Sync"| K8sB["☸️ OpenShift PROD"]
        
        ArgoB -.->|"5. Notifications: OK"| PortalB
        ArgoB -.->|"6. ArgoCD Plugin"| PortalB
    end

💡 Architectural Summary & Conclusion: Connecting Backstage IDP and ServiceNow / Jira ITSM directly to GitOps repositories provides automated evidence collection (SLSA 3, Syft SBOM, Trivy CVE scans) and compliant change approvals without exposing OpenShift deployment credentials to developer tools.

2. Enterprise Sequence Workflow: ServiceNow / Jira ITSM & ArgoCD

Click to expand: ⚡ ServiceNow / Jira ITSM Automated Change Approval & Reconciliation Sequence Diagram

sequenceDiagram
    autonumber
    actor Ops as 👩‍💻 Release Manager
    participant ITSM as 📋 ServiceNow / Jira
    participant GitHub as 🐙 GitHub (GitOps Repo)
    participant Jenkins as 🏗️ Lean Jenkins CI
    participant ArgoCD as 🐙 ArgoCD 3.5 Engine
    participant Cluster as ☸️ OpenShift PROD

    Note over Jenkins: 1. CI Build, Syft SBOM,<br/>Trivy Scan & Cosign SLSA 3
    Jenkins-->>ITSM: Attach Evidence (SBOM & Cosign)
    
    Ops->>ITSM: Review & Approve CHG009876
    ITSM->>GitHub: Merge Promotion PR #89
    
    GitHub->>ArgoCD: Git Push Event on 'prod'
    activate ArgoCD
    ArgoCD->>Cluster: Progressive Canary (Argo Rollouts)
    Cluster-->>ArgoCD: Health Checks (HTTP 200)
    deactivate ArgoCD
    
    ArgoCD-->>ITSM: Notifications: Close CHG009876
    ArgoCD-->>ITSM: Post Deployment Audit Metrics

3. Backstage IDP & ServiceNow Integration Details

🎭 Backstage IDP (Developer Self-Service & Catalog):
- Promotion Scaffolder Template: Backstage uses @backstage/plugin-scaffolder-backend action publish:github:pull-request to create or merge Promotion PRs in the GitOps repository.
- Live Cluster Observability: Developers view real-time sync status, health checks, pod logs, and canary rollout graphs directly within the Backstage Entity Page via @roadiehq/backstage-plugin-argo-cd or Red Hat Developer Hub (RHDH).
📋 ServiceNow / Jira ITSM (Change Management & SOX Compliance):
- Automated Evidence Collection: Jenkins CI attaches the Cosign SLSA 3 signature link, CycloneDX SBOM, and Trivy vulnerability scan report directly to the pending ServiceNow Change Request (CHG009876).
- Automated Ticket Closure: The ArgoCD Notifications Controller detects healthy cluster deployment and sends a webhook to ServiceNow (PATCH /api/now/table/change_request/<id>) transitioning the ticket to Closed / Implemented.

4. Comparison Matrix: ITSM & Backstage Integration (Push vs. Pure GitOps)

Integration Dimension	Legacy Pattern 2 (`jenkins-git-parameter`)	Pure GitOps (`jenkins-without-git-parameter`)	Advantage & Recommendation
Trigger Mechanism	Imperative API: Backstage/ITSM calls Jenkins `buildWithParameters`.	Declarative Git: Backstage/ITSM creates or merges a GitOps PR.	🏆 GitOps: Standard Git audit trail (cryptographically signed).
Credential Storage	Backstage/ServiceNow stores Jenkins admin credentials; Jenkins holds cluster tokens.	Zero-Trust: ServiceNow only holds a scoped GitHub PR merge token. No cluster tokens.	🏆 GitOps: Minimal attack surface.
Developer Experience in Backstage	Blind: Backstage only shows Jenkins job console logs.	Rich & Live: Backstage ArgoCD Plugin renders live Pod topology, sync status, and rollout progress.	🏆 GitOps: Superior inner-to-outer loop developer UX.
Audit & Compliance (SOX / SOC2)	Fragmented across Jenkins build history and ServiceNow.	Unified in Git: Every production change has an immutable Git commit, peer reviews, and ServiceNow Change ID.	🏆 GitOps: 100% auditable Single Source of Truth.
Failure & Rollback Flow	Developer must log into Jenkins or ServiceNow and manually re-run an older build.	Instant: `git revert <commit>` in Git, or click 1-click Rollback in ArgoCD/Backstage UI.	🏆 GitOps: Sub-second deterministic rollback.
Post-Deploy Ticket Closure	Jenkins Pipeline script executes custom Groovy REST calls at the end of the job.	Native ArgoCD Notifications: Event-driven webhook engine handles ticket state transitions.	🏆 GitOps: Resilient; not tied to a running Jenkins pod.

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via `.substvar`

TIBCO BWCE microservices strictly adhere to 12-Factor App principles. The application .ear contains no hardcoded environmental parameters. At runtime, the container reads the appropriate profile based on the environment variable BW_PROFILE:

DEV: BW_PROFILE: DEV.substvar (Internal mock endpoints, debug logging).
STAGING: BW_PROFILE: STAGING.substvar (UAT databases, staging endpoints).
PROD: BW_PROFILE: PROD.substvar (Production databases, Vault secrets, warning logging).

2. Engine Sizing & JVM Performance Tuning

BWCE runs inside an optimized OpenJ9 or HotSpot JVM. Recommended enterprise container sizing:

resources:
  requests:
    cpu: "500m"
    memory: "1024Mi"
  limits:
    cpu: "2000m"
    memory: "2048Mi"
env:
  - name: BW_JAVA_OPTS
    value: "-Xms1024m -Xmx1536m -XX:+UseG1GC -javaagent:/opt/datadog/dd-java-agent.jar"
  - name: BW_ENGINE_THREADCOUNT
    value: "16"

Datadog Full-Stack Observability Architecture

1. Jenkins CI/CD Visibility Plugin

The official Datadog Jenkins Plugin emits pipeline performance traces, queue times, and step execution metrics directly to Datadog CI Visibility.

2. TIBCO BWCE Java APM Tracer & DogStatsD

The Datadog Java Tracer (dd-java-agent.jar) is injected into the BWCE container runtime. It captures:

End-to-end distributed traces across REST, SOAP, and JMS activities.
Engine process metrics: Active process instances, execution duration, thread pool utilization.
JVM garbage collection and heap utilization.

3. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

During production deployments, Argo Rollouts executes a canary rollout (20% ➔ 50% ➔ 100%) and queries Datadog metrics via an AnalysisTemplate. If the HTTP 5xx error rate exceeds 0.5%, the rollout automatically aborts and rolls back to stable.

Comprehensive Mermaid Architecture Diagrams & Workflows

1. Parameter Flow Comparison (Jenkins Proxy vs. Pure GitOps)

Click to expand: 🔄 Parameter Flow Comparison Diagram (Jenkins Proxy vs. Pure GitOps)

flowchart TB
    subgraph Pattern1["Pattern 1: Proxy"]
        direction TB
        Dev1["👩‍💻 Developer"] -->|"1. Opens Jenkins UI"| JenkinsUI["📋 Jenkins Job Form<br/>(Git Dropdown)"]
        JenkinsUI -->|"2. Queries Remote SCM"| SCMQuery["🔍 Jenkins SCM Query<br/>(origin-app & vars)"]
        SCMQuery -->|"3. Triggers Build"| JenkinsAgent1["⚙️ Ephemeral Agent Pod<br/>(bwce-builder)"]
        JenkinsAgent1 -->|"4. Builds EAR & Image"| Reg1["🐳 Container Registry<br/>(Dev Registry)"]
        JenkinsAgent1 -->|"5. Commits to GitOps"| GitOps1["🌐 GitOps Repo<br/>(global-vars)"]
        JenkinsAgent1 -->|"6. Calls argoAppSync"| Argo1["🐙 ArgoCD Controller<br/>(Triggers Sync)"]
        Argo1 -->|"7. Reconciles State"| Cluster1["☸️ OpenShift Cluster<br/>(DEV / STG / PROD)"]
    end

    subgraph Pattern2["Pattern 2: Pure GitOps"]
        direction TB
        Dev2["👩‍💻 Developer"] -->|"1. Git PR / Tag"| Git2["🐙 Git Repo (SSOT)<br/>(App Code & GitOps)"]
        Git2 -.->|"2. Webhook Event"| JenkinsCI["🏗️ Lean Jenkins CI<br/>(Multibranch Webhook)"]
        JenkinsCI -->|"3. Build EAR & Scan"| Reg2["🐳 Container Registry<br/>(SLSA Level 3)"]
        JenkinsCI -->|"4. Auto-Commits Tag"| GitOps2["🌐 GitOps Manifests<br/>(Overlays & Clusters)"]
        
        GitOps2 -->|"5. Native Revision"| Argo2["🐙 ArgoCD 3.5 Engine<br/>(ApplicationSets)"]
        Argo2 -->|"6. Self-Healing Sync"| Cluster2["☸️ OpenShift Cluster<br/>(DEV / STG / PROD)"]
        
        Dev2 -.->|"Direct Override"| Argo2
    end

📋 Key Contrasts (Proxy vs. Native):

Pattern 1: Jenkins Parameter Proxy (Legacy Push):
- Developer opens Jenkins UI form ➔ Selects BWCE app branch & substvar tag ➔ Jenkins queries remote Git SCM ➔ Builds EAR & image ➔ Agent holds cluster tokens and commits/pushes to OpenShift.
Pattern 2: Pure GitOps Native Selection (Recommended):
- Developer creates Git PR or tag ➔ Webhook triggers parameterless Jenkins CI ➔ Builds EAR, runs BWUnit tests, signs image ➔ Auto-commits digest to GitOps repo ➔ ArgoCD continuously pulls and reconciles OpenShift state.

💡 Architectural Summary & Conclusion: Removing Jenkins as a parameter proxy eliminates UI render latency and multi-remote SCM synchronization bugs. Releases are triggered naturally via Git PRs and tags, allowing ArgoCD to manage multi-cluster BWCE rollouts declaratively.

2. Legacy Global-Vars vs. Pure GitOps Repository Topology

Click to expand: 🌐 Legacy Global-Vars vs. Pure GitOps Repository Topology Diagram

flowchart TB
    subgraph LegacyModel["1. Legacy Push Model"]
        direction TB
        AppRepo1["📦 BWCE App Repo<br/>(tibco-order-service)"]
        GlobalVars1["🌐 Global Vars Repo<br/>(global-vars)"]
        Jenkins1["⚙️ Jenkins Master<br/>• Dropdown 1: BWCE<br/>• Dropdown 2: Vars"]
        Cluster1["☸️ OpenShift Clusters<br/>(DEV / STG / PROD)"]

        AppRepo1 --> Jenkins1
        GlobalVars1 --> Jenkins1
        Jenkins1 -->|"Imperative Push"| Cluster1
    end

    subgraph GitOpsModel["2. Pure GitOps Model"]
        direction TB
        AppRepo2["📦 BWCE App Repo<br/>(sample-apps)"]
        Jenkins2["🏗️ Lean Jenkins CI<br/>• Builds EAR & Image<br/>• Signs (SLSA 3)<br/>• Auto-commits tag"]
        GitOpsRepo["🌐 GitOps Repo (SSOT)<br/>• clusters.yaml<br/>• overlays:<br/>(dev, stg, prod)"]
        ArgoCD["🐙 ArgoCD 3.5 Engine<br/>(Continuous Sync)"]
        Cluster2["☸️ OpenShift Clusters<br/>(DEV / STG / PROD)"]

        AppRepo2 -.->|"Webhook"| Jenkins2
        Jenkins2 -->|"Auto-commits Tag"| GitOpsRepo
        GitOpsRepo -->|"Continuous Sync"| ArgoCD
        ArgoCD -->|"Declarative Pull"| Cluster2
    end

📋 Repository Decoupling Highlights:

Legacy Topology: Application repo and jenkins-*-global-vars repo coordinated via Jenkins UI parameters.
Pure GitOps Topology: Self-contained GitOps repository structure where ArgoCD owns cluster state and environment overlays (DEV.substvar, STAGING.substvar, PROD.substvar), while Jenkins runs 100% automated via webhooks.

💡 Architectural Summary & Conclusion: Consolidating platform assets into a self-contained GitOps structure eliminates fragile cross-repository dependencies. ArgoCD natively owns environment profiles and cluster state, freeing Jenkins CI to act strictly as a high-speed EAR compiler and container packager.

3. End-to-End Multi-Cluster Platform Topology

Click to expand: 🗺️ End-to-End Multi-Cluster Platform Topology Diagram

flowchart TB
    subgraph DeveloperWorkspace["1. Dev & Git (SSOT)"]
        direction TB
        Dev["👩‍💻 Developer /<br/>Release Manager"]
        AppRepo["📦 BWCE App Repo<br/>(sample-apps/bwce)"]
        GitOpsRepo["🌐 GitOps Repository<br/>(Manifests & Config)"]
        GitHubPR["🔀 GitHub Pull Requests"]
    end

    subgraph OCP_DEV["Cluster 1: DEV"]
        direction TB
        subgraph JenkinsPlatform["Jenkins CI (Lean)"]
            Master["Jenkins Controller<br/>(JCasC & Multibranch)"]
            Seed["00-Seed-Job<br/>Provisioner"]
            CIJob["01-Multibranch-CI<br/>(Webhook Triggered)"]
        end

        subgraph Agents["Ephemeral Agent Pods"]
            BwceAgent["bwce-builder Agent<br/>(Maven BW6 & EAR)"]
            SecurityAgent["security-tools Agent<br/>(Cosign & Syft)"]
        end

        subgraph OCPDevRegistry["Internal Registry"]
            DevReg["image-registry:5000<br/>nubenetes-dev-bwce"]
        end

        subgraph ArgoCDMaster["ArgoCD 3.5 Engine"]
            ArgoServer["ArgoCD Server &<br/>ApplicationSets"]
            PRGen["PR Preview Generator"]
            MatrixGen["Matrix Generator"]
        end

        subgraph ObservabilityStack["Datadog Full-Stack"]
            DDAgent["Datadog Cluster Agent<br/>(APM & DogStatsD)"]
            DDMonitors["Datadog Monitors<br/>& Live Dashboards"]
        end

        DevApps["DEV Workloads<br/>(DEV.substvar)"]
        PreviewApps["Ephemeral PR Previews<br/>(pr-preview-*-bwce)"]
    end

    subgraph OCP_STG["Cluster 2: STAGING"]
        StgApps["Staging Workloads<br/>(STAGING.substvar)"]
    end

    subgraph OCP_PRD["Cluster 3: PROD"]
        PrdApps["Production Workloads<br/>(PROD.substvar)"]
        Rollout["Argo Rollouts<br/>(Canary & Datadog SLA)"]
    end

    %% Developer interactions
    Dev -->|"1. Push Code / PR"| AppRepo
    Dev -->|"2. Tag / Merge PR"| GitOpsRepo
    AppRepo -.->|"Webhook Event"| CIJob
    GitHubPR -.->|"PR Webhook"| PRGen

    %% Jenkins CI Flow
    CIJob -->|"Spawns"| BwceAgent
    BwceAgent -->|"Build EAR & Image"| DevReg
    CIJob -->|"Spawns"| SecurityAgent
    SecurityAgent -->|"Cosign & Syft"| DevReg
    SecurityAgent -->|"3. Commit Digest"| GitOpsRepo

    %% ArgoCD GitOps Flow
    GitOpsRepo -->|"Continuous Sync"| ArgoServer
    ArgoServer -->|"Target: main"| DevApps
    PRGen -->|"Target: head_sha"| PreviewApps
    MatrixGen -->|"Target: staging"| StgApps
    MatrixGen -->|"Target: prod"| Rollout
    Rollout -->|"Progressive Canary"| PrdApps

    %% Observability
    DevApps -.->|"APM Traces"| DDAgent
    Rollout -.->|"Query Error Rate"| DDAgent

📋 Architectural Breakdown & Workflow Steps:

Developer & Git Ecosystem (SSOT):
- TIBCO BWCE project source code and declarative GitOps manifests live in a unified version-controlled repository.
Lean Jenkins CI (DEV Cluster):
- Webhook-triggered multibranch pipeline builds EAR with bw6-maven-plugin, executes BWUnit tests, layers onto tibco/bwce:2.9.2, signs with Cosign SLSA 3, and auto-commits digest to Git.
ArgoCD 3.5 GitOps Engine:
- Reconciles BWCE workloads across DEV (DEV.substvar), STAGING (STAGING.substvar), and PROD (PROD.substvar).
Datadog Full-Stack Observability:
- Datadog Cluster Agent and APM tracing (dd-java-agent.jar) provide continuous performance monitoring and canary SLA validation.

💡 Architectural Summary & Conclusion: This topology provides a complete blueprint for running containerized TIBCO BWCE workloads across multi-cluster OpenShift environments. Combining lean Jenkins CI, ArgoCD ApplicationSets, and Datadog APM tracing delivers enterprise-grade reliability and automated governance.

4. Jenkins SCM Pre-Execution Lifecycle Blindspot

Click to expand: ⚠️ Jenkins SCM Pre-Execution Lifecycle Blindspot Diagram

flowchart TB
    subgraph UI_Phase["1. Master (Pre-Exec)"]
        direction TB
        User["👤 User opens<br/>Build UI Form"]
        Master["⚙️ Jenkins Master<br/>reads Job XML"]
        GitParam["🔍 git-parameter<br/>queries SCM refs"]
        Dropdown["📋 Renders Branch/Tag<br/>Dropdown in Browser"]

        User --> Master --> GitParam --> Dropdown
    end

    subgraph Runtime_Phase["2. Runtime (Agent)"]
        direction TB
        AllocAgent["☸️ Ephemeral Agent<br/>Pod Allocated"]
        RunStage["📦 Pipeline Stage:<br/>checkout repo 2"]

        AllocAgent --> RunStage
    end

    Gap["⚠️ SCM Blindspot:<br/>Dynamic checkouts<br/>occur during build and<br/>invisible at render"]

    Dropdown -.-> Gap
    Gap -.-> RunStage

📋 Why Git Parameter Fails for BWCE:

The Pre-Execution Paradox: Jenkins calculates parameter dropdowns before agent allocation; dynamic stage checkouts for .substvar files are invisible during UI rendering.
The GitOps Solution: Pure GitOps removes Jenkins from the parameter selection path; environment profiles are bound declaratively in Kustomize overlays.

💡 Architectural Summary & Conclusion: Dynamic checkouts of .substvar profiles inside Jenkins stages fail under legacy parameter dropdown plugins due to the pre-execution render paradox. Pure GitOps solves this by managing profile selections natively in GitOps manifests.

5. Side-by-Side Flow Comparison: Push vs. Pull

Click to expand: 🔄 Side-by-Side Architectural Flow (Push vs. Pull)

flowchart LR
    subgraph PatternA["Pattern A: Push Model"]
        direction TB
        A1["👩‍💻 User Opens<br/>Jenkins UI"] --> A2["📋 Selects Branch/Tag<br/>in Dropdown"]
        A2 --> A3["⚙️ Jenkins Master<br/>Queries Git SCM"]
        A3 --> A4["📦 Jenkins Agent<br/>Builds BWCE EAR"]
        A4 --> A5["🚀 Agent Pushes<br/>Direct to Cluster<br/>(oc apply / sync)"]
        A5 --> A6["⚠️ Jenkins holds<br/>secrets in CI & drift"]
    end

    subgraph PatternB["Pattern B: Pull GitOps"]
        direction TB
        B1["👩‍💻 Developer Pushes<br/>to Git / Opens PR"] --> B2["⚡ Webhook Triggers<br/>Jenkins CI"]
        B2 --> B3["📦 Jenkins Builds EAR &<br/>Signs Image (SLSA 3)"]
        B3 --> B4["📝 Auto-Commits Tag<br/>to GitOps Repo"]
        B4 --> B5["🔄 ArgoCD Pulls &<br/>Reconciles Cluster"]
        B5 --> B6["✅ Result: Zero secrets<br/>in CI & self-healing"]
    end

📋 Key Contrasts (Push vs. Pull for BWCE):

Pattern A: Jenkins Push Model (Legacy): Jenkins agent compiles EAR, holds OpenShift credentials, and pushes directly to cluster.
Pattern B: Pure GitOps Pull Model (Recommended): Jenkins compiles EAR and signs image; ArgoCD pulls manifests and reconciles BWCE pods with zero cluster secrets in CI.

💡 Architectural Summary & Conclusion: Shifting from imperative push deployments to declarative GitOps pull eliminates the need to store sensitive OpenShift credentials in Jenkins, ensuring that clusters automatically reconcile and self-heal from manual configuration drift.

6. Dynamic Ephemeral Pull Request (PR) Preview Environments for BWCE

Click to expand: ⚡ Dynamic Ephemeral PR Preview Environments Sequence Diagram

sequenceDiagram
    autonumber
    actor Dev as 👩‍💻 Developer
    participant GitHub as 🐙 GitHub (sample-apps/bwce)
    participant Jenkins as 🏗️ Lean Jenkins CI
    participant Registry as 🐳 OpenShift Registry
    participant ArgoCD as 🐙 ArgoCD AppSets
    participant Cluster as ☸️ OpenShift DEV

    Dev->>GitHub: Open PR #42 (feature-order-flow)
    GitHub->>Jenkins: Webhook: PR #42 created
    Jenkins->>Jenkins: Build EAR, Trivy scan & Cosign sign
    Jenkins->>Registry: Push image: bwce:pr-42-sha7
    
    GitHub->>ArgoCD: Polling / PR Webhook
    Note over ArgoCD: Discovers PR #42<br/>label: preview-env
    ArgoCD->>Cluster: Create ns 'pr-preview-42-bwce'
    ArgoCD->>Cluster: Deploy BWCE (head_sha, DEV.substvar)
    ArgoCD-->>GitHub: Post Preview URL in PR #42
    
    Dev->>Cluster: Verify integration on live preview ns
    
    Dev->>GitHub: Merge PR #42 into main
    GitHub->>ArgoCD: PR closed event
    ArgoCD->>Cluster: Tear down ns 'pr-preview-42-bwce'

📋 Ephemeral Preview Lifecycle Steps:

1. PR Creation: Developer opens PR #42 with label preview-environment.
2. CI Build: Jenkins packages EAR, runs BWUnit tests, scans with Trivy, signs with Cosign, and pushes image bwce:pr-42-sha7.
3. Ephemeral Namespace Provisioning: ArgoCD ApplicationSet PR Generator creates namespace pr-preview-42-bwce and deploys BWCE with DEV.substvar.
4. Integration Testing: ArgoCD comments on GitHub PR with live preview URL for integration testing.
5. Automated Teardown: Closing PR #42 triggers ArgoCD to destroy the preview namespace automatically.

💡 Architectural Summary & Conclusion: Ephemeral preview environments allow integration developers to test BWCE business processes and REST APIs in isolated live namespaces before merging code. Automated provisioning and teardown ensure rapid validation without infrastructure sprawl.

7. Automated BWCE CI -> GitOps Promotion Sequence

Click to expand: 🚀 Automated BWCE CI -> GitOps Promotion Sequence Diagram

sequenceDiagram
    autonumber
    participant GitApp as 📦 BWCE App (Code)
    participant Jenkins as 🏗️ Lean Jenkins CI
    participant Registry as 🐳 OpenShift Registry
    participant GitOps as 🌐 GitOps Repo (Manifests)
    participant ArgoCD as 🐙 ArgoCD 3.5 Engine
    participant OCP as ☸️ OpenShift (DEV/STG/PROD)

    GitApp->>Jenkins: Git Push to 'main'
    activate Jenkins
    Jenkins->>Jenkins: Package EAR with bw6-maven-plugin
    Jenkins->>Jenkins: Execute BWUnit Process Tests
    Jenkins->>Jenkins: Syft SBOM & Trivy Scan
    Jenkins->>Registry: Push image (BWCE base + EAR)
    Jenkins->>Registry: Sign with Cosign (SLSA 3)
    
    Jenkins->>GitOps: gitopsCommit(app, tag, env)
    Note over Jenkins,GitOps: Updates kustomization.yaml<br/>with Bot identity
    deactivate Jenkins

    GitOps->>ArgoCD: Webhook / Sync Polling
    activate ArgoCD
    ArgoCD->>ArgoCD: Detect Out-of-Sync Manifest Diff
    ArgoCD->>OCP: Reconcile & Deploy to DEV
    ArgoCD->>OCP: Verify Health (HTTP 200)
    deactivate ArgoCD

📋 Continuous Promotion Execution Chain:

1. Git Trigger: Push to main triggers Jenkins multibranch pipeline.
2. Package & Test: Packages EAR via bw6-maven-plugin, executes BWUnit tests, and generates Syft SBOM.
3. Supply Chain Hardening: Signs container image with Cosign (SLSA Level 3) and scans with Trivy.
4. GitOps Auto-Commit: Auto-commits new image digest to kustomization.yaml using Bot identity.
5. ArgoCD Deployment: ArgoCD reconciles DEV cluster and verifies HTTP 200 health check.

💡 Architectural Summary & Conclusion: The automated promotion pipeline guarantees that every BWCE .ear archive is compiled, unit-tested (BWUnit), scanned for vulnerabilities, and signed with Cosign SLSA 3 before being promoted to GitOps manifests.

8. ArgoCD Multi-Cluster Matrix Reconciliation Engine for BWCE

Click to expand: ⚙️ ArgoCD Multi-Cluster Matrix Reconciliation Engine Diagram

flowchart TB
    subgraph GitOpsSource["1. GitOps Repo (SSOT)"]
        direction TB
        Manifests["📁 Workload Overlays<br/>• k8s/overlays/dev<br/>• k8s/overlays/stg<br/>• k8s/overlays/prod"]
        ClusterList["📋 Cluster Inventory<br/>• config/clusters.yaml<br/>(dev, staging, prod)"]
    end

    subgraph AppSetEngine["2. AppSet Engine"]
        direction TB
        Matrix["⚙️ Matrix Engine:<br/>Combines Clusters<br/>x Overlays"]
        AppDev["Application: dev<br/>• target: main<br/>• DEV.substvar"]
        AppStg["Application: staging<br/>• target: staging<br/>• STAGING.substvar"]
        AppPrd["Application: prod<br/>• target: prod<br/>• PROD.substvar"]
    end

    subgraph TargetClusters["3. OpenShift Runtime"]
        direction TB
        OCPDev["☸️ OCP DEV Cluster<br/>(dev-bwce)"]
        OCPStg["☸️ OCP STAGING Cluster<br/>(staging-bwce)"]
        OCPPrd["☸️ OCP PROD Cluster<br/>(prod-bwce)"]
    end

    Manifests --> Matrix
    ClusterList --> Matrix
    Matrix --> AppDev -->|"Automated Sync"| OCPDev
    Matrix --> AppStg -->|"Automated Sync"| OCPStg
    Matrix --> AppPrd -->|"Canary Sync Waves"| OCPPrd

📋 Matrix Generation Mechanics:

Input 1 (Overlays & Profiles): Kustomize overlays defining BW_PROFILE: DEV.substvar, STAGING.substvar, and PROD.substvar.
Input 2 (Cluster Inventory): config/clusters.yaml defining target OpenShift cluster API endpoints.
Matrix Output: ApplicationSet automatically materializes bwce-dev, bwce-staging, and bwce-prod.

💡 Architectural Summary & Conclusion: The Matrix Generator automates multi-cluster and multi-environment BWCE deployments by binding cluster inventories with environment .substvar profiles, eliminating manual manifest duplication across clusters.

9. Zero-Trust Security & RBAC Boundary Architecture

Click to expand: 🛡️ Zero-Trust Security & RBAC Boundary Architecture Diagram

flowchart TB
    subgraph UntrustedZone["1. CI Zone"]
        direction TB
        JenkinsMaster["Jenkins Controller<br/>• No Cluster RBAC"]
        JenkinsAgent["Ephemeral Agent Pod<br/>• bwce / security"]
        Registry["Internal Registry<br/>• Push & Cosign Sig"]
    end

    subgraph GitOpsTrustZone["2. GitOps Plane"]
        direction TB
        GitRepo["Git Repository<br/>• Source of Truth"]
        ArgoCD["ArgoCD 3.5 Engine<br/>• AppSets Engine"]
    end

    subgraph WorkloadClusters["3. OpenShift Clusters"]
        direction TB
        DEV["OCP DEV Cluster<br/>• SCC restricted-v2"]
        STG["OCP STAGING Cluster<br/>• SCC restricted-v2"]
        PRD["OCP PROD Cluster<br/>• Canary & Hardened"]
    end

    JenkinsMaster -->|"Spawns"| JenkinsAgent
    JenkinsAgent -->|"Push Image & Sig"| Registry
    JenkinsAgent -->|"Commit Digest"| GitRepo
    
    JenkinsAgent -.->|"⛔ BLOCKED"| DEV
    JenkinsAgent -.->|"⛔ BLOCKED"| STG
    JenkinsAgent -.->|"⛔ BLOCKED"| PRD

    GitRepo -->|"Continuous Sync"| ArgoCD
    ArgoCD -->|"Reconcile via Token"| DEV
    ArgoCD -->|"Reconcile via Token"| STG
    ArgoCD -->|"Reconcile via Token"| PRD

📋 Zero-Trust Security & Isolation Principles:

CI Zone (Least Privilege): Jenkins holds zero cluster-admin credentials and cannot deploy directly to OpenShift.
GitOps Control Plane: Only ArgoCD holds short-lived service account tokens to apply Kubernetes manifests.
Protected Runtime: BWCE pods run under OpenShift SCC restricted-v2 with non-root user execution.

💡 Architectural Summary & Conclusion: Isolating Jenkins build pods from OpenShift workload clusters enforces zero-trust architecture. Jenkins CI has zero deployment privileges, preventing compromised build agents from affecting production environments.

10. Progressive Delivery with Argo Rollouts & Datadog SLA

Click to expand: 📊 Progressive Delivery & Datadog SLA Analysis Diagram

flowchart TB
    subgraph RolloutController["1. Argo Rollouts"]
        direction TB
        Step1["1. Initiate Canary<br/>(Set Weight to 20%)"]
        Step2["2. Datadog SLA<br/>• Error Rate < 0.5%<br/>• Latency p95 < 200ms"]
        Step3["3. Promote Canary<br/>(Set Weight to 50%)"]
        Step4["4. Full Production<br/>Promotion (100%)"]
        Abort["🚨 Auto-Rollback<br/>to Stable Version"]
    end

    subgraph RoutingLayer["2. Ingress & Routing"]
        direction TB
        CanaryService["Canary Service Pods<br/>(20% Test Traffic)"]
        StableService["Stable Service Pods<br/>(80% Live Traffic)"]
    end

    Step1 --> CanaryService
    Step1 --> StableService
    CanaryService -->|"Telemetry Spans"| Step2
    Step2 -->|"SLA Passed"| Step3
    Step2 -->|"SLA Breached"| Abort
    Step3 --> Step4

📋 Canary Rollout & Datadog Analysis Steps:

1. Traffic Split: Routes 20% traffic to canary BWCE pods and 80% to stable pods.
2. Datadog APM SLA Analysis: Evaluates error rate $< 0.5%$ and p95 latency $< 200 ext{ms}$ over 5 minutes.
3. Promotion / Rollback: Scales to 50% and 100% on SLA success; executes instant automated rollback if SLA is breached.

💡 Architectural Summary & Conclusion: Integrating Argo Rollouts with Datadog APM metrics enables automated canary releases with real-time error rate and latency validation, guaranteeing automated rollback if BWCE services breach operational SLAs.

11. Full-Stack Observability & Trace Context Propagation with Datadog

Click to expand: 🔭 Full-Stack Observability & Datadog Trace Propagation Diagram

flowchart LR
    subgraph PipelineSpan["1. Jenkins CI Span"]
        direction TB
        JTrace["Datadog CI Visibility<br/>Trace: 7492...048"]
        JBuild["mvn package bw6 EAR"]
        JSign["cosign sign & syft"]
    end

    subgraph GitOpsSpan["2. GitOps Span"]
        direction TB
        GCommit["git commit (trace)"]
        ASync["ArgoCD Sync Check"]
    end

    subgraph AppRuntimeSpan["3. BWCE Runtime Span"]
        direction TB
        AppStart["JVM Startup (dd-agent)"]
        HTTPReq["REST /orders & APM"]
    end

    subgraph UnifiedDatadog["4. Datadog Dashboard"]
        DDDash["Datadog APM Dashboard<br/>Traces & Metrics"]
    end

    JTrace --> JBuild --> JSign
    JSign --> GCommit --> ASync
    ASync --> AppStart --> HTTPReq
    PipelineSpan -.->|"Datadog Agent"| UnifiedDatadog
    GitOpsSpan -.->|"Datadog Agent"| UnifiedDatadog
    AppRuntimeSpan -.->|"Datadog Agent"| UnifiedDatadog

📋 End-to-End Tracing Journey:

1. Jenkins CI Visibility: Datadog CI plugin captures build spans, EAR compilation time, and test results.
2. GitOps & ArgoCD Span: Injects trace ID into Git commit metadata and correlates deployment syncs.
3. BWCE Runtime Tracing: Injects dd-java-agent.jar at JVM startup to trace BWCE REST activities and database palettes.
4. Unified Datadog Dashboard: Correlates pipeline metrics, deployment events, and live APM traces in real time.

💡 Architectural Summary & Conclusion: Injecting dd-java-agent.jar into BWCE container runtimes connects CI build events and GitOps deployments with live distributed traces, enabling instantaneous root-cause analysis across enterprise integration flows.

Which Pattern is Easier, Recommended, and Why?

🏆 Final Recommendation: Pure GitOps (`jenkins-without-git-parameter-bwce`)

Easier for TIBCO BWCE Developers:
- Developers stay in TIBCO Business Studio and Git. Opening a PR automatically triggers CI tests and spins up an ephemeral preview environment on OpenShift with DEV.substvar profile.
- No need to log into Jenkins, wait for remote branches to fetch into dropdowns, and manually submit build forms.
Easier to Maintain for Platform Teams:
- No Jenkins SCM multi-remote refspec hacks (origin-app, origin-vars), no parameter plugin bugs, no pre-execution lifecycle bottlenecks.
Drastically More Secure (Zero-Trust):
- Jenkins has no Kubernetes or OpenShift deploy credentials. If a build container is compromised, the production clusters cannot be altered.
Self-Healing & Drift Remediation:
- ArgoCD guarantees that cluster state matches Git 24/7. Manual tampering with cluster resources is automatically reverted.

Repository Structure

.
├── .gitignore
├── LICENSE
├── Makefile                                    # Automation CLI (deploy, destroy, reinstall, promote)
├── README.md                                   # Master architectural blueprint & documentation
├── deploy.sh                                   # 1-Click platform deployment script
├── destroy.sh                                  # Clean teardown script
├── reinstall.sh                                # Full wipe and fresh redeployment
├── config/
│   ├── clusters.yaml                           # Multi-cluster topologies (DEV, STG, PRD) & .substvar profiles
│   └── environments.env                        # Environment variables & platform domain endpoints
├── argocd-apps/
│   ├── root-app-of-apps.yaml                   # ArgoCD Root App-of-Apps bootstrapper
│   ├── applicationset-clusters.yaml            # Multi-cluster matrix generator for BWCE
│   ├── applicationset-pull-request-preview.yaml# Dynamic PR preview environment generator for BWCE
│   ├── applicationset-git-branches.yaml        # Git branch/tag tracking generator
│   └── apps/                                   # TargetRevision manifests (dev, staging, prod)
├── helm/
│   ├── jenkins/
│   │   ├── Chart.yaml
│   │   ├── values.yaml                         # Standard Jenkins Helm values
│   │   ├── values-openshift.yaml               # OpenShift 4.20+ hardened values (restricted-v2)
│   │   └── plugins.txt                         # Lean plugins (NO git-parameter plugin)
│   ├── argocd/
│   │   └── values-argocd-3.5.yaml              # ArgoCD 3.5 Helm values with ApplicationSets
│   └── observability/
│       └── datadog-agent-values.yaml           # Datadog Agent Helm values (APM, DogStatsD, Logs)
├── jcasc/
│   ├── jenkins-jcasc.yaml                      # JCasC Master configuration
│   ├── pod-templates.yaml                      # BWCE builder agent pod templates (restricted-v2)
│   └── github-app-credentials.yaml             # GitHub App credentials for SCM & GitOps PRs
├── jenkinsfiles/
│   └── ci/
│       └── Jenkinsfile.app-bwce                # Pure CI: BW6 EAR build, Trivy, Syft, Cosign, GitOps commit
├── jobdsl/
│   ├── seed-job.groovy                         # Master seed job folder provisioner
│   └── pipelines-ci.groovy                     # Multibranch Pipeline Job DSL (Zero parameters)
├── observability/
│   ├── dashboards/                             # Datadog APM, CI Visibility, and GitOps dashboards
│   └── monitors/                               # Datadog monitors & alert rules
├── sample-apps/
│   ├── tibco-bwce-order-service/               # Reference TIBCO BWCE Cloud-Native Microservice
│   │   ├── Dockerfile                          # EAR overlay on tibco/bwce:2.9.2 base image
│   │   ├── pom.xml                             # bw6-maven-plugin configuration
│   │   ├── META-INF/                           # DEV.substvar, STAGING.substvar, PROD.substvar profiles
│   │   ├── Processes/                          # OrderProcess.bwp, HealthCheckProcess.bwp
│   │   ├── rollout/                            # Argo Rollouts Canary & Datadog AnalysisTemplate
│   │   └── k8s/                                # Base & overlays for dev, staging, prod
│   ├── tibco-bwce-customer-api/                # Secondary BWCE REST microservice
│   └── gitops-manifests/                       # Centralized GitOps configuration repository
├── scripts/
│   ├── deploy.sh, destroy.sh, reinstall.sh
│   ├── gitops-promote.sh                       # CLI helper for GitOps release promotion
│   ├── deploy-datadog-agent.sh                 # Datadog Agent deployment script
│   └── setup-argocd-clusters.sh                # Multi-cluster ArgoCD secret registration
├── security/
│   ├── openshift-image-signature-policy.yaml   # Cosign image signature verification policy
│   └── external-secrets-operator/              # Vault / ESO zero-trust secrets integration
└── shared-library/
    └── vars/
        ├── bwceEarBuild.groovy                 # EAR build step
        ├── cosignSign.groovy                   # Cosign container signing step
        ├── sbomGenerate.groovy                 # Syft SBOM generation step
        └── gitopsCommit.groovy                 # Automated GitOps commit / PR step

Quick Start: 1-Click Automated Deployment

Prerequisites

Red Hat OpenShift Container Platform (OCP) 4.20+ or Kubernetes 1.31+ cluster.
oc or kubectl CLI logged in with cluster-admin permissions.
helm v3.14+ installed.

1. Deploy the Complete Platform

git clone https://github.com/nubenetes/jenkins-without-git-parameter-bwce.git
cd jenkins-without-git-parameter-bwce

# Deploy the complete TIBCO BWCE platform (Jenkins CI, ArgoCD 3.5, Datadog APM)
./deploy.sh
# or
make deploy

2. Verify Platform Endpoints

Jenkins CI Controller: https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal
ArgoCD 3.5 Server: https://argocd-server.apps.ocp-dev.nubenetes.internal
Datadog APM Dashboard: https://app.datadoghq.com/dashboard/lists

Decommissioning & Reinstallation

Clean Decommission

./destroy.sh
# or
make destroy

Full Reinstallation

./reinstall.sh
# or
make reinstall

Video Walkthroughs & Architecture References (YouTube)

Architectural deep dives, video walkthroughs, and technical shorts for jenkins-without-git-parameter-bwce, TIBCO BWCE modernization, and Datadog GitOps progressive delivery on OpenShift 4.20+ are hosted on the Nubenetes YouTube Channel (@nubenetes).

📂 Full-Length Technical Deep Dives (Architecture Masterclasses)

1. Modernize TIBCO BWCE on OpenShift: GitOps, Datadog & Argo Rollouts

🔗 Direct Link: https://www.youtube.com/watch?v=XZX2pD3XqQM
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 8:12
🏷️ Engineering Domain: Cloud-Native Modernization, CFS Quota Fix & Zero-Trust Secrets
📝 Technical Overview: Architectural blueprint for modernizing legacy TIBCO BusinessWorks Container Edition (BWCE) microservices on Red Hat OpenShift 4.20+. Explains 12-factor configuration externalization via .substvar, eliminating pod CPU limits to prevent Linux CFS quota bandwidth throttling on 64-thread engines, OpenMetrics scraping on port 8090, and automated canary rollouts with Argo Rollouts and Datadog APM.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

2. Datadog in GitOps: Full-Stack Observability & Automated Canary Rollouts

🔗 Direct Link: https://www.youtube.com/watch?v=VQKNKBGRxQM
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 7:56
🏷️ Engineering Domain: Full-Stack Observability, Jenkins CI Visibility & Argo Rollouts SLA Tripwires
📝 Technical Overview: Deep dive into using Datadog as the central nervous system for multi-cluster GitOps platforms on OpenShift 4.20+. Covers the DaemonSet architecture (port 8126 APM, port 8125 DogStatsD, JSON logs), Jenkins CI Visibility plugin for build trace correlation and agent queue bottlenecks, runtime Java APM tracing, and metric-driven progressive delivery with automated rollbacks when 5xx errors exceed 0.1% or P99 latency exceeds 250ms.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

3. Jenkins Pure GitOps: Decoupling CI from ArgoCD Multi-Cluster CD

🔗 Direct Link: https://www.youtube.com/watch?v=qntcMvzBx4w
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 7:52
🏷️ Engineering Domain: CI/CD Decoupling, SCM Blind Spot & Pull-Based GitOps for TIBCO BWCE
📝 Technical Overview: Architectural breakdown of the SCM pre-execution render paradox when orchestrating TIBCO BWCE multi-repo deployments with Jenkins git-parameter, and how shifting to pure GitOps with ArgoCD solves it. Details the transition to parameterless Jenkins pipelines and declarative GitOps synchronization with ArgoCD 3.5 ApplicationSets.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

4. TIBCO BWCE Pure GitOps: The Paradigm Shift from Jenkins Push to OpenShift Pull

🔗 Direct Link: https://www.youtube.com/watch?v=sxETOfv6k_U
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 8:30
🏷️ Engineering Domain: Pure GitOps Paradigm Shift, CI/CD Decoupling & ArgoCD 3.5 Pull Synchronization
📝 Technical Overview: Architectural masterclass analyzing the transformation from legacy Jenkins push pipelines to declarative Pure GitOps pull architectures on OpenShift 4.20+. Explains why managing deployments through Jenkins UI dropdowns creates SCM blind spots and exposes cluster credentials, and how restricting Jenkins to parameterless CI while delegating CD to ArgoCD ensures zero-trust security and SLA governance.
🛠️ Direct Links: Watch on YouTube | Edit in YouTube Studio

📂 Technical Shorts Matrix & Architecture Breakdowns

⚡ Video Shorts Matrix

#	Short Title	Architectural Domain & Focus	Duration	Action
01	How CFS Throttling Freezes TIBCO BWCE	Linux CFS Kernel Quota Throttling Eliminating pod CPU limits on 64-thread JVMs & managing quota via ResourceQuotas	`1:13`	▶️ Watch
02	How Datadog Automates Microservice Canary Rollouts	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:27`	▶️ Watch
03	How Pure GitOps Reverses Deployments	Pure GitOps Architecture In-cluster ArgoCD pulling state vs fragile external push scripts	`1:13`	▶️ Watch
04	The Shift to Pure GitOps Deployments	Eliminating UI Deploy Buttons Replacing Jenkins UI dropdowns with Git pull requests & ArgoCD synchronization	`1:23`	▶️ Watch
05	How Pure GitOps Replaces Jenkins Push	Zero-Trust CI/CD Security Stripping cluster admin credentials from Jenkins & relying on in-cluster ArgoCD pull	`1:19`	▶️ Watch
06	Cómo funciona un despliegue Canary automatizado 🇪🇸	Despliegue Progresivo y SLA en Tiempo Real División de tráfico 20/80 con Argo Rollouts y rollbacks automáticos con Datadog APM	`1:13`	▶️ Watch
07	How Canary Rollouts Catch Bad Code	Progressive Delivery & SLA Tripwires Argo Rollouts 20/80 traffic split with Datadog real-time 5xx/latency rollbacks	`1:18`	▶️ Watch

1. How CFS Throttling Freezes TIBCO BWCE

🔗 Direct Link: https://www.youtube.com/shorts/XyKAGxQScVo
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:13
🏷️ Engineering Domain: Linux Kernel cgroups, CFS Quota Throttling & 64-Thread JVM Sizing
📝 Technical Overview: Explains the Linux CFS Quota throttling trap on multi-threaded (64 threads) TIBCO BWCE containers: why pod-level CPU limits cause kernel freezes and latency spikes despite idle node CPU, and why capacity must be managed at the namespace level via OpenShift ResourceQuotas.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

2. How Datadog Automates Microservice Canary Rollouts

🔗 Direct Link: https://www.youtube.com/shorts/RPtczCFl2vU
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:27
🏷️ Engineering Domain: Progressive Delivery, Argo Rollouts & Datadog SLA Tripwire
📝 Technical Overview: How Argo Rollouts and Datadog APM automate canary validation for critical microservices: routing 20% traffic, evaluating live SLA thresholds (error rate under 0.1%, latency under 250ms), and triggering instant rollbacks if latency degrades.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

3. How Pure GitOps Reverses Deployments

🔗 Direct Link: https://www.youtube.com/shorts/0-NIxNk7cuM
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:13
🏷️ Engineering Domain: Pure GitOps Architecture & Pull Model vs Push Scripts
📝 Technical Overview: How GitOps completely inverts traditional deployment architecture: replacing fragile external push scripts with an internal ArgoCD controller pulling state from Git without exposing cluster credentials.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

4. The Shift to Pure GitOps Deployments

🔗 Direct Link: https://www.youtube.com/shorts/iKTgIsbQCcQ
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:23
🏷️ Engineering Domain: Eliminating UI Deploy Buttons & SCM Blind Spot Resolution
📝 Technical Overview: Why leading platform engineering teams eliminate manual UI deploy buttons, moving from fragile multi-repo parameter dropdowns to webhook-triggered CI and pull-based ArgoCD synchronization.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

5. How Pure GitOps Replaces Jenkins Push

🔗 Direct Link: https://www.youtube.com/shorts/akk2ZJrK_Io
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:19
🏷️ Engineering Domain: Zero-Trust CI/CD Security, Credential Elimination & Pull Synchronization
📝 Technical Overview: Details why legacy Jenkins push models expose critical cluster-admin credentials to CI agents and struggle with UI parameter loading latency. Explains how Pure GitOps strips Jenkins of deployment power, restricting it to container builds while in-cluster ArgoCD pulls state securely.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

6. Cómo funciona un despliegue Canary automatizado

🔗 Direct Link: https://www.youtube.com/shorts/DgkKpSksmjQ
🌐 Origin Language: Spanish 🇪🇸 (Subtitles in 20+ languages)
⏱️ Duration: 1:13
🏷️ Engineering Domain: Despliegue Progresivo, Argo Rollouts y Análisis de Métricas en Tiempo Real con Datadog APM
📝 Technical Overview: Explica cómo los despliegues progresivos Canary protegen microservicios críticos en OpenShift. Detalla la división de tráfico (20% canary / 80% estable) y la evaluación automatizada de SLAs con Datadog APM (tasa de error inferior al 0.1% y latencia P99 inferior a 250ms), ejecutando un rollback automático en milisegundos si se detectan anomalías.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

7. How Canary Rollouts Catch Bad Code

🔗 Direct Link: https://www.youtube.com/shorts/chEnuYwsRgQ
🌐 Origin Language: English (Subtitles in 20+ languages)
⏱️ Duration: 1:18
🏷️ Engineering Domain: Progressive Delivery, Argo Rollouts & Datadog SLA Tripwire
📝 Technical Overview: Demonstrates how automated canary rollouts protect production microservices from breaking releases. Details traffic splitting with Argo Rollouts (routing 20% live traffic to the canary and 80% to stable) and continuous Datadog APM SLA verification (5xx error rate under 0.1%, P99 latency below 250ms), executing instant automated rollbacks in milliseconds if anomalies are detected.
🛠️ Direct Links: Watch Short | Edit in YouTube Studio

References & Standards

TIBCO BusinessWorks™ Container Edition:
- TIBCO BWCE Official Documentation: https://docs.tibco.com/
- TIBCO BW6 Maven Plugin: https://github.com/TIBCOSoftware/bw6-plugin-maven
GitOps & Cloud-Native Standards:
- OpenGitOps Principles & Standards: https://opengitops.net/
- ArgoCD 3.5 Official Documentation: https://argo-cd.readthedocs.io/en/stable/
- Argo Rollouts Progressive Delivery: https://argoproj.github.io/argo-rollouts/
Datadog APM & Observability:
- Datadog Java APM Tracer: https://docs.datadoghq.com/tracing/setup_overview/setup/java/
- Datadog CI Visibility: https://docs.datadoghq.com/continuous_integration/
Supply Chain Security & Attestation (SLSA Level 3):
- Sigstore Cosign Container Signing: https://docs.sigstore.dev/cosign/overview/
- Anchore Syft SBOM Generator: https://github.com/anchore/syft
- Aqua Security Trivy Scanner: https://trivy.dev/
Red Hat OpenShift 4.20+ Hardening:
- OpenShift Container Platform Security Context Constraints: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/authentication_and_authorization/managing-pod-security-policies

Name		Name	Last commit message	Last commit date
argocd-apps		argocd-apps
config		config
helm		helm
jcasc		jcasc
jenkinsfiles/ci		jenkinsfiles/ci
jobdsl		jobdsl
observability		observability
sample-apps		sample-apps
scripts		scripts
security		security
shared-library		shared-library
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
deploy.sh		deploy.sh
destroy.sh		destroy.sh
reinstall.sh		reinstall.sh

Latest commit

History

Folders and files

Repository files navigation

🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform on OpenShift 4.20+ (Pure GitOps)

⚠️ AI Generation & Model Review Notice

🔗 Architecture Paradigm & Linked Repositories

🗺️ Quick Navigation Map

🧭 Repository Architecture Blueprint

🎬 AI-Generated Multimedia Series (YouTube)

📽️ Full-Length Technical Deep Dives (Architecture Masterclasses)

⚡ Video Shorts Matrix

📑 Table of Contents

Executive Summary & Architectural Paradigm Shift

In-Depth Architectural Comparison: Push vs. Pull Model for TIBCO BWCE

0. The Architectural Evolution: From Two-Pipeline CI/CD Hand-off to Pure GitOps for BWCE

📊 Three-Way Architecture Comparison Matrix for TIBCO BWCE

⚖️ Decoupled Push (Pattern 2) vs. Pure GitOps Pull Implementation Analysis for BWCE

📋 Key Architectural & Implementation Differences:

💻 Code Walkthrough in Pure GitOps BWCE CI Pipeline:

1. Comprehensive Comparison Matrix: Jenkins Git Parameter vs. Pure GitOps for BWCE

2. The Root Cause of Jenkins SCM Friction in BWCE Builds

3. How Git & ArgoCD Solve BWCE Parameterization Natively

4. Why There is No jenkins-without-git-parameter-bwce-global-vars Repository

5. Decoupled Architecture: Docker Images vs. Environment Variables & Placeholders

1. Lifecycle Decoupling (Artifact vs. Configuration in BWCE)

2. Why This Repository Uses a Unified Platform Monorepo

3. Platform Monorepo Blueprint vs. Enterprise Polyrepo GitOps Matrix

6. Enterprise Developer Portals & Governance: Backstage IDP & ServiceNow / Jira ITSM Integration

1. The Paradigm Shift: From Jenkins API Trigger to GitOps Declarative Governance

2. Enterprise Sequence Workflow: ServiceNow / Jira ITSM & ArgoCD

3. Backstage IDP & ServiceNow Integration Details

4. Comparison Matrix: ITSM & Backstage Integration (Push vs. Pure GitOps)

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via .substvar

2. Engine Sizing & JVM Performance Tuning

Datadog Full-Stack Observability Architecture

1. Jenkins CI/CD Visibility Plugin

2. TIBCO BWCE Java APM Tracer & DogStatsD

3. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

Comprehensive Mermaid Architecture Diagrams & Workflows

1. Parameter Flow Comparison (Jenkins Proxy vs. Pure GitOps)

📋 Key Contrasts (Proxy vs. Native):

2. Legacy Global-Vars vs. Pure GitOps Repository Topology

📋 Repository Decoupling Highlights:

3. End-to-End Multi-Cluster Platform Topology

📋 Architectural Breakdown & Workflow Steps:

4. Jenkins SCM Pre-Execution Lifecycle Blindspot

📋 Why Git Parameter Fails for BWCE:

5. Side-by-Side Flow Comparison: Push vs. Pull

📋 Key Contrasts (Push vs. Pull for BWCE):

6. Dynamic Ephemeral Pull Request (PR) Preview Environments for BWCE

📋 Ephemeral Preview Lifecycle Steps:

7. Automated BWCE CI -> GitOps Promotion Sequence

📋 Continuous Promotion Execution Chain:

8. ArgoCD Multi-Cluster Matrix Reconciliation Engine for BWCE

📋 Matrix Generation Mechanics:

9. Zero-Trust Security & RBAC Boundary Architecture

📋 Zero-Trust Security & Isolation Principles:

10. Progressive Delivery with Argo Rollouts & Datadog SLA

📋 Canary Rollout & Datadog Analysis Steps:

11. Full-Stack Observability & Trace Context Propagation with Datadog

📋 End-to-End Tracing Journey:

Which Pattern is Easier, Recommended, and Why?

🏆 Final Recommendation: Pure GitOps (jenkins-without-git-parameter-bwce)

Repository Structure

Quick Start: 1-Click Automated Deployment

Prerequisites

1. Deploy the Complete Platform

2. Verify Platform Endpoints

Decommissioning & Reinstallation

Clean Decommission

Full Reinstallation

Video Walkthroughs & Architecture References (YouTube)

1. Modernize TIBCO BWCE on OpenShift: GitOps, Datadog & Argo Rollouts

2. Datadog in GitOps: Full-Stack Observability & Automated Canary Rollouts

3. Jenkins Pure GitOps: Decoupling CI from ArgoCD Multi-Cluster CD

4. TIBCO BWCE Pure GitOps: The Paradigm Shift from Jenkins Push to OpenShift Pull

⚡ Video Shorts Matrix

1. How CFS Throttling Freezes TIBCO BWCE

4. Why There is No `jenkins-without-git-parameter-bwce-global-vars` Repository

1. 12-Factor Profile Externalization via `.substvar`

🏆 Final Recommendation: Pure GitOps (`jenkins-without-git-parameter-bwce`)