Update directory structure of observability and add Loki, Otel (#42)

feat: Add complete OpenTelemetry Observability Stack and Operator

This series of changes introduces a comprehensive observability solution based on the OpenTelemetry ecosystem, focusing on production-ready, security-hardened deployments.

Key changes include:
* **Feature Addition:** Deploy the full OpenTelemetry Kube Stack (v0.11.1) and the OpenTelemetry Operator (v0.98.0) for auto-instrumentation and complete telemetry collection (traces, metrics, logs).
* **Hardening:** Implement security best practices for all new components, including non-root execution, read-only filesystems, resource limits, high availability (HA), and cert-manager integration.
* **Documentation:** Update service documentation to reflect the migration to the unified 'opentelemetry-kube-stack' and enhance feature descriptions.
* **Refactoring/Fixes:** Refactor the observability setup to prevent accidental namespace deletion, fix minor configuration issues (NS labels, default values), and update the telemetry pipeline configuration (removing the metrics pipeline).
* **Other:** Add base manifests for Loki deployment and update the resource type for ID RSA management.

Resolves: OpenTelemetry operator deployment for auto-instrumentation

---------

Co-authored-by: Pratik Bandarkar <[email protected]>
Co-authored-by: Victor Palma <[email protected]>
Co-authored-by: Miguel Parada <[email protected]>

Author: 3 people

README.md

@@ -29,6 +29,7 @@ applications/
 | **kube-prometheus-stack** | Core Service | `observability` | Complete monitoring and alerting stack |
 | **metallb** | Core Service | `metallb-system` | Bare metal load balancer |
 | **olm** | Core Service | `olm` | Operator Lifecycle Manager |
+| **opentelemetry-kube-stack** | Core Service | `observability` | Complete OpenTelemetry observability stack |
 | **sealed-secrets** | Core Service | `sealed-secrets` | Encrypted secrets management |
 | **velero** | Core Service | `velero` | Cluster backup and disaster recovery |
 | **alert-proxy** | Managed Service | `rackspace` | Rackspace alert aggregation |
@@ -103,6 +104,18 @@ applications/
   - Dependency resolution
   - Automatic updates
 
+#### **opentelemetry-kube-stack**
+- **Purpose**: Complete OpenTelemetry observability stack for Kubernetes
+- **Source**: OpenTelemetry Kube Stack Helm repository (`https://charts.opentelemetry.io`)
+- **Namespace**: `observability`
+- **Features**:
+  - OpenTelemetry Operator for auto-instrumentation and collector management
+  - Pre-configured OpenTelemetry Collector for metrics, traces, and logs
+  - Automatic service discovery and monitoring
+  - Multi-language auto-instrumentation support (Java, Node.js, Python, .NET, Go)
+  - Integration with Prometheus and Jaeger for complete observability
+  - Custom resource definitions for OpenTelemetry configuration
+
 #### **sealed-secrets**
 - **Purpose**: Encrypted secrets management
 - **Namespace**: `sealed-secrets`

applications/base/services/kube-prometheus-stack/README.md renamed to applications/base/services/observability/kube-prometheus-stack/README.md

@@ -2,7 +2,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - "namespace.yaml"
   - "source.yaml"
   - "helmrelease.yaml"
 secretGenerator:

applications/base/services/kube-prometheus-stack/helm-values/alerting-rules-overrides.yaml renamed to applications/base/services/observability/kube-prometheus-stack/helm-values/alerting-rules-overrides.yaml

@@ -0,0 +1,15 @@
+# Loki – Base Configuration
+
+This directory contains the **base manifests** for deploying [Grafana Loki](https://grafana.com/oss/loki/), a horizontally-scalable, highly-available log aggregation system designed for cloud-native environments.
+It is designed to be **consumed by cluster repositories** as a remote base, allowing each cluster to apply **custom overrides** as needed.
+
+**About Grafana Loki:**
+
+- Provides a **cost-effective log aggregation solution** optimized for storing and querying logs from Kubernetes clusters and applications.
+- Deployed in **Simple Scalable mode** with separate read and write paths for high availability and horizontal scaling.
+- Integrates natively with **OpenTelemetry** for log collection using OTLP protocol, eliminating the need for additional log shippers.
+- Indexes only metadata (labels) rather than full-text, resulting in **significantly lower storage costs** compared to traditional solutions.
+- Queries logs using **LogQL**, a query language similar to PromQL, enabling powerful filtering and aggregation.
+- Supports **multi-tenancy**, **retention policies**, and **compaction** for efficient long-term log storage.
+- Automatically integrates with **Grafana** for unified visualization of logs alongside metrics and traces.
+- Commonly used for troubleshooting application issues, audit logging, security analysis, and operational insights.