CSPL-3723: Initial implementation for GenAI deployment with AIPlatform & AIService controllers

[vivekr-splunk]

This PR introduces the foundational code to enable GenAI deployments using the new AIPlatform and AIService custom resources. It focuses on supporting SAIA (Splunk AI Assistant) initially, while keeping the design extensible to add SECA and future features.

---

Key Highlights

✅ AIPlatform Custom Resource & Controller

• Defines the AIPlatform Spec and Status fields to manage the end-to-end lifecycle of GenAI workloads.
• Implements the AIPlatform controller to reconcile all required services and components.

✅ AIService Custom Resource & Controller

• Introduces AIService as an internal component managed by AIPlatform.
• Enables modular feature deployment, starting with SAIA, with future extensibility to SECA.

✅ Supporting Services & Utilities

• Added helper services and functions for AI feature reconciliation, TLS, sidecar configurations, and resource lifecycle management.
• Clear abstractions to separate SAIA-specific logic from core platform orchestration.

✅ Unit Tests

• Comprehensive unit test coverage for AIPlatform and AIService reconciliation logic.
• Mocks/fake clients used to validate CR lifecycle, status updates, and error handling.

✅ Manual Validation

• Manually tested with an AIPlatform CR to verify that:

  • The AI Platform deployment is successfully created
  • Associated services and sidecars (e.g., Weaviate, Ray, etc.) are deployed correctly

✅ Documentation

• Initial developer documentation added to explain CRD specs, deployment flow, and usage.

---

What’s Not in This PR

• Integration Tests are still pending and will be added in a follow-up PR to validate end-to-end workflows across multiple controllers.

---

How to Test

1. Deploy the CRDs for AIPlatform and AIService.
2. Apply a sample AIPlatform CR with SAIA enabled.
3. Verify the created resources (Deployments, Services, ConfigMaps, StatefulSets).
4. Check the controller logs to ensure no reconciliation errors.

---

Next Steps

• Add integration test cases for full-stack validation.
• Extend feature support for SECA in future iterations.
• Enhance observability and metrics for AI workloads.

[Copilot]

Pull Request Overview

This PR introduces functionality for worker group creation in the Ray builder along with a large number of supporting components and configuration files across the operator, including new storage clients, helm chart samples, RBAC and scorecard configurations. Key changes include:

• Implementation of a GenerateWorkerGroups function that groups model specifications into Ray worker groups.
• Addition of extensive controller reconciler logic, sidecar injections, and configuration management for the Splunk AI Operator.
• New test suites, helm charts, and RBAC configuration files to support a full, production-ready deployment.

Reviewed Changes

Copilot reviewed 87 out of 91 changed files in this pull request and generated 2 comments.

File	Description
pkg/ai/raybuilder/workergroup.go	Implements GenerateWorkerGroups to aggregate models into worker groups.
pkg/ai/raybuilder/configmap_serve.go	Updates serve config processing with storage client instantiation.
pkg/ai/raybuilder/builder.go	Contains RayService build logic and worker/head pod templates.
(Other new files)	Includes various reconciler, RBAC, scorecard, helm and test files.

Comments suppressed due to low confidence (1)

pkg/ai/raybuilder/builder.go:426

• The 'GPU_TYPE' environment variable is hard-coded to 'L40S' in the worker template; consider deriving this value dynamically from the model or node configuration to support heterogeneous GPU types.

			Env: []corev1.EnvVar{