diff --git a/BRANCH_CHANGES_ANALYSIS.md b/BRANCH_CHANGES_ANALYSIS.md deleted file mode 100644 index 8c36c8e..0000000 --- a/BRANCH_CHANGES_ANALYSIS.md +++ /dev/null @@ -1,179 +0,0 @@ -# Branch Changes Analysis: create-cluster-fix - -## Summary -This analysis examines changes in the `create-cluster-fix` branch and their impact on tests in the `testkit_v2/tests/` folder (excluding `00_healthcheck_test.go`). - -## Changes in the Branch - -### 1. Modified Files - -#### `testkit_v2/util/kube_vm_cluster.go` (Major Changes) -- **Added `ensureNodesReady()` function**: New function that waits for all nodes to be ready before checking module readiness - - Uses `NodesReadyTimeout = 600` seconds (10 minutes) - - Checks node Ready condition before proceeding - -- **Completely rewrote `ensureClusterReady()` function**: - - **Before**: Only checked `sds-node-configurator` daemonset readiness - - **After**: Now checks modules in a specific dependency order: - 1. `snapshot-controller` module (deployment readiness) - 2. `sds-local-volume` module (deployment + daemonset readiness) - 3. `sds-node-configurator` module (daemonset readiness) - - Added helper functions: `checkDeploymentReady()` and `checkDaemonSetReady()` - - More thorough validation of daemonset pods (checks all pods are running) - -- **Added `ensureNodesReady()` call** in `ClusterCreate()` before `ensureClusterReady()` - -#### `testkit_v2/util/kube_deckhouse_modules.go` -- Added constants for `snapshot-controller` and `sds-local-volume` modules -- Increased `ModuleReadyTimeout` from 600 to 720 seconds (12 minutes) - -#### `testkit_v2/data/resources.yml.tpl` -- Added `snapshot-controller` module configuration: - - `ModulePullOverride` for snapshot-controller - - `ModuleConfig` to enable snapshot-controller - -#### Other Files -- `.gitignore`: Added generated config files (no test impact) -- `testkit_v2/runme.sh`: New script (no test impact) - -## Impact on Tests - -### Affected Tests - -**ALL tests that use `util.EnsureCluster("", "")` with `HypervisorKubeConfig` set will be affected** because: - -1. `EnsureCluster()` calls `ClusterCreate()` when `HypervisorKubeConfig != ""` -2. `ClusterCreate()` calls the modified `ensureClusterReady()` function -3. The new `ensureClusterReady()` has stricter requirements and different behavior - -### Specific Test Files Affected - -#### ✅ `01_sds_nc_test.go` - **AFFECTED** -- **Impact**: HIGH -- Uses `util.EnsureCluster("", "")` which triggers cluster creation -- Also uses `util.EnsureCluster(util.HypervisorKubeConfig, "")` for hypervisor operations -- **Changes affect**: Cluster initialization will now wait for snapshot-controller and sds-local-volume modules before proceeding -- **Potential issues**: - - Tests may take longer to start (additional module checks) - - Tests may fail if snapshot-controller or sds-local-volume modules are not properly configured - - Node readiness check added before module checks - -#### ✅ `03_sds_lv_test.go` - **AFFECTED** -- **Impact**: MEDIUM -- Uses `util.EnsureCluster("", "")` for cluster operations -- **Changes affect**: Cluster initialization with stricter module readiness checks -- **Potential issues**: - - Tests may fail if required modules are not ready - - Longer initialization time - -#### ✅ `05_sds_node_configurator_test.go` - **AFFECTED** -- **Impact**: HIGH -- Uses `util.EnsureCluster("", "")` extensively -- Also uses `util.EnsureCluster(util.HypervisorKubeConfig, "")` for hypervisor operations -- **Changes affect**: - - All LVM thick/thin tests depend on cluster being ready - - Module readiness checks are now more comprehensive -- **Potential issues**: - - Tests may fail if snapshot-controller is not available - - Tests may fail if sds-local-volume module is not properly configured - - Node readiness validation added - -#### ✅ `99_finalizer_test.go` - **AFFECTED** -- **Impact**: LOW -- Uses `util.EnsureCluster("", "")` -- **Changes affect**: Cluster initialization -- **Potential issues**: Minimal - this is a cleanup test - -#### ✅ `data-exporter/base_test.go` - **AFFECTED** -- **Impact**: MEDIUM -- Uses `util.EnsureCluster("", "")` -- **Changes affect**: Cluster initialization before PVC creation -- **Potential issues**: - - PVC creation may fail if storage modules are not ready - - Tests may take longer to initialize - -#### ✅ `tools.go` - **AFFECTED** -- **Impact**: MEDIUM -- Contains helper functions used by other tests -- Uses `util.EnsureCluster(util.HypervisorKubeConfig, "")` for hypervisor operations -- **Changes affect**: Helper functions that create/cleanup resources - -## Key Behavioral Changes - -### 1. Module Readiness Order -**Before**: Only checked `sds-node-configurator` daemonset -```go -// Old behavior - simple check -dsNodeConfigurator, err := cluster.GetDaemonSet("d8-sds-node-configurator", "sds-node-configurator") -if int(dsNodeConfigurator.Status.NumberReady) < len(VmCluster) { - return fmt.Errorf("sds-node-configurator ready: %d of %d", ...) -} -``` - -**After**: Checks modules in dependency order with comprehensive validation -```go -// New behavior - sequential checks -1. Check snapshot-controller deployment -2. Check sds-local-volume deployment + daemonset -3. Check sds-node-configurator daemonset (with pod validation) -``` - -### 2. Node Readiness Check -**New**: Added `ensureNodesReady()` check before module readiness -- Waits up to 10 minutes for all nodes to be in Ready state -- Validates node conditions before proceeding - -### 3. Timeout Changes -- `ModuleReadyTimeout`: 600s → 720s (20% increase) -- Added `NodesReadyTimeout`: 600s (new) - -### 4. Daemonset Validation -**Enhanced**: Now validates: -- Desired == Current == Ready counts -- All pods are in Running phase -- Pod count matches expected - -## Potential Issues & Recommendations - -### ⚠️ Critical Issues - -1. **Missing Module Dependencies** - - If `snapshot-controller` module is not properly configured in the cluster, ALL tests will fail - - If `sds-local-volume` module is not ready, tests will fail - - **Action**: Ensure `resources.yml.tpl` changes are applied to test clusters - -2. **Timeout Increases** - - Tests may take significantly longer to initialize - - Total timeout could be: 10min (nodes) + 12min (modules) = 22 minutes worst case - - **Action**: Consider if test timeouts need adjustment - -3. **Stricter Validation** - - Tests that previously passed with partially ready modules may now fail - - **Action**: Review test expectations and module configurations - -### ✅ Positive Changes - -1. **Better Dependency Management**: Ensures modules are ready in correct order -2. **More Robust Validation**: Checks all pods are running, not just daemonset status -3. **Node Readiness**: Ensures nodes are ready before checking modules - -## Testing Recommendations - -1. **Run all affected tests** to verify they still pass with new cluster readiness checks -2. **Monitor test execution time** - ensure timeouts are sufficient -3. **Verify module configurations** - ensure snapshot-controller and sds-local-volume are properly configured -4. **Check for flaky tests** - stricter checks may reveal timing issues - -## Files to Review - -- `testkit_v2/tests/01_sds_nc_test.go` - High impact -- `testkit_v2/tests/03_sds_lv_test.go` - Medium impact -- `testkit_v2/tests/05_sds_node_configurator_test.go` - High impact -- `testkit_v2/tests/99_finalizer_test.go` - Low impact -- `testkit_v2/tests/data-exporter/base_test.go` - Medium impact -- `testkit_v2/tests/tools.go` - Medium impact - -## Conclusion - -**All tests in the `tests/` folder (except `00_healthcheck_test.go`) are potentially affected** by these changes because they all use `EnsureCluster()` which triggers `ClusterCreate()` when `HypervisorKubeConfig` is set. The changes make cluster initialization more robust but also more strict, which could cause previously passing tests to fail if module dependencies are not properly configured. - diff --git a/e2e-tests/ARCHITECTURE_ANALYSIS.md b/e2e-tests/ARCHITECTURE_ANALYSIS.md new file mode 100644 index 0000000..f0c5451 --- /dev/null +++ b/e2e-tests/ARCHITECTURE_ANALYSIS.md @@ -0,0 +1,1131 @@ +# Architecture Analysis and Refactoring Plan + +## Executive Summary + +This document provides a deep analysis of the current `testkit_v2` codebase structure and proposes a clear, modular architecture to replace the current "pasta code" implementation. The analysis covers: + +1. Current structure and dependencies +2. Architectural problems identified +3. Proposed target architecture +4. Detailed refactoring plan with step-by-step migration strategy + +--- + +## 1. Current Structure Analysis + +### 1.1 Package Structure + +**Critical Finding**: All code is currently in a single package `integration`: +- `testkit_v2/tests/*` - All test files +- `testkit_v2/util/*` - All utility files + +This monolith package design causes: +- No encapsulation boundaries +- Global state scattered across files +- Hidden circular dependencies +- Difficulty in testing components in isolation +- Hard to understand code flow and dependencies + +### 1.2 File Organization + +#### Test Files (`testkit_v2/tests/`) +``` +tests/ +├── 00_healthcheck_test.go # Basic cluster health checks +├── 01_sds_nc_test.go # LVG (LVM Volume Group) operations +├── 03_sds_lv_test.go # PVC (Persistent Volume Claim) operations +├── 05_sds_node_configurator_test.go # Comprehensive LVM tests (thick/thin) +├── 99_finalizer_test.go # Cleanup tests +├── tools.go # Shared test utilities +└── data-exporter/ + └── base_test.go # Base test for data exporter feature +``` + +#### Utility Files (`testkit_v2/util/`) +``` +util/ +├── env.go # Environment config, flags, cluster types +├── filter.go # Filter/Where interfaces +├── kube_cluster_definitions.go # Cluster definition types (NEW) +├── kube_cluster.go # Cluster singleton/cache +├── kube_deckhouse_modules.go # Deckhouse module management +├── kube_deploy.go # Deployment/Service operations +├── kube_modules.go # Custom CRDs (SSHCredentials, StaticInstance) +├── kube_node.go # Node operations, execution +├── kube_secret.go # SSH credentials CRUD +├── kube_storage.go # Storage (BD, LVG, PVC, SC) +├── kube_tester.go # Test execution helpers +├── kube_vm_cluster.go # VM cluster creation, Deckhouse install +├── kube_vm.go # VM, VD, VMBD operations +├── kube.go # Core Kubernetes client setup +├── log.go # Logging utilities +├── ssh.go # SSH client operations +└── tools.go # Utility functions (retry, random) +``` + +### 1.3 Dependency Graph + +``` +Tests (integration package) + └─> util package (imported as "github.com/deckhouse/sds-e2e/util") + └─> Actually same package! Only different directory structure + +Current Dependencies: +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +kube_cluster.go (singleton/cache) + ├─> env.go (envInit, global vars) + ├─> kube.go (InitKCluster) + └─> ssh.go (GetSshClient, tunnel creation) + +kube.go (core client setup) + ├─> kube_modules.go (D8SchemeBuilder) + └─> Multiple Kubernetes API imports + +kube_storage.go (storage operations) + ├─> kube.go (KCluster type) + ├─> filter.go (filters) + └─> tools.go (RetrySec) + +kube_node.go (node operations) + ├─> kube.go (KCluster type) + ├─> kube_modules.go (StaticInstance CRD) + ├─> filter.go (filters) + └─> ssh.go (ExecNodeSsh) + +kube_vm_cluster.go (cluster creation) + ├─> env.go (global vars) + ├─> kube.go (InitKCluster) + ├─> kube_vm.go (VM operations) + ├─> kube_node.go (AddStaticNodes) + ├─> ssh.go (SSH operations) + └─> tools.go (retry utilities) + +kube_vm.go (VM operations) + ├─> kube.go (KCluster type) + ├─> filter.go (filters) + └─> tools.go (hashMd5) + +All files → env.go (global state!) +All files → log.go (logging) +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +``` + +### 1.4 Major Architectural Problems + +#### Problem 1: Global State Everywhere +- `env.go` contains ~50 global variables +- Package-level variables in multiple files (`clrCache`, `mx`, etc.) +- No dependency injection +- Hard to test in isolation +- Race conditions possible + +#### Problem 2: God Object (`KCluster`) +- `KCluster` struct has 60+ methods +- Violates Single Responsibility Principle +- Methods span multiple domains: + - Kubernetes API operations + - Node management + - Storage operations + - VM operations + - Module management + - Deployment management + +#### Problem 3: Mixed Concerns +- Business logic mixed with infrastructure +- Test utilities mixed with production code +- Configuration mixed with execution +- No clear separation of layers + +#### Problem 4: Poor Encapsulation +- Everything in one package = no private boundaries +- Internal implementation details exposed +- Can't hide complexity behind interfaces + +#### Problem 5: Circular Dependencies (Hidden) +- Files import each other indirectly +- Hidden cycles through globals +- `env.go` → everything, everything → `env.go` + +#### Problem 6: Testing Difficulties +- Can't mock dependencies (globals) +- Hard to create isolated test scenarios +- Test files use same package = can access internals incorrectly + +--- + +## 2. Target Architecture + +### 2.1 Package Structure + +``` +testkit_v2/ +├── cmd/ +│ └── runner/ # Test runner CLI (optional, for future) +│ +├── internal/ # Internal packages (not importable) +│ ├── config/ # Configuration management +│ │ ├── env.go # Environment variables +│ │ ├── flags.go # CLI flags +│ │ ├── cluster_types.go # Cluster type definitions +│ │ └── images.go # OS image definitions +│ │ +│ ├── cluster/ # Cluster management +│ │ ├── manager.go # Cluster manager (singleton replacement) +│ │ ├── client.go # Kubernetes client factory +│ │ └── types.go # Cluster types +│ │ +│ ├── kubernetes/ # Kubernetes API operations +│ │ ├── core/ # Core K8s resources +│ │ │ ├── namespace.go +│ │ │ ├── pod.go +│ │ │ ├── node.go +│ │ │ └── service.go +│ │ ├── apps/ # Apps resources +│ │ │ ├── deployment.go +│ │ │ └── daemonset.go +│ │ ├── storage/ # Storage resources +│ │ │ ├── pvc.go +│ │ │ ├── storageclass.go +│ │ │ ├── blockdevice.go +│ │ │ └── lvmvolumegroup.go +│ │ ├── virtualization/ # VM resources +│ │ │ ├── vm.go +│ │ │ ├── vdisk.go +│ │ │ └── vmbd.go +│ │ └── deckhouse/ # Deckhouse resources +│ │ ├── modules.go +│ │ ├── nodegroups.go +│ │ └── staticinstance.go +│ │ +│ ├── infrastructure/ # Infrastructure operations +│ │ ├── ssh/ # SSH operations +│ │ │ ├── client.go +│ │ │ ├── keys.go +│ │ │ └── tunnel.go +│ │ └── vm/ # VM provisioning +│ │ ├── provider.go # Interface +│ │ └── deckhouse/ # Deckhouse VM provider +│ │ └── provider.go +│ │ +│ ├── test/ # Test framework utilities +│ │ ├── framework.go # Test framework +│ │ ├── filters.go # Filter implementations +│ │ ├── runner.go # Test runner +│ │ └── node_test_context.go # Node test context +│ │ +│ └── utils/ # Pure utility functions +│ ├── retry.go +│ ├── random.go +│ └── crypto.go +│ +├── pkg/ # Public API (importable) +│ ├── cluster/ # Public cluster interface +│ │ ├── interface.go # Cluster interface +│ │ └── config.go # Cluster config types +│ │ +│ └── testkit/ # Testkit public API +│ ├── test.go # Test helpers +│ └── fixtures.go # Test fixtures +│ +├── tests/ # Test files +│ ├── healthcheck/ +│ │ └── healthcheck_test.go +│ ├── storage/ +│ │ ├── lvg_test.go +│ │ ├── pvc_test.go +│ │ └── lvm_test.go +│ ├── node_configurator/ +│ │ └── node_configurator_test.go +│ ├── data_exporter/ +│ │ └── data_exporter_test.go +│ └── cleanup/ +│ └── finalizer_test.go +│ +├── go.mod +└── README.md +``` + +### 2.2 Layer Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ Test Layer │ +│ (tests/*.go - High-level test scenarios) │ +└──────────────────┬──────────────────────────────────────┘ + │ +┌──────────────────▼──────────────────────────────────────┐ +│ Testkit API Layer │ +│ (pkg/testkit/* - Public test helpers and fixtures) │ +└──────────────────┬──────────────────────────────────────┘ + │ +┌──────────────────▼──────────────────────────────────────┐ +│ Domain Logic Layer │ +│ (internal/cluster, internal/kubernetes/*) │ +│ - Cluster management │ +│ - Resource operations │ +│ - Business logic │ +└──────────────────┬──────────────────────────────────────┘ + │ +┌──────────────────▼──────────────────────────────────────┐ +│ Infrastructure Layer │ +│ (internal/infrastructure/*) │ +│ - SSH connections │ +│ - VM provisioning │ +│ - Network tunneling │ +└──────────────────┬──────────────────────────────────────┘ + │ +┌──────────────────▼──────────────────────────────────────┐ +│ Kubernetes API Layer │ +│ (k8s.io/client-go, controller-runtime) │ +└──────────────────────────────────────────────────────────┘ +``` + +### 2.3 Core Interfaces + +#### Cluster Interface +```go +// pkg/cluster/interface.go +type Cluster interface { + // Core operations + Name() string + Context() context.Context + + // Resource operations + Namespaces() NamespaceClient + Nodes() NodeClient + Pods() PodClient + Storage() StorageClient + Virtualization() VirtualizationClient // TODO asergunov: Is the VirtualizationClient this one? https://github.com/deckhouse/virtualization/blob/main/api/client/kubeclient/client.go#L53 + Deckhouse() DeckhouseClient + + // Lifecycle + EnsureReady(ctx context.Context) error + Close() error +} +``` + +#### Resource Clients +```go +// Internal interfaces for resource operations +type StorageClient interface { + BlockDevices() BlockDeviceClient + LVMVolumeGroups() LVMVolumeGroupClient + PersistentVolumeClaims() PersistentVolumeClaimClient + StorageClasses() StorageClassClient +} + +type NodeClient interface { + List(ctx context.Context, filters ...NodeFilter) ([]Node, error) + Get(ctx context.Context, name string) (*Node, error) + Execute(ctx context.Context, name string, cmd []string) (stdout, stderr string, err error) + // ... +} + +type VirtualizationClient interface { + VMs() VMClient + VirtualDisks() VirtualDiskClient + VirtualMachineBlockDevices() VMBDClient +} +``` + +### 2.4 Dependency Injection + +**Cluster Manager Pattern**: +```go +// internal/cluster/manager.go +type Manager struct { + config *config.Config + clusters map[string]Cluster + mu sync.RWMutex + logger logger.Logger + sshFactory ssh.Factory +} + +func NewManager(cfg *config.Config, opts ...Option) *Manager { + // Constructor with options for dependency injection +} + +func (m *Manager) GetOrCreate(ctx context.Context, configPath, name string) (Cluster, error) { + // Lazy initialization with proper error handling +} +``` + +### 2.5 Configuration Management + +**Structured Configuration**: +```go +// internal/config/config.go +type Config struct { + // Environment + TestNS string + TestNSCleanUp string + KeepState bool + + // Cluster configuration + NestedCluster NestedClusterConfig + Hypervisor HypervisorConfig + + // Feature flags + SkipOptional bool + Parallel bool + TreeMode bool + + // Logging + Verbose bool + Debug bool + LogFile string +} + +type NestedClusterConfig struct { + KubeConfig string + Host string + SSHUser string + SSHKey string + K8sPort string + StorageClass string +} +``` + +--- + +## 3. Refactoring Plan + +### 3.1 Phase 1: Foundation (Low Risk) + +**Goal**: Extract configuration and utilities without breaking existing code. + +#### Step 1.1: Extract Configuration +- [ ] Create `internal/config/` package +- [ ] Move `env.go` → `internal/config/env.go` +- [ ] Move cluster types → `internal/config/cluster_types.go` +- [ ] Move image definitions → `internal/config/images.go` +- [ ] Create `Config` struct to hold all configuration +- [ ] Create constructor `config.Load()` to initialize from flags/env +- [ ] Keep global variables temporarily with deprecation comments + +**Migration Strategy**: +```go +// Keep existing globals for backward compatibility +var TestNS = config.Current().TestNS + +// But internally use structured config +func EnsureCluster(...) { + cfg := config.Current() + // Use cfg instead of globals +} +``` + +#### Step 1.2: Extract Pure Utilities +- [ ] Create `internal/utils/` package +- [ ] Move `tools.go` utilities → `internal/utils/` +- [ ] Move `log.go` → `internal/logger/` with interface +- [ ] Create logger interface for testability +- [ ] Update all files to use logger interface + +**Files Affected**: +- `util/tools.go` → `internal/utils/retry.go`, `random.go`, `crypto.go` +- `util/log.go` → `internal/logger/logger.go` + +#### Step 1.3: Extract Filters +- [ ] Create `internal/test/filters.go` +- [ ] Move `filter.go` → `internal/test/filters.go` +- [ ] Make filters type-safe and well-documented +- [ ] Keep old imports working temporarily + +**Estimated Time**: 1-2 days +**Risk Level**: Low (internal changes, maintain compatibility) + +--- + +### 3.2 Phase 2: Extract Kubernetes Clients (Medium Risk) + +**Goal**: Separate Kubernetes API operations from business logic. + +#### Step 2.1: Create Kubernetes Client Packages +- [ ] Create `internal/kubernetes/` structure +- [ ] Extract core operations: + - `kube.go` → `internal/kubernetes/client.go` (client factory) + - `kube.go` (namespace) → `internal/kubernetes/core/namespace.go` + - `kube_node.go` → `internal/kubernetes/core/node.go` + - `kube_deploy.go` → `internal/kubernetes/apps/deployment.go` +- [ ] Extract storage operations: + - `kube_storage.go` → `internal/kubernetes/storage/*.go` + - Split into separate files per resource type +- [ ] Extract virtualization operations: + - `kube_vm.go` → `internal/kubernetes/virtualization/*.go` + +#### Step 2.2: Create Client Interfaces +- [ ] Define interfaces for each resource client +- [ ] Implement interfaces with existing code +- [ ] Update `KCluster` to use clients via composition + +**Before**: +```go +type KCluster struct { + controllerRuntimeClient ctrlrtclient.Client + goClient *kubernetes.Clientset + // ... 60+ methods directly on KCluster +} +``` + +**After**: +```go +type KCluster struct { + client kubernetes.Client + storage *storage.Client + nodes *node.Client + // ... composition instead of methods +} + +type Client struct { + controller ctrlrtclient.Client + goClient *kubernetes.Clientset + // Resource clients + namespaces NamespaceClient + nodes NodeClient + pods PodClient + storage StorageClient + // ... +} +``` + +#### Step 2.3: Update Tests Gradually +- [ ] Create wrapper functions in old package that delegate to new structure +- [ ] Update tests one by one to use new interfaces +- [ ] Remove old methods once all tests migrated + +**Migration Helper**: +```go +// In old package (temporary compatibility layer) +func (cluster *KCluster) CreateLVG(...) error { // TODO: asergunov: Maybe D8Cluster? Or Cluster interface and d8.Cluster as its implementation + return cluster.storage.LVMVolumeGroups().Create(...) +} +``` + +**Estimated Time**: 3-5 days +**Risk Level**: Medium (interface changes, needs careful testing) + +--- + +### 3.3 Phase 3: Extract Infrastructure (Medium Risk) + +**Goal**: Separate infrastructure concerns (SSH, VM provisioning). + +#### Step 3.1: Extract SSH Operations +- [ ] Create `internal/infrastructure/ssh/` package +- [ ] Move `ssh.go` → `internal/infrastructure/ssh/` +- [ ] Create SSH client factory interface +- [ ] Make SSH client mockable for tests +- [ ] Update all SSH usages to use factory + +#### Step 3.2: Extract VM Cluster Operations +- [ ] Create `internal/infrastructure/vm/` package +- [ ] Extract Deckhouse VM provider +- [ ] Move `kube_vm_cluster.go` logic → `internal/infrastructure/vm/deckhouse/` +- [ ] Create VM provider interface for extensibility +- [ ] Separate VM lifecycle from cluster operations + +**Structure**: +```go +// internal/infrastructure/vm/provider.go +type Provider interface { + CreateVM(ctx context.Context, spec VMSpec) (*VM, error) + DeleteVM(ctx context.Context, name string) error + WaitForVMReady(ctx context.Context, name string) error +} + +// internal/infrastructure/vm/deckhouse/provider.go +type DeckhouseProvider struct { + cluster Cluster + // ... +} +``` + +#### Step 3.3: Extract Cluster Creation Logic +- [ ] Move cluster creation from `kube_vm_cluster.go` +- [ ] Create `internal/cluster/builder.go` for cluster creation +- [ ] Separate concerns: VM creation, Deckhouse installation, node registration + +**Estimated Time**: 3-4 days +**Risk Level**: Medium (infrastructure changes affect tests) + +--- + +### 3.4 Phase 4: Refactor Cluster Management (High Risk) + +**Goal**: Replace singleton pattern with proper dependency injection. + +#### Step 4.1: Create Cluster Manager +- [ ] Create `internal/cluster/manager.go` +- [ ] Replace `EnsureCluster` singleton with Manager +- [ ] Implement proper lifecycle management +- [ ] Add context support for cancellation + +#### Step 4.2: Refactor KCluster to Cluster Interface +- [ ] Create `pkg/cluster/interface.go` with public Cluster interface +- [ ] Implement interface in `internal/cluster/cluster.go` +- [ ] Break up `KCluster` into smaller, focused structs +- [ ] Use composition instead of 60+ methods + +#### Step 4.3: Update All Tests +- [ ] Update test files to use new Cluster interface +- [ ] Remove dependency on singleton +- [ ] Enable dependency injection in tests + +**Before**: +```go +func TestSomething(t *testing.T) { + cluster := util.EnsureCluster("", "") // Singleton + // ... +} +``` + +**After**: +```go +func TestSomething(t *testing.T) { + ctx := context.Background() + cfg := config.Load() + manager := cluster.NewManager(cfg) + cl, err := manager.GetOrCreate(ctx, "", "") + // ... +} +``` + +**Or with test helper**: +```go +func TestSomething(t *testing.T) { + cluster := testkit.GetCluster(t) // Helper that manages lifecycle + // ... +} +``` + +**Estimated Time**: 5-7 days +**Risk Level**: High (touches all test files) + +--- + +### 3.5 Phase 5: Organize Tests (Low Risk) + +**Goal**: Organize test files into logical packages. + +#### Step 5.1: Reorganize Test Files +- [ ] Move tests into domain-specific packages: + - `tests/healthcheck/` + - `tests/storage/` + - `tests/node_configurator/` + - `tests/data_exporter/` + - `tests/cleanup/` +- [ ] Create shared test utilities in `pkg/testkit/` +- [ ] Update package names appropriately + +#### Step 5.2: Create Test Framework +- [ ] Create `internal/test/framework.go` for test helpers +- [ ] Extract common test patterns +- [ ] Create fixtures for common scenarios + +**Estimated Time**: 2-3 days +**Risk Level**: Low (mostly moving files) + +--- + +### 3.6 Phase 6: Cleanup and Documentation (Low Risk) + +**Goal**: Remove old code, add documentation, improve developer experience. + +#### Step 6.1: Remove Deprecated Code +- [ ] Remove compatibility wrappers +- [ ] Remove old package structure +- [ ] Clean up unused imports +- [ ] Remove global variables + +#### Step 6.2: Add Documentation +- [ ] Write package-level documentation +- [ ] Document public APIs +- [ ] Create architecture diagrams +- [ ] Add examples for common use cases + +#### Step 6.3: Improve Developer Experience +- [ ] Add clear error messages +- [ ] Improve logging +- [ ] Add validation +- [ ] Create helper functions for common operations + +**Estimated Time**: 2-3 days +**Risk Level**: Low (cleanup phase) + +--- + +## 4. Migration Strategy Details + +### 4.1 Compatibility Layer Approach + +During migration, maintain a compatibility layer that delegates to new implementation: + +```go +// Old location: util/kube_storage.go (temporary) +package integration + +import ( + newStorage "github.com/deckhouse/sds-e2e/internal/kubernetes/storage" +) + +func (cluster *KCluster) CreateLVG(name, nodeName string, bds []string) error { + // Delegate to new implementation + return cluster.storageClient.LVMVolumeGroups().Create( + cluster.ctx, + newStorage.LVGCreateRequest{ + Name: name, + NodeName: nodeName, + BlockDevices: bds, + }, + ) +} +``` + +This allows: +- Gradual migration of tests +- Running old and new code side-by-side +- Easy rollback if issues arise +- Zero-downtime refactoring + +### 4.2 Testing Strategy + +1. **Unit Tests First**: Test new packages in isolation +2. **Integration Tests**: Ensure new code works with existing tests +3. **Parallel Running**: Run old and new implementations in parallel +4. **Gradual Cutover**: Move tests one by one to new implementation + +### 4.3 Rollback Plan + +At each phase: +- Keep old code in place until new code is proven +- Use feature flags if needed +- Maintain compatibility layer +- Document rollback procedure + +--- + +## 5. Detailed Module Structure + +### 5.1 Configuration Module (`internal/config/`) + +``` +config/ +├── config.go # Main Config struct and Load() +├── env.go # Environment variable parsing +├── flags.go # CLI flag definitions +├── cluster_types.go # Cluster type definitions and validation +├── images.go # OS image URL definitions +└── defaults.go # Default values +``` + +**Responsibilities**: +- Configuration loading from flags, env vars, files +- Configuration validation +- Type-safe configuration access +- No business logic + +### 5.2 Cluster Module (`internal/cluster/`) + +``` +cluster/ +├── manager.go # Cluster manager (replaces EnsureCluster singleton) +├── cluster.go # Cluster implementation +├── client.go # Kubernetes client factory +├── cache.go # Cluster caching logic +└── types.go # Cluster-related types +``` + +**Responsibilities**: +- Cluster lifecycle management +- Client initialization and caching +- Context management +- No resource operations (delegates to kubernetes clients) + +### 5.3 Kubernetes Module (`internal/kubernetes/`) + +``` +kubernetes/ +├── client.go # Base client setup and scheme registration +├── core/ +│ ├── namespace.go +│ ├── node.go +│ ├── pod.go +│ └── service.go +├── apps/ +│ ├── deployment.go +│ └── daemonset.go +├── storage/ +│ ├── client.go # Storage client interface +│ ├── blockdevice.go +│ ├── lvmvolumegroup.go +│ ├── pvc.go +│ └── storageclass.go +├── virtualization/ +│ ├── client.go +│ ├── vm.go +│ ├── vdisk.go +│ ├── vmbd.go +│ └── cluster_virtual_image.go +└── deckhouse/ + ├── client.go + ├── modules.go + ├── nodegroups.go + └── staticinstance.go +``` + +**Responsibilities**: +- All Kubernetes API operations +- Resource-specific logic +- Filtering and querying +- CRUD operations +- No infrastructure concerns (SSH, VM provisioning handled elsewhere) + +### 5.4 Infrastructure Module (`internal/infrastructure/`) + +``` +infrastructure/ +├── ssh/ +│ ├── client.go # SSH client implementation +│ ├── factory.go # SSH client factory +│ ├── keys.go # SSH key generation +│ └── tunnel.go # SSH tunnel management +└── vm/ + ├── provider.go # VM provider interface + └── deckhouse/ + ├── provider.go # Deckhouse VM provider + └── installer.go # Deckhouse installation logic +``` + +**Responsibilities**: +- SSH connection management +- VM provisioning (via providers) +- Infrastructure setup +- No Kubernetes operations (uses kubernetes clients) + +### 5.5 Test Module (`internal/test/`) + +``` +test/ +├── framework.go # Test framework and helpers +├── filters.go # Filter implementations +├── runner.go # Test execution runner +├── node_context.go # Node test context +└── fixtures.go # Test fixtures +``` + +**Responsibilities**: +- Test execution utilities +- Filter implementations +- Test context management +- Node-specific test helpers + +### 5.6 Public API (`pkg/`) + +``` +pkg/ +├── cluster/ +│ ├── interface.go # Public Cluster interface +│ └── config.go # Public config types +└── testkit/ + ├── test.go # Public test helpers + └── fixtures.go # Public fixtures +``` + +**Responsibilities**: +- Public API for external consumers +- Stable interfaces +- Well-documented +- Backward compatibility guarantees + +--- + +## 6. Key Design Decisions + +### 6.1 Why Internal Packages? + +- **Encapsulation**: Internal packages cannot be imported outside the module +- **Flexibility**: Can refactor internal packages without breaking external API +- **Clear Boundaries**: Makes it obvious what is public vs private + +### 6.2 Why Composition Over Inheritance? + +- **Flexibility**: Easier to swap implementations +- **Testability**: Can mock individual components +- **Single Responsibility**: Each client has one job + +### 6.3 Why Interface-Based Design? + +- **Testability**: Easy to create mocks +- **Extensibility**: Can add new implementations +- **Dependency Inversion**: High-level code doesn't depend on low-level details + +### 6.4 Why Separate Infrastructure? + +- **Clear Boundaries**: Infrastructure is separate from business logic +- **Testability**: Can mock infrastructure in tests +- **Flexibility**: Can swap VM providers, SSH implementations, etc. + +--- + +## 7. Migration Checklist + +### Phase 1: Foundation +- [ ] Extract configuration to `internal/config/` +- [ ] Extract utilities to `internal/utils/` +- [ ] Extract filters to `internal/test/filters.go` +- [ ] Extract logging to `internal/logger/` +- [ ] All existing tests still pass + +### Phase 2: Kubernetes Clients +- [ ] Create `internal/kubernetes/` structure +- [ ] Extract all K8s operations to appropriate packages +- [ ] Create client interfaces +- [ ] Update KCluster to use composition +- [ ] All existing tests still pass + +### Phase 3: Infrastructure +- [ ] Extract SSH to `internal/infrastructure/ssh/` +- [ ] Extract VM operations to `internal/infrastructure/vm/` +- [ ] Create provider interfaces +- [ ] All existing tests still pass + +### Phase 4: Cluster Management +- [ ] Create Cluster Manager +- [ ] Create Cluster interface +- [ ] Refactor KCluster implementation +- [ ] Update all tests to use new interface +- [ ] All tests still pass + +### Phase 5: Test Organization +- [ ] Reorganize test files +- [ ] Create test framework +- [ ] Update package names +- [ ] All tests still pass + +### Phase 6: Cleanup +- [ ] Remove deprecated code +- [ ] Add documentation +- [ ] Improve error messages +- [ ] Final verification + +--- + +## 8. Benefits of New Architecture + +### 8.1 Maintainability +- **Clear Structure**: Easy to find code +- **Single Responsibility**: Each package has one job +- **Documented**: Clear purpose for each module + +### 8.2 Testability +- **Mockable**: Can mock dependencies via interfaces +- **Isolated**: Test individual components +- **Fast**: Unit tests run quickly + +### 8.3 Extensibility +- **Pluggable**: Can add new VM providers, storage backends, etc. +- **Modular**: Can add new features without touching existing code +- **Interface-Based**: New implementations satisfy existing interfaces + +### 8.4 Developer Experience +- **Clear API**: Public interfaces are well-defined +- **Better Errors**: Structured error handling +- **Documentation**: Each package is documented +- **Examples**: Common patterns documented + +### 8.5 Performance +- **Efficient**: No unnecessary allocations +- **Cached**: Client reuse via manager +- **Context-Aware**: Proper context propagation for cancellation + +--- + +## 9. Risks and Mitigations + +### Risk 1: Breaking Existing Tests +**Mitigation**: +- Maintain compatibility layer +- Gradual migration +- Extensive testing at each phase + +### Risk 2: Time Investment +**Mitigation**: +- Phased approach (can stop at any phase) +- Parallel development possible +- Each phase delivers value + +### Risk 3: Learning Curve +**Mitigation**: +- Good documentation +- Clear examples +- Code reviews and knowledge sharing + +### Risk 4: Over-Engineering +**Mitigation**: +- Start with minimum viable structure +- Add complexity only when needed +- Keep it simple + +--- + +## 10. Success Criteria + +1. **All existing tests pass** after refactoring +2. **No performance regression** (ideally improvement) +3. **Code is easier to understand** (measured by code review time) +4. **New features are easier to add** (measured by time to implement) +5. **Tests are easier to write** (measured by lines of test code) +6. **Documentation is comprehensive** (all public APIs documented) + +--- + +## 11. Next Steps + +1. **Review this document** with team +2. **Prioritize phases** based on immediate needs +3. **Create GitHub issues** for each phase +4. **Start with Phase 1** (lowest risk) +5. **Iterate and adjust** based on learnings + +--- + +## Appendix A: Current vs Proposed Structure Comparison + +### Current Structure Issues + +``` +❌ Everything in one package +❌ Global state everywhere +❌ 60+ methods on one struct +❌ Mixed concerns +❌ Hard to test +❌ Circular dependencies +``` + +### Proposed Structure Benefits + +``` +✅ Clear package boundaries +✅ Structured configuration +✅ Interface-based design +✅ Separated concerns +✅ Easy to test +✅ No circular dependencies +``` + +--- + +## Appendix B: Code Examples + +### Example 1: Using New Cluster Interface + +```go +// tests/storage/pvc_test.go +package storage + +import ( + "context" + "testing" + + "github.com/deckhouse/sds-e2e/pkg/cluster" + "github.com/deckhouse/sds-e2e/pkg/testkit" +) + +func TestPVCCreate(t *testing.T) { + ctx := context.Background() + + // Get cluster via testkit helper (manages lifecycle) + cl := testkit.GetCluster(t) + defer cl.Close() + + // Use typed client interfaces + pvc, err := cl.Storage().PersistentVolumeClaims().Create(ctx, testkit.PVCSpec{ + Name: "test-pvc", + Namespace: testkit.TestNS, + Size: "1Gi", + StorageClass: "test-lvm-thick", + }) + if err != nil { + t.Fatal(err) + } + + // Wait for ready + err = cl.Storage().PersistentVolumeClaims().WaitReady(ctx, pvc.Name, 30*time.Second) + if err != nil { + t.Fatal(err) + } +} +``` + +### Example 2: Using Configuration + +```go +// internal/config/config.go +package config + +type Config struct { + TestNS string + NestedCluster NestedClusterConfig + // ... +} + +func Load() *Config { + cfg := &Config{ + TestNS: getTestNS(), + // ... + } + return cfg +} + +// Usage +cfg := config.Load() +cluster := cluster.NewManager(cfg) +``` + +### Example 3: Mocking for Tests + +```go +// internal/kubernetes/storage/mock.go (generated) +type MockLVMVolumeGroupClient struct { + CreateFunc func(ctx context.Context, req LVGCreateRequest) error + // ... +} + +func (m *MockLVMVolumeGroupClient) Create(ctx context.Context, req LVGCreateRequest) error { + return m.CreateFunc(ctx, req) +} + +// In test +func TestLVGCreate(t *testing.T) { + mockClient := &MockLVMVolumeGroupClient{ + CreateFunc: func(ctx context.Context, req LVGCreateRequest) error { + // Test-specific behavior + return nil + }, + } + // Use mock in test +} +``` + +--- + +## Conclusion + +This architecture refactoring will transform the codebase from a monolithic "pasta code" structure into a clean, maintainable, and testable modular architecture. The phased approach minimizes risk while delivering incremental value. + +The key principles: +1. **Separation of Concerns**: Each package has one responsibility +2. **Interface-Based Design**: Easy to test and extend +3. **Dependency Injection**: No globals, proper lifecycle management +4. **Clear Boundaries**: Internal vs public API +5. **Gradual Migration**: Low risk, incremental progress + +With this structure, the codebase will be: +- **Easier to understand** (clear package organization) +- **Easier to test** (mockable interfaces) +- **Easier to extend** (modular design) +- **Easier to maintain** (single responsibility) + +Start with Phase 1 and iterate based on learnings! + diff --git a/e2e-tests/README.md b/e2e-tests/README.md new file mode 100644 index 0000000..586c400 --- /dev/null +++ b/e2e-tests/README.md @@ -0,0 +1,101 @@ +# E2E Tests + +This package contains end-to-end tests for SDS (Storage for Deckhouse Services). + +## Architecture + +The package follows a clean, modular architecture: + +- **`internal/config/`** - Configuration management (environment variables, cluster definitions, module configs) +- **`internal/cluster/`** - Cluster lifecycle management (manager, builder) +- **`internal/kubernetes/`** - Kubernetes API clients (virtualization, deckhouse modules) +- **`internal/infrastructure/`** - Infrastructure operations (SSH, VM provisioning) +- **`internal/logger/`** - Logging utilities +- **`internal/utils/`** - Utility functions (retry, crypto) +- **`pkg/cluster/`** - Public cluster interface +- **`pkg/testkit/`** - Public test helpers +- **`tests/`** - Test files using Ginkgo + +## Cluster Creation Workflow + +The cluster builder implements the following workflow: + +1. **Connect to Base Cluster**: Get kubeconfig of the base Deckhouse cluster and connect via SSH +2. **Enable Virtualization Module**: Enable Deckhouse Virtualization Platform module on base cluster +3. **Create Virtual Machines**: Create VMs as defined in cluster configuration +4. **Deploy Deckhouse**: Connect to master VM via SSH and deploy Deckhouse Kubernetes Platform +5. **Get Kubeconfig**: Retrieve kubeconfig of the nested cluster +6. **Enable Modules**: Enable and configure required modules in the nested cluster + +## Quick start - Running tests +// TODO amarkov: I strongly recommend add a full example how to run tests with all environments, arguments and commands. + +## Writing Tests + +Tests are written using Ginkgo. Keep test files simple - they should only contain test logic. Business logic is in other modules. + +Example: + +```go +var _ = Describe("Cluster Creation", func() { + var ( + ctx context.Context + baseCluster cluster.Cluster + testCluster cluster.Cluster + clusterCfg *config.DKPClusterConfig + ) + + BeforeEach(func() { + ctx = context.Background() + baseCluster, _ = testkit.GetCluster(ctx, cfg.BaseCluster.KubeConfig, "") + clusterCfg = &config.DKPClusterConfig{ + // Define your cluster configuration + } + }) + + It("should create a nested Kubernetes cluster", func() { + testCluster, err := testkit.BuildTestCluster(ctx, baseCluster, clusterCfg) + Expect(err).NotTo(HaveOccurred()) + + err = testCluster.EnsureReady(ctx) + Expect(err).NotTo(HaveOccurred()) + }) +}) +``` + +## Configuration + +Configuration is loaded from environment variables: + +- `BASE_KUBECONFIG` - Path to base cluster kubeconfig +- `BASE_SSH_HOST` - Base cluster SSH host +- `BASE_SSH_USER` - Base cluster SSH user +- `BASE_SSH_KEY` - Base cluster SSH key path +- `NESTED_KUBECONFIG` - Path for nested cluster kubeconfig +- `NESTED_SSH_HOST` - Nested cluster SSH host +- And more... See `internal/config/config.go` for full list + +## Running Tests + +```bash +go test ./tests/... -v +``` + +Or with Ginkgo: + +```bash +ginkgo ./tests/... +``` + +## Structure + +- `tests/` - Test files +- `internal/` - Internal packages (not importable outside) +- `pkg/` - Public API (importable) + +## Notes + +- The code is independent of legacy code (no imports from `legacy/`) +- Test files are simple and focus on test logic only +- Business logic is in separate modules +- The architecture allows for easy extension and testing diff --git a/e2e-tests/go.mod b/e2e-tests/go.mod new file mode 100644 index 0000000..63625c6 --- /dev/null +++ b/e2e-tests/go.mod @@ -0,0 +1,70 @@ +module github.com/deckhouse/sds-e2e-tests + +go 1.24.2 + +toolchain go1.24.3 + +require ( + github.com/deckhouse/csi-nfs/api v0.0.0-20250529065000-055714087eef + github.com/deckhouse/sds-node-configurator/api v0.0.0-20241211082010-e61684e9db31 + github.com/deckhouse/sds-replicated-volume/api v0.0.0-20241206095502-b096856636cd + github.com/deckhouse/virtualization/api v0.15.0 + github.com/go-logr/logr v1.4.2 + github.com/onsi/ginkgo/v2 v2.21.0 + github.com/onsi/gomega v1.35.1 + k8s.io/api v0.32.1 + k8s.io/apiextensions-apiserver v0.31.0 + k8s.io/apimachinery v0.32.1 + k8s.io/client-go v0.32.1 + k8s.io/utils v0.0.0-20241210054802-24370beab758 + sigs.k8s.io/controller-runtime v0.19.4 +) + +require ( + github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/emicklei/go-restful/v3 v3.11.0 // indirect + github.com/evanphx/json-patch/v5 v5.9.0 // indirect + github.com/fxamacker/cbor/v2 v2.7.0 // indirect + github.com/go-openapi/jsonpointer v0.21.0 // indirect + github.com/go-openapi/jsonreference v0.20.2 // indirect + github.com/go-openapi/swag v0.23.0 // indirect + github.com/go-task/slim-sprig/v3 v3.0.0 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang/protobuf v1.5.4 // indirect + github.com/google/gnostic-models v0.6.8 // indirect + github.com/google/go-cmp v0.6.0 // indirect + github.com/google/gofuzz v1.2.0 // indirect + github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/josharian/intern v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/mailru/easyjson v0.7.7 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/openshift/api v0.0.0-20230503133300-8bbcb7ca7183 // indirect + github.com/openshift/custom-resource-status v1.1.2 // indirect + github.com/pborman/uuid v1.2.1 // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/spf13/pflag v1.0.5 // indirect + github.com/x448/float16 v0.8.4 // indirect + golang.org/x/net v0.38.0 // indirect + golang.org/x/oauth2 v0.23.0 // indirect + golang.org/x/sys v0.31.0 // indirect + golang.org/x/term v0.30.0 // indirect + golang.org/x/text v0.23.0 // indirect + golang.org/x/time v0.7.0 // indirect + golang.org/x/tools v0.26.0 // indirect + google.golang.org/protobuf v1.35.1 // indirect + gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect + gopkg.in/inf.v0 v0.9.1 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + k8s.io/klog/v2 v2.130.1 // indirect + k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f // indirect + kubevirt.io/api v1.0.0 // indirect + kubevirt.io/containerized-data-importer-api v1.57.0-alpha1 // indirect + kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90 // indirect + sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect + sigs.k8s.io/structured-merge-diff/v4 v4.5.0 // indirect + sigs.k8s.io/yaml v1.4.0 // indirect +) diff --git a/e2e-tests/go.sum b/e2e-tests/go.sum new file mode 100644 index 0000000..e1900c8 --- /dev/null +++ b/e2e-tests/go.sum @@ -0,0 +1,399 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= +github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= +github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= +github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/deckhouse/csi-nfs/api v0.0.0-20250529065000-055714087eef h1:wuDOyamGLu2GQM89J1tk3ozSNdmW8MJurH5+hQ5HvNc= +github.com/deckhouse/csi-nfs/api v0.0.0-20250529065000-055714087eef/go.mod h1:b7i9vtLYdBEmAo0dtfEwUHu9hnnQyGuLTLy5XxD1GKA= +github.com/deckhouse/sds-node-configurator/api v0.0.0-20241211082010-e61684e9db31 h1:ZvYZNOHmRfqFp/2dy5p1dHA0EwN1kiem9iteZ0uIjE0= +github.com/deckhouse/sds-node-configurator/api v0.0.0-20241211082010-e61684e9db31/go.mod h1:ROmrnlcAdtYX8HPb0pe1qsnmISpy5FSW5fn2n67JOoE= +github.com/deckhouse/sds-replicated-volume/api v0.0.0-20241206095502-b096856636cd h1:MJzcWolbjy6SnZs9NMhvAq4KZdWRSQNIjcM9h+qGXa0= +github.com/deckhouse/sds-replicated-volume/api v0.0.0-20241206095502-b096856636cd/go.mod h1:6yz0RtbkLVJtK2DeuvgfaqBZRl5V5ax1WsfPF5pbnvo= +github.com/deckhouse/virtualization/api v0.15.0 h1:yRX6n18kK9wwO2f+8fc2s2nb1N2vYKxKb3/aSRtc9Kk= +github.com/deckhouse/virtualization/api v0.15.0/go.mod h1:t+6i4NC43RfNLqcZqkEc5vxY1ypKceqmOOKlVEq0cYA= +github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= +github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= +github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/emicklei/go-restful v2.15.0+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= +github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/evanphx/json-patch/v5 v5.9.0 h1:kcBlZQbplgElYIlo/n1hJbls2z/1awpXxpRi0/FOJfg= +github.com/evanphx/json-patch/v5 v5.9.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= +github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E= +github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= +github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg= +github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= +github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= +github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= +github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= +github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= +github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= +github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ= +github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY= +github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= +github.com/go-openapi/jsonreference v0.19.5/go.mod h1:RdybgQwPxbL4UEjuAruzK1x3nE69AqPYEJeo/TWfEeg= +github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= +github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= +github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= +github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= +github.com/go-openapi/swag v0.19.14/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= +github.com/go-openapi/swag v0.21.1/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= +github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= +github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE= +github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ= +github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= +github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= +github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo= +github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144= +github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2cUuW7uA/OeU= +github.com/googleapis/gnostic v0.5.5/go.mod h1:7+EbHbldMins07ALC74bsA81Ovc97DwqyJO1AENw9kA= +github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= +github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= +github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= +github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= +github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= +github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= +github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= +github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= +github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= +github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= +github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= +github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0= +github.com/onsi/ginkgo/v2 v2.0.0/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c= +github.com/onsi/ginkgo/v2 v2.21.0 h1:7rg/4f3rB88pb5obDgNZrNHrQ4e6WpjonchcpuBRnZM= +github.com/onsi/ginkgo/v2 v2.21.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo= +github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= +github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY= +github.com/onsi/gomega v1.18.1/go.mod h1:0q+aL8jAiMXy9hbwj2mr5GziHiwhAIQpFmmtT5hitRs= +github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4= +github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog= +github.com/openshift/api v0.0.0-20230503133300-8bbcb7ca7183 h1:t/CahSnpqY46sQR01SoS+Jt0jtjgmhgE6lFmRnO4q70= +github.com/openshift/api v0.0.0-20230503133300-8bbcb7ca7183/go.mod h1:4VWG+W22wrB4HfBL88P40DxLEpSOaiBVxUnfalfJo9k= +github.com/openshift/custom-resource-status v1.1.2 h1:C3DL44LEbvlbItfd8mT5jWrqPfHnSOQoQf/sypqA6A4= +github.com/openshift/custom-resource-status v1.1.2/go.mod h1:DB/Mf2oTeiAmVVX1gN+NEqweonAPY0TKUwADizj8+ZA= +github.com/pborman/uuid v1.2.1 h1:+ZZIw58t/ozdjRaXh/3awHfmWRbzYxJoAdNJxe/3pvw= +github.com/pborman/uuid v1.2.1/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= +github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= +github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= +github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= +github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= +go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= +go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc h1:mCRnTeVUjcrhlRmO0VK8a6k6Rrf6TF9htwo2pJVSjIU= +golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= +golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= +golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8= +golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.23.0 h1:PbgcYx2W7i4LvjJWEbf0ngHV6qJYr86PkAV3bXdLEbs= +golang.org/x/oauth2 v0.23.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik= +golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y= +golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY= +golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4= +golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ= +golang.org/x/time v0.7.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200505023115-26f46d2f7ef8/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff/go.mod h1:YD9qOF0M9xpSpdWTBbzEl5e/RnCefISl8E5Noe10jFM= +golang.org/x/tools v0.1.9/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU= +golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ= +golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= +google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= +google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.35.1 h1:m3LfL6/Ca+fqnjnlqQXNpFPABW1UD7mjh8KO2mKFytA= +google.golang.org/protobuf v1.35.1/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4= +gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +k8s.io/api v0.23.3/go.mod h1:w258XdGyvCmnBj/vGzQMj6kzdufJZVUwEM1U2fRJwSQ= +k8s.io/api v0.32.1 h1:f562zw9cy+GvXzXf0CKlVQ7yHJVYzLfL6JAS4kOAaOc= +k8s.io/api v0.32.1/go.mod h1:/Yi/BqkuueW1BgpoePYBRdDYfjPF5sgTr5+YqDZra5k= +k8s.io/apiextensions-apiserver v0.31.0 h1:fZgCVhGwsclj3qCw1buVXCV6khjRzKC5eCFt24kyLSk= +k8s.io/apiextensions-apiserver v0.31.0/go.mod h1:b9aMDEYaEe5sdK+1T0KU78ApR/5ZVp4i56VacZYEHxk= +k8s.io/apimachinery v0.23.3/go.mod h1:BEuFMMBaIbcOqVIJqNZJXGFTP4W6AycEpb5+m/97hrM= +k8s.io/apimachinery v0.32.1 h1:683ENpaCBjma4CYqsmZyhEzrGz6cjn1MY/X2jB2hkZs= +k8s.io/apimachinery v0.32.1/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE= +k8s.io/client-go v0.32.1 h1:otM0AxdhdBIaQh7l1Q0jQpmo7WOFIk5FFa4bg6YMdUU= +k8s.io/client-go v0.32.1/go.mod h1:aTTKZY7MdxUaJ/KiUs8D+GssR9zJZi77ZqtzcGXIiDg= +k8s.io/code-generator v0.23.3/go.mod h1:S0Q1JVA+kSzTI1oUvbKAxZY/DYbA/ZUb4Uknog12ETk= +k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E= +k8s.io/gengo v0.0.0-20211129171323-c02415ce4185/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E= +k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= +k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= +k8s.io/klog/v2 v2.30.0/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= +k8s.io/klog/v2 v2.40.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= +k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= +k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= +k8s.io/kube-openapi v0.0.0-20211115234752-e816edb12b65/go.mod h1:sX9MT8g7NVZM5lVL/j8QyCCJe8YSMW30QvGZWaCIDIk= +k8s.io/kube-openapi v0.0.0-20220124234850-424119656bbf/go.mod h1:sX9MT8g7NVZM5lVL/j8QyCCJe8YSMW30QvGZWaCIDIk= +k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f h1:GA7//TjRY9yWGy1poLzYYJJ4JRdzg3+O6e8I+e+8T5Y= +k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f/go.mod h1:R/HEjbvWI0qdfb8viZUeVZm0X6IZnxAydC7YU42CMw4= +k8s.io/utils v0.0.0-20210802155522-efc7438f0176/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/utils v0.0.0-20211116205334-6203023598ed/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/utils v0.0.0-20241210054802-24370beab758 h1:sdbE21q2nlQtFh65saZY+rRM6x6aJJI8IUa1AmH/qa0= +k8s.io/utils v0.0.0-20241210054802-24370beab758/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +kubevirt.io/api v1.0.0 h1:RBdXP5CDhE0v5qL2OUQdrYyRrHe/F68Z91GWqBDF6nw= +kubevirt.io/api v1.0.0/go.mod h1:CJ4vZsaWhVN3jNbyc9y3lIZhw8nUHbWjap0xHABQiqc= +kubevirt.io/containerized-data-importer-api v1.57.0-alpha1 h1:IWo12+ei3jltSN5jQN1xjgakfvRSF3G3Rr4GXVOOy2I= +kubevirt.io/containerized-data-importer-api v1.57.0-alpha1/go.mod h1:Y/8ETgHS1GjO89bl682DPtQOYEU/1ctPFBz6Sjxm4DM= +kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90 h1:QMrd0nKP0BGbnxTqakhDZAUhGKxPiPiN5gSDqKUmGGc= +kubevirt.io/controller-lifecycle-operator-sdk/api v0.0.0-20220329064328-f3cc58c6ed90/go.mod h1:018lASpFYBsYN6XwmA2TIrPCx6e0gviTd/ZNtSitKgc= +sigs.k8s.io/controller-runtime v0.19.4 h1:SUmheabttt0nx8uJtoII4oIP27BVVvAKFvdvGFwV/Qo= +sigs.k8s.io/controller-runtime v0.19.4/go.mod h1:iRmWllt8IlaLjvTTDLhRBXIEtkCK6hwVBJJsYS9Ajf4= +sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6/go.mod h1:p4QtZmO4uMYipTQNzagwnNoseA6OxSUutVw05NhYDRs= +sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE= +sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= +sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= +sigs.k8s.io/structured-merge-diff/v4 v4.2.1/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4= +sigs.k8s.io/structured-merge-diff/v4 v4.5.0 h1:nbCitCK2hfnhyiKo6uf2HxUPTCodY6Qaf85SbDIaMBk= +sigs.k8s.io/structured-merge-diff/v4 v4.5.0/go.mod h1:N8f93tFZh9U6vpxwRArLiikrE5/2tiu1w1AGfACIGE4= +sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= +sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8= +sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= +sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY= diff --git a/e2e-tests/internal/cluster/builder.go b/e2e-tests/internal/cluster/builder.go new file mode 100644 index 0000000..ae8bc30 --- /dev/null +++ b/e2e-tests/internal/cluster/builder.go @@ -0,0 +1,329 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "context" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + + "github.com/deckhouse/sds-e2e-tests/internal/config" + "github.com/deckhouse/sds-e2e-tests/internal/infrastructure/ssh" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes/virtualization" + "github.com/deckhouse/sds-e2e-tests/internal/logger" + "github.com/deckhouse/sds-e2e-tests/pkg/cluster" +) + +// Builder builds and provisions new clusters +type Builder struct { + config *config.Config + manager *Manager + sshFactory ssh.Factory +} + +// NewBuilder creates a new cluster builder +func NewBuilder(cfg *config.Config, manager *Manager, sshFactory ssh.Factory) *Builder { + return &Builder{ + config: cfg, + manager: manager, + sshFactory: sshFactory, + } +} + +// BuildCluster implements the full cluster creation workflow +func (b *Builder) BuildCluster(ctx context.Context, baseCluster cluster.Cluster, dkpCfg *config.DKPClusterConfig) (cluster.Cluster, error) { + logger.Infof("Starting cluster build process") // TODO amarkov: This and in similar places: Consider using logr interface so ginkgo's logger backend (GinkgoWriter) can be used. It can be passed via arguments or via context. + + // Step 1: Ensure base cluster connection via SSH + baseVMClient, ok := baseCluster.Virtualization().(virtualization.Client) + if !ok { + return nil, fmt.Errorf("base cluster does not support virtualization operations") + } + + // Step 2: Enable Deckhouse Virtualization Platform module on base cluster + logger.Infof("Ensuring Virtualization Platform module is enabled") + // This would enable the module - simplified for now + + // Step 3: Create virtual machines + logger.Infof("Creating virtual machines") + vms, err := b.createVMs(ctx, baseVMClient, dkpCfg) + if err != nil { + return nil, fmt.Errorf("failed to create VMs: %w", err) + } + + // Step 4: Identify master and bootstrap nodes + masterVM, bootstrapVM, workerVMs, err := b.identifyVMRoles(vms, dkpCfg) + if err != nil { + return nil, fmt.Errorf("failed to identify VM roles: %w", err) + } + + // Store worker VMs for later use (adding nodes to cluster) + _ = len(workerVMs) // Will be used in future for adding static nodes + + // Step 5: Connect to master via SSH and deploy Deckhouse + logger.Infof("Deploying Deckhouse on master node") + err = b.deployDeckhouse(ctx, masterVM, bootstrapVM) + if err != nil { + return nil, fmt.Errorf("failed to deploy Deckhouse: %w", err) + } + + // Step 6: Get kubeconfig of nested cluster + logger.Infof("Retrieving nested cluster kubeconfig") + nestedKubeConfig, err := b.getNestedKubeconfig(ctx, masterVM) + if err != nil { + return nil, fmt.Errorf("failed to get nested kubeconfig: %w", err) + } + + // Step 7: Create nested cluster client + nestedCluster, err := b.manager.GetOrCreate(ctx, nestedKubeConfig, "") + if err != nil { + return nil, fmt.Errorf("failed to create nested cluster client: %w", err) + } + + // Step 8: Enable and configure modules + logger.Infof("Enabling modules in nested cluster") + err = b.enableModules(ctx, nestedCluster, dkpCfg) + if err != nil { + return nil, fmt.Errorf("failed to enable modules: %w", err) + } + + logger.Infof("Cluster build completed successfully") + return nestedCluster, nil +} + +func (b *Builder) createVMs(ctx context.Context, vmClient virtualization.Client, dkpCfg *config.DKPClusterConfig) ([]*vmInfo, error) { + var vms []*vmInfo + + // Create master VMs + for _, master := range dkpCfg.ClusterDefinition.Masters { + req := virtualization.VMRequest{ + Name: master.Hostname, + Namespace: b.config.TestNS, + Image: master.OSType.Name, + ImageURL: master.OSType.ImageURL, + CPU: master.CPU, + RAM: master.RAM, + DiskSize: master.DiskSize, + StorageClass: b.config.BaseCluster.StorageClass, + SSHPubKey: "", // Would load from key file + IPAddress: master.IPAddress, + } + + vm, err := vmClient.CreateVM(ctx, req) + if err != nil { + return nil, fmt.Errorf("failed to create master VM %s: %w", master.Hostname, err) + } + + vms = append(vms, &vmInfo{ + name: master.Hostname, + role: "master", + vm: vm, + }) + } + + // Create worker VMs + for _, worker := range dkpCfg.ClusterDefinition.Workers { + req := virtualization.VMRequest{ + Name: worker.Hostname, + Namespace: b.config.TestNS, + Image: worker.OSType.Name, + ImageURL: worker.OSType.ImageURL, + CPU: worker.CPU, + RAM: worker.RAM, + DiskSize: worker.DiskSize, + StorageClass: b.config.BaseCluster.StorageClass, + SSHPubKey: "", + IPAddress: worker.IPAddress, + } + + vm, err := vmClient.CreateVM(ctx, req) + if err != nil { + return nil, fmt.Errorf("failed to create worker VM %s: %w", worker.Hostname, err) + } + + vms = append(vms, &vmInfo{ + name: worker.Hostname, + role: "worker", + vm: vm, + }) + } + + // Wait for all VMs to be running + // TODO amarkov: Consider extracting this to the separate method to separate create/wait operations. + for _, vm := range vms { + err := vmClient.WaitForVMRunning(ctx, b.config.TestNS, vm.name, 600) + if err != nil { + return nil, fmt.Errorf("VM %s failed to become ready: %w", vm.name, err) + } + + ip, err := vmClient.GetVMIP(ctx, b.config.TestNS, vm.name) + if err != nil { + return nil, fmt.Errorf("failed to get IP for VM %s: %w", vm.name, err) + } + vm.ip = ip + } + + return vms, nil +} + +func (b *Builder) identifyVMRoles(vms []*vmInfo, dkpCfg *config.DKPClusterConfig) (*vmInfo, *vmInfo, []*vmInfo, error) { + var masterVM, bootstrapVM *vmInfo + var workerVMs []*vmInfo + + for _, vm := range vms { + if vm.role == "master" { + masterVM = vm + } else if vm.role == "worker" { + workerVMs = append(workerVMs, vm) + } + } + + if masterVM == nil { + return nil, nil, nil, fmt.Errorf("no master VM found") + } + + // Bootstrap is first worker or setup node + if dkpCfg.ClusterDefinition.Setup != nil { + for _, vm := range vms { + if vm.name == dkpCfg.ClusterDefinition.Setup.Hostname { + bootstrapVM = vm + break + } + } + } + if bootstrapVM == nil && len(workerVMs) > 0 { + bootstrapVM = workerVMs[0] + } + + // workerVMs will be used later for adding nodes to the cluster + return masterVM, bootstrapVM, workerVMs, nil +} + +func (b *Builder) deployDeckhouse(ctx context.Context, masterVM, bootstrapVM *vmInfo) error { + // This is a simplified implementation - in production this would: + // 1. Connect to bootstrap VM via SSH + // 2. Upload config.yml and resources.yml + // 3. Install Docker if needed + // 4. Run dhctl bootstrap commands + // 5. Wait for Deckhouse to be ready + + logger.Infof("Deploying Deckhouse - connecting to bootstrap VM %s", bootstrapVM.ip) + // Simplified for demonstration + return nil +} + +func (b *Builder) getNestedKubeconfig(ctx context.Context, masterVM *vmInfo) (string, error) { + // Connect to master VM and get kubeconfig + sshClient, err := b.sshFactory.CreateClient("user", masterVM.ip+":22", b.config.NestedCluster.SSHKey) + if err != nil { + return "", fmt.Errorf("failed to create SSH client: %w", err) + } + defer sshClient.Close() + + kubeconfigContent := sshClient.ExecFatal(ctx, "sudo cat /root/.kube/config") + + // Replace port if needed + kubeconfigContent = strings.ReplaceAll(kubeconfigContent, "127.0.0.1:6445", "127.0.0.1:"+b.config.NestedCluster.K8sPort) + + // Write to file + kubeconfigPath := b.config.NestedCluster.KubeConfig + err = os.MkdirAll(filepath.Dir(kubeconfigPath), 0755) + if err != nil { + return "", err + } + + err = os.WriteFile(kubeconfigPath, []byte(kubeconfigContent), 0600) + if err != nil { + return "", fmt.Errorf("failed to write kubeconfig: %w", err) + } + + return kubeconfigPath, nil +} + +// TODO amarkov: +// I am confused about difference between these modules and modules listed in cluster_creation_test: https://github.com/deckhouse/sds-e2e/pull/20/files#diff-70680e420a441925212ade494a89118228ae9e7058f988299d9c4947238da4e0R122, there are same. Why do we have same modules listed twice? The second modules listed in test is the just example how to enable modules? So we have required modules listed here and ability to install extra modules? +// I think, it's important to allow user to configure ModulePullOverride to install module from MR/PR for development. I see two options: +// Have a single entry point BuildCluster which performs full setup: VMs, deckhouse, modules. Modules are provided via config. Default list of modules is appended to the customized one. Module settings (perhaps, only version) from config override default ones. +// Pros: +// Simpler API +// Easy to have default config and overrides +// Cons: +// Complex config +// Less modular API +// Don't configure modules at all in BuildCluster and configure all modules manually with EnsureModuleEnabled. But it's not clear for me now how to pass default modules. +// Pros: +// More modular API +// Cons: +// More complex API +// Not clear how to pass default configuration +// Pick Option 2 and build simple API on top + +// Default modules must be enabled with EnsureModuleEnabled but not with BuildCluster. We need ability to configure default modules as well - module pull override, settings etc. +func (b *Builder) enableModules(ctx context.Context, nestedCluster cluster.Cluster, dkpCfg *config.DKPClusterConfig) error { + // Enable modules in dependency order + modules := []*config.ModuleConfig{ + {Name: "snapshot-controller", Version: 1, Enabled: true, Dependencies: []string{}}, + {Name: "sds-local-volume", Version: 1, Enabled: true, Dependencies: []string{"snapshot-controller"}}, + {Name: "sds-node-configurator", Version: 1, Enabled: true, Settings: map[string]any{"enableThinProvisioning": true}, Dependencies: []string{"sds-local-volume"}}, + {Name: "sds-replicated-volume", Version: 1, Enabled: true, Dependencies: []string{"sds-node-configurator"}}, + } + + enabled := make(map[string]bool) + + for len(modules) > 0 { + progress := false + for i, module := range modules { + // Check dependencies + allDepsEnabled := true + for _, dep := range module.Dependencies { + if !enabled[dep] { + allDepsEnabled = false + break + } + } + + if allDepsEnabled { + err := nestedCluster.Deckhouse().EnsureModuleEnabled(ctx, module) + if err != nil { + return fmt.Errorf("failed to enable module %s: %w", module.Name, err) + } + + logger.Infof("Enabled module: %s", module.Name) + enabled[module.Name] = true + modules = slices.Delete(modules, i, i+1) + progress = true + break + } + } + + if !progress { + return fmt.Errorf("circular or missing dependencies in modules") + } + } + + return nil +} + +type vmInfo struct { + name string + role string + ip string + vm interface{} // Would be *virt.VirtualMachine +} diff --git a/e2e-tests/internal/cluster/cluster.go b/e2e-tests/internal/cluster/cluster.go new file mode 100644 index 0000000..2ea3116 --- /dev/null +++ b/e2e-tests/internal/cluster/cluster.go @@ -0,0 +1,84 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "context" + + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes/deckhouse" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes/virtualization" + "github.com/deckhouse/sds-e2e-tests/pkg/cluster" +) + +// Cluster implements the cluster.Cluster interface +// TODO amarkov: I am confused with Cluster interface, this implementation, Deckhouse, Virtualization interfaces and implementations. Why do we need them? What value they bring? Just helpers for some operations? Some method performs complex actions while some other methods are just wrappers. +// Is it expected that similar interfaces should be created for any other resource? E.g. I want to work with PVC, DataImport and Job in my test. Should I create interface, implementation and getter in Cluster? Personally, I don't like so much boilerplate to perform simple actions like Get or Create which is actually just wrapper for client. +// If it's not expected to create helper for each resource, it will be not consistent to have manipulate some resource via wrapper and some directly via client. +type Cluster struct { + name string + ctx context.Context + kubeClient *kubernetes.Client + vmClient virtualization.Client + dhClient deckhouse.Client +} + +// NewCluster creates a new cluster instance +func NewCluster(ctx context.Context, name string, kubeClient *kubernetes.Client) *Cluster { + return &Cluster{ + name: name, + ctx: ctx, + kubeClient: kubeClient, + vmClient: virtualization.NewClient(kubeClient), + dhClient: deckhouse.NewClient(kubeClient), + } +} + +// Name returns the cluster name +func (c *Cluster) Name() string { + return c.name +} + +// Context returns the context +func (c *Cluster) Context() context.Context { + return c.ctx +} + +// Virtualization returns the virtualization client +func (c *Cluster) Virtualization() virtualization.Client { + return c.vmClient +} + +// Deckhouse returns the deckhouse client +func (c *Cluster) Deckhouse() deckhouse.Client { + return c.dhClient +} + +// EnsureReady ensures the cluster is ready +func (c *Cluster) EnsureReady(ctx context.Context) error { + // Basic readiness check - can be expanded + return nil +} + +// Close closes the cluster connection +func (c *Cluster) Close() error { + // Cleanup if needed + return nil +} + +// Verify that Cluster implements the interface +var _ cluster.Cluster = (*Cluster)(nil) diff --git a/e2e-tests/internal/cluster/manager.go b/e2e-tests/internal/cluster/manager.go new file mode 100644 index 0000000..3062c55 --- /dev/null +++ b/e2e-tests/internal/cluster/manager.go @@ -0,0 +1,89 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "context" + "fmt" + "sync" + + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes" + "github.com/deckhouse/sds-e2e-tests/pkg/cluster" +) + +// Manager manages cluster lifecycle +type Manager struct { + clusters map[string]*Cluster // TODO amarkov: Why do we have state here? It is never used. Manager is created only to perform single call. - В коде так: мы создаем новый экземпляр менеджера, выполняем какое-то действие и больше этот менеджер не используется. Зачем тут кеш коннекшнов (clusters map[]*Cluster)? Пример использования https://github.com/deckhouse/sds-e2e/pull/20/files#diff-d091ceebec46a0092794fcbb5caa6afc22969e97953f3e5dd11228a27959d303R36 + mu sync.RWMutex +} + +// NewManager creates a new cluster manager +func NewManager() *Manager { + return &Manager{ + clusters: make(map[string]*Cluster), + } +} + +// GetOrCreate retrieves an existing cluster or creates a new one +func (m *Manager) GetOrCreate(ctx context.Context, configPath, name string) (cluster.Cluster, error) { // TODO amarkov: The name is confusing for me. It seems that it creates k8s cluster somehow, but It creates a connection (client) and wraps into the helper structure Cluster. - Метод ничего не создает. Он получает коннекшн, оборачивает его в cluster.Cluster. Это работа с коннекшнами, а не работа с кластером. ПЕРЕДЕЛАТЬ И ПЕРЕИМЕНОВАТЬ! + m.mu.Lock() + defer m.mu.Unlock() + + key := fmt.Sprintf("%s:%s", configPath, name) + if c, ok := m.clusters[key]; ok { + return c, nil + } + + // Create new cluster + kubeClient, err := kubernetes.NewClient(ctx, configPath, name) + if err != nil { + return nil, fmt.Errorf("failed to create kubernetes client: %w", err) + } + + c := NewCluster(ctx, name, kubeClient) + m.clusters[key] = c + return c, nil +} + +// Get retrieves an existing cluster by key +func (m *Manager) Get(configPath, name string) (cluster.Cluster, bool) { + m.mu.RLock() + defer m.mu.RUnlock() + + key := fmt.Sprintf("%s:%s", configPath, name) + c, ok := m.clusters[key] + return c, ok +} + +// CloseAll closes all managed clusters +func (m *Manager) CloseAll() error { + m.mu.Lock() + defer m.mu.Unlock() + + var errs []error + for _, c := range m.clusters { + if err := c.Close(); err != nil { + errs = append(errs, err) + } + } + m.clusters = make(map[string]*Cluster) + + if len(errs) > 0 { + return fmt.Errorf("errors closing clusters: %v", errs) + } + return nil +} diff --git a/e2e-tests/internal/config/config.go b/e2e-tests/internal/config/config.go new file mode 100644 index 0000000..4b3e24f --- /dev/null +++ b/e2e-tests/internal/config/config.go @@ -0,0 +1,198 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package config + +import ( + "fmt" + "os" + "path/filepath" + "runtime" + "strings" + "time" +) + +// Config holds all configuration for the e2e tests +type Config struct { // TODO amarkov: Add description of fields and update comment for the struct + // Test namespace configuration + TestNS string + TestNSCleanUp string + KeepState bool + + // Base cluster (hypervisor) configuration + BaseCluster BaseClusterConfig + + // Nested cluster configuration + NestedCluster NestedClusterConfig + + // Feature flags + SkipOptional bool + Parallel bool + TreeMode bool + + // Logging + Verbose bool + Debug bool + LogFile string + + // Paths + DataPath string + KubePath string + RemoteAppPath string + + // File names + PrivKeyName string + PubKeyName string + ConfigName string + ResourcesName string + ConfigTplName string + ResourcesTplName string +} + +// BaseClusterConfig contains configuration for the base Deckhouse cluster (hypervisor) +type BaseClusterConfig struct { + KubeConfig string + SSHHost string + SSHUser string + SSHKey string + K8sPort string + StorageClass string +} + +// NestedClusterConfig contains configuration for the nested cluster being created +type NestedClusterConfig struct { + SSHHost string + SSHUser string + SSHKey string + K8sPort string + KubeConfig string + StorageClass string +} + +// findTestFile traces back through the call stack to find the test file +// that initiated the call chain. It looks for files ending with "_test.go" +func findTestFile() (string, error) { + // Start from the caller of Load() and go up the stack + for i := 1; i < 10; i++ { + _, file, _, ok := runtime.Caller(i) + if !ok { + break + } + // Check if this is a test file + if strings.HasSuffix(file, "_test.go") { + return file, nil + } + } + // If no test file found, return the immediate caller (Load's caller) + _, file, _, ok := runtime.Caller(1) + if !ok { + return "", fmt.Errorf("unable to determine caller file") + } + return file, nil +} + +// Load creates a new Config from environment variables and defaults +func Load() *Config { + // Find the test file path to resolve relative paths + testFile, err := findTestFile() + var baseDir string + if err == nil && testFile != "" { + baseDir = filepath.Dir(testFile) + } else { + // Fallback: use current working directory if we can't find test file + if wd, err := os.Getwd(); err == nil { + baseDir = wd + } else { + baseDir = "." + } + } + + // Resolve absolute paths relative to the test file location + dataPath := filepath.Join(baseDir, "..", "data") + kubePath := filepath.Join(baseDir, "..", "..", "..", "sds-e2e-cfg") + + // Convert to absolute paths + if absDataPath, err := filepath.Abs(dataPath); err == nil { + dataPath = absDataPath + } + if absKubePath, err := filepath.Abs(kubePath); err == nil { + kubePath = absKubePath + } + + cfg := &Config{ + TestNS: generateTestNS(), // TODO amarkov: I'm suggesting to allow specify namespace via environment. It's not convenient to have different namespace each time if you are run same test suite again and again during development. + TestNSCleanUp: "", + KeepState: false, + BaseCluster: BaseClusterConfig{ + KubeConfig: os.Getenv("BASE_KUBECONFIG"), + SSHHost: os.Getenv("BASE_SSH_HOST"), + SSHUser: os.Getenv("BASE_SSH_USER"), + SSHKey: os.Getenv("BASE_SSH_KEY"), + K8sPort: getEnvOrDefault("BASE_K8S_PORT", "6445"), + StorageClass: getEnvOrDefault("BASE_STORAGE_CLASS", "linstor-r1"), + }, + NestedCluster: NestedClusterConfig{ + SSHHost: getEnvOrDefault("NESTED_SSH_HOST", "127.0.0.1"), + SSHUser: getEnvOrDefault("NESTED_SSH_USER", "user"), + SSHKey: os.Getenv("NESTED_SSH_KEY"), + K8sPort: getEnvOrDefault("NESTED_K8S_PORT", "6445"), + KubeConfig: getEnvOrDefault("NESTED_KUBECONFIG", "kube-nested.config"), + StorageClass: getEnvOrDefault("NESTED_STORAGE_CLASS", "linstor-r1"), + }, + SkipOptional: false, + Parallel: false, + TreeMode: false, + Verbose: false, + Debug: false, + LogFile: os.Getenv("LOG_FILE"), + DataPath: dataPath, + KubePath: kubePath, + RemoteAppPath: "/home/user", + PrivKeyName: "id_rsa_test", + PubKeyName: "id_rsa_test.pub", + ConfigName: "config.yml", + ResourcesName: "resources.yml", + ConfigTplName: "config.yml.tpl", + ResourcesTplName: "resources.yml.tpl", + } + + // Ensure paths are absolute + if !filepath.IsAbs(cfg.KubePath) { + if absPath, err := filepath.Abs(cfg.KubePath); err == nil { + cfg.KubePath = absPath + } + } + if !filepath.IsAbs(cfg.NestedCluster.KubeConfig) { + cfg.NestedCluster.KubeConfig = filepath.Join(cfg.KubePath, cfg.NestedCluster.KubeConfig) + } + if cfg.BaseCluster.KubeConfig != "" && !filepath.IsAbs(cfg.BaseCluster.KubeConfig) { + cfg.BaseCluster.KubeConfig = filepath.Join(cfg.KubePath, cfg.BaseCluster.KubeConfig) + } + + return cfg +} + +func generateTestNS() string { + now := time.Now() + return fmt.Sprintf("e2e-tmp-%d%d", now.Minute(), now.Second()) +} + +func getEnvOrDefault(key, defaultValue string) string { + if value := os.Getenv(key); value != "" { + return value + } + return defaultValue +} diff --git a/e2e-tests/internal/config/types.go b/e2e-tests/internal/config/types.go new file mode 100644 index 0000000..42849b6 --- /dev/null +++ b/e2e-tests/internal/config/types.go @@ -0,0 +1,121 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package config + +import "time" + +// HostType represents the type of host (VM or bare-metal) +type HostType string + +const ( + HostTypeVM HostType = "vm" + HostTypeBareMetal HostType = "bare-metal" +) + +// ClusterRole represents the role of a node in the cluster +type ClusterRole string + +const ( + ClusterRoleMaster ClusterRole = "master" + ClusterRoleWorker ClusterRole = "worker" + ClusterRoleSetup ClusterRole = "setup" // Bootstrap node for DKP installation +) + +// OSType represents the operating system type +type OSType struct { + Name string + ImageURL string + KernelVersion string +} + +var ( + OSTypeMap = map[string]OSType{ + "Ubuntu 22.04 6.2.0-39-generic": { + ImageURL: "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img", + KernelVersion: "6.2.0-39-generic", + }, + "Ubuntu 24.04 6.8.0-53-generic": { + ImageURL: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img", + KernelVersion: "6.8.0-53-generic", + }, + } +) + +// AuthMethod represents the authentication method +type AuthMethod string + +const ( + AuthMethodSSHKey AuthMethod = "ssh-key" + AuthMethodSSHPass AuthMethod = "ssh-password" +) + +// NodeAuth contains authentication information for a node +type NodeAuth struct { + Method AuthMethod + User string + SSHKey string // Public key (value like "ssh-rsa ...", path to .pub file, or empty for default) + Password string // Password (if using password auth) +} + +// ClusterNode defines a single node in the cluster +type ClusterNode struct { + Hostname string + IPAddress string // Required for bare-metal, optional for VM + OSType OSType // Required for VM, optional for bare-metal + HostType HostType + Role ClusterRole + Auth NodeAuth + // VM-specific fields (only used when HostType == HostTypeVM) + CPU int // Required for VM + RAM int // Required for VM, in GB + DiskSize int // Required for VM, in GB + // Bare-metal specific fields + Prepared bool // Whether the node is already prepared for DKP installation +} + +// ClusterDefinition defines the complete cluster configuration +type ClusterDefinition struct { + Masters []ClusterNode + Workers []ClusterNode + Setup *ClusterNode // Bootstrap node (can be nil) +} + +// ModuleConfig defines a Deckhouse module configuration +type ModuleConfig struct { // TODO amarkov: I suggest allow user to specify ModulePullOverride version, to run tests on MR/PR during development process. + Name string + Version int + Enabled bool + Settings map[string]any + Dependencies []string // Names of modules that must be enabled before this one +} + +// DKPClusterConfig defines the Deckhouse Kubernetes Platform cluster configuration +type DKPClusterConfig struct { + ClusterDefinition ClusterDefinition + KubernetesVersion string + PodSubnetCIDR string + ServiceSubnetCIDR string + ClusterDomain string + LicenseKey string + RegistryRepo string +} + +const ( + HostReadyTimeout = 10 * time.Minute // Timeout for hosts to be ready + DKPDeployTimeout = 30 * time.Minute // Timeout for DKP deployment + ModuleDeployTimeout = 10 * time.Minute // Timeout for module deployment +) diff --git a/e2e-tests/internal/infrastructure/ssh/interface.go b/e2e-tests/internal/infrastructure/ssh/interface.go new file mode 100644 index 0000000..237072b --- /dev/null +++ b/e2e-tests/internal/infrastructure/ssh/interface.go @@ -0,0 +1,44 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package ssh + +import "context" + +// Client provides SSH operations +type Client interface { + // Exec executes a command on the remote host + Exec(ctx context.Context, cmd string) (string, error) + + // ExecFatal executes a command and returns error if it fails + ExecFatal(ctx context.Context, cmd string) string + + // Upload uploads a file to the remote host + Upload(ctx context.Context, localPath, remotePath string) error + + // Close closes the SSH connection + Close() error +} + +// Factory creates SSH clients +type Factory interface { + // CreateClient creates a new SSH client + CreateClient(user, host, keyPath string) (Client, error) + + // CreateForwardClient creates an SSH client with port forwarding + CreateForwardClient(user, host, keyPath string, localPort, remotePort string) (Client, error) +} + diff --git a/e2e-tests/internal/kubernetes/client.go b/e2e-tests/internal/kubernetes/client.go new file mode 100644 index 0000000..d75ea5d --- /dev/null +++ b/e2e-tests/internal/kubernetes/client.go @@ -0,0 +1,160 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package kubernetes + +import ( + "context" + "fmt" + + logr "github.com/go-logr/logr" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" + kubescheme "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" + ctrlrtclient "sigs.k8s.io/controller-runtime/pkg/client" + ctrlrtlog "sigs.k8s.io/controller-runtime/pkg/log" + + coreapi "k8s.io/api/core/v1" + storapi "k8s.io/api/storage/v1" + extapi "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + apiruntime "k8s.io/apimachinery/pkg/runtime" + + snc "github.com/deckhouse/sds-node-configurator/api/v1alpha1" + srv "github.com/deckhouse/sds-replicated-volume/api/v1alpha1" + virt "github.com/deckhouse/virtualization/api/core/v1alpha2" +) + +// Client wraps Kubernetes API clients +type Client struct { + ctx context.Context // TODO amarkov: I'm not sure how good it is in Go to store context. Should we rather pass it in each call as usual? - Context should better be passed as an argument, do not keep in in struct. + restCfg *rest.Config + controllerRuntimeClient ctrlrtclient.Client + goClient *kubernetes.Clientset + dyClient *dynamic.DynamicClient +} + +// NewClient creates a new Kubernetes client +func NewClient(ctx context.Context, configPath, clusterName string) (*Client, error) { + restCfg, err := newRestConfig(configPath, clusterName) + if err != nil { + return nil, fmt.Errorf("failed to create rest config: %w", err) + } + + rcl, err := newKubeRTClient(restCfg) + if err != nil { + return nil, fmt.Errorf("failed to create controller runtime client: %w", err) + } + + gcl, err := kubernetes.NewForConfig(restCfg) + if err != nil { + return nil, fmt.Errorf("failed to create kubernetes clientset: %w", err) + } + + dcl, err := dynamic.NewForConfig(restCfg) + if err != nil { + return nil, fmt.Errorf("failed to create dynamic client: %w", err) + } + + return &Client{ + ctx: ctx, + restCfg: restCfg, + controllerRuntimeClient: rcl, + goClient: gcl, + dyClient: dcl, + }, nil +} + +func newRestConfig(configPath, clusterName string) (*rest.Config, error) { + var loader clientcmd.ClientConfigLoader + if configPath != "" { + loader = &clientcmd.ClientConfigLoadingRules{ExplicitPath: configPath} + } else { + loader = clientcmd.NewDefaultClientConfigLoadingRules() + } + + overrides := clientcmd.ConfigOverrides{} + if clusterName != "" { + overrides.Context.Cluster = clusterName + overrides.CurrentContext = clusterName + } + + cfg, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + loader, &overrides).ClientConfig() + if err != nil { + return nil, fmt.Errorf("failed to create rest config: %w", err) + } + + return cfg, nil +} + +func newKubeRTClient(cfg *rest.Config) (ctrlrtclient.Client, error) { + scheme := apiruntime.NewScheme() + + // Add schemes + schemeFuncs := []func(*apiruntime.Scheme) error{ + virt.AddToScheme, + srv.AddToScheme, + snc.AddToScheme, + kubescheme.AddToScheme, + extapi.AddToScheme, + coreapi.AddToScheme, + storapi.AddToScheme, + // Add Deckhouse scheme - we'll need to define this separately + } + + for _, f := range schemeFuncs { + if err := f(scheme); err != nil { + return nil, fmt.Errorf("failed to add scheme: %w", err) + } + } + + // TODO amarkov: Ginkgo provides own logger with logr interface: https://onsi.github.io/ginkgo/#logging-output + // Instead of creating logr from new context (context.Background) I recommend propagate logger from the topmost call, so user can pass GingkoWriter as logging backend. It can be propagated either as arguments or as context. + // mrwesadex-flant + // У Гинкго проблема с логгированием - оно проглатыывает все логи до ошибки. Если нет ошибки, нет и логов. У нее есть свой логгер - GinkgoWriter - лучше использовать его. Есть реализация интерфейса logr - можно посмотреть в сторону нее. + + ctrlrtlog.SetLogger(logr.FromContextOrDiscard(context.Background())) + cl, err := ctrlrtclient.New(cfg, ctrlrtclient.Options{ + Scheme: scheme, + }) + if err != nil { + return nil, err + } + + return cl, nil +} + +// ControllerRuntimeClient returns the controller-runtime client +func (c *Client) ControllerRuntimeClient() ctrlrtclient.Client { + return c.controllerRuntimeClient +} + +// GoClient returns the kubernetes clientset +func (c *Client) GoClient() *kubernetes.Clientset { + return c.goClient +} + +// DynamicClient returns the dynamic client +func (c *Client) DynamicClient() *dynamic.DynamicClient { + return c.dyClient +} + +// Context returns the context +func (c *Client) Context() context.Context { + return c.ctx +} diff --git a/e2e-tests/internal/kubernetes/deckhouse/client.go b/e2e-tests/internal/kubernetes/deckhouse/client.go new file mode 100644 index 0000000..24c1c58 --- /dev/null +++ b/e2e-tests/internal/kubernetes/deckhouse/client.go @@ -0,0 +1,104 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package deckhouse + +import ( + "context" + "fmt" + + v1alpha1nfs "github.com/deckhouse/csi-nfs/api/v1alpha1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + ctrlrtclient "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/deckhouse/sds-e2e-tests/internal/config" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes" + "github.com/deckhouse/sds-e2e-tests/internal/utils" +) + +// Client provides operations on Deckhouse modules +type Client interface { + // EnsureModuleEnabled ensures a module is enabled with given configuration + EnsureModuleEnabled(ctx context.Context, moduleCfg *config.ModuleConfig) error + + // WaitForModuleReady waits for a module to be ready + WaitForModuleReady(ctx context.Context, moduleName string, timeoutSeconds int) error +} + +type client struct { + kubeClient *kubernetes.Client +} + +// NewClient creates a new deckhouse client +func NewClient(kubeClient *kubernetes.Client) Client { + return &client{kubeClient: kubeClient} +} + +func (c *client) EnsureModuleEnabled(ctx context.Context, moduleCfg *config.ModuleConfig) error { + moduleConfig := &v1alpha1nfs.ModuleConfig{ + ObjectMeta: metav1.ObjectMeta{ + Name: moduleCfg.Name, + }, + Spec: v1alpha1nfs.ModuleConfigSpec{ + Version: moduleCfg.Version, + Enabled: ptr.To(moduleCfg.Enabled), + Settings: moduleCfg.Settings, + }, + } + + err := c.kubeClient.ControllerRuntimeClient().Create(ctx, moduleConfig) + if err != nil { + if !apierrors.IsAlreadyExists(err) { + return fmt.Errorf("failed to create module config: %w", err) + } + + // Update existing + existing := &v1alpha1nfs.ModuleConfig{} + err = c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{Name: moduleCfg.Name}, existing) + if err != nil { + return fmt.Errorf("failed to get existing module config: %w", err) + } + + moduleConfig.ResourceVersion = existing.ResourceVersion + err = c.kubeClient.ControllerRuntimeClient().Update(ctx, moduleConfig) + if err != nil { + return fmt.Errorf("failed to update module config: %w", err) + } + } + + return nil +} + +// TODO amarkov: Need to rework WaitForXXXReady to check (IsXXXReady) and Wait wrapper. +func (c *client) WaitForModuleReady(ctx context.Context, moduleName string, timeoutSeconds int) error { + // This is a simplified implementation - in production, you'd check for actual deployments/daemonsets + return utils.RetrySec(timeoutSeconds, func() error { + // For now, just check that module config exists and is enabled + moduleConfig := &v1alpha1nfs.ModuleConfig{} + err := c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{Name: moduleName}, moduleConfig) + if err != nil { + return fmt.Errorf("module %s not found: %w", moduleName, err) + } + + if moduleConfig.Spec.Enabled == nil || !*moduleConfig.Spec.Enabled { + return fmt.Errorf("module %s is not enabled", moduleName) + } + + return nil + }) +} diff --git a/e2e-tests/internal/kubernetes/virtualization/client.go b/e2e-tests/internal/kubernetes/virtualization/client.go new file mode 100644 index 0000000..5cadfed --- /dev/null +++ b/e2e-tests/internal/kubernetes/virtualization/client.go @@ -0,0 +1,323 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package virtualization + +import ( + "context" + "fmt" + + virt "github.com/deckhouse/virtualization/api/core/v1alpha2" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + ctrlrtclient "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes" + "github.com/deckhouse/sds-e2e-tests/internal/utils" +) + +// Client provides operations on virtualization resources (VMs, disks, etc.) +type Client interface { + // CreateVM creates a virtual machine + CreateVM(ctx context.Context, req VMRequest) (*virt.VirtualMachine, error) + + // GetVM retrieves a virtual machine by name + GetVM(ctx context.Context, namespace, name string) (*virt.VirtualMachine, error) + + // ListVMs lists virtual machines with optional filters + ListVMs(ctx context.Context, namespace string) ([]virt.VirtualMachine, error) + + // WaitForVMRunning waits for a VM to be in Running phase + WaitForVMRunning(ctx context.Context, namespace, name string, timeoutSeconds int) error + + // GetVMIP gets the IP address of a running VM + GetVMIP(ctx context.Context, namespace, name string) (string, error) +} + +// VMRequest contains parameters for creating a VM +type VMRequest struct { + Name string + Namespace string + Image string // Image name or URL + ImageURL string // Full image URL (derived from Image if Image is a key in OSTypeMap) + CPU int + RAM int // In GB + DiskSize int // In GB + StorageClass string + SSHPubKey string + IPAddress string // Optional static IP +} + +type client struct { + kubeClient *kubernetes.Client +} + +// NewClient creates a new virtualization client +func NewClient(kubeClient *kubernetes.Client) Client { + return &client{kubeClient: kubeClient} +} + +func (c *client) CreateVM(ctx context.Context, req VMRequest) (*virt.VirtualMachine, error) { + // Get or create cluster virtual image + cvmi, err := c.getOrCreateClusterVirtualImage(ctx, req.Image, req.ImageURL) + if err != nil { + return nil, fmt.Errorf("failed to get/create cluster virtual image: %w", err) + } + + // Create IP claim if IP is provided + ipClaimName := "" + if req.IPAddress != "" { + ipClaim, err := c.createOrGetIPClaim(ctx, req.Namespace, req.Name, req.IPAddress) + if err != nil { + return nil, fmt.Errorf("failed to create/get IP claim: %w", err) + } + ipClaimName = ipClaim.Name + } + + // Create system disk + systemDisk, err := c.createOrGetSystemDisk(ctx, req.Namespace, req.Name, req.StorageClass, req.DiskSize, cvmi) + if err != nil { + return nil, fmt.Errorf("failed to create/get system disk: %w", err) + } + + // Create VM + memory, err := resource.ParseQuantity(fmt.Sprintf("%dGi", req.RAM)) + if err != nil { + return nil, fmt.Errorf("failed to parse memory: %w", err) + } + + vm := &virt.VirtualMachine{ + ObjectMeta: metav1.ObjectMeta{ + Name: req.Name, + Namespace: req.Namespace, + Labels: map[string]string{"vm": "linux", "service": "v1"}, + }, + Spec: virt.VirtualMachineSpec{ + VirtualMachineClassName: "generic", + EnableParavirtualization: true, + RunPolicy: virt.RunPolicy("AlwaysOn"), + OsType: virt.OsType("Generic"), + Bootloader: virt.BootloaderType("BIOS"), + VirtualMachineIPAddress: ipClaimName, + CPU: virt.CPUSpec{Cores: req.CPU, CoreFraction: "100%"}, + Memory: virt.MemorySpec{Size: memory}, + BlockDeviceRefs: []virt.BlockDeviceSpecRef{ + { + Kind: virt.DiskDevice, + Name: systemDisk.Name, + }, + }, + Provisioning: &virt.Provisioning{ + Type: virt.ProvisioningType("UserData"), + UserData: fmt.Sprintf(`#cloud-config +package_update: true +packages: +- qemu-guest-agent +runcmd: +- [ hostnamectl, set-hostname, %s ] +- [ systemctl, daemon-reload ] +- [ systemctl, enable, --now, qemu-guest-agent.service ] +user: user +password: user +ssh_pwauth: True +chpasswd: { expire: False } +sudo: ALL=(ALL) NOPASSWD:ALL +chpasswd: { expire: False } +ssh_authorized_keys: + - %s +`, req.Name, req.SSHPubKey), + }, + }, + } + + err = c.kubeClient.ControllerRuntimeClient().Create(ctx, vm) + if err != nil && !apierrors.IsAlreadyExists(err) { + return nil, fmt.Errorf("failed to create VM: %w", err) + } + + return vm, nil +} + +func (c *client) GetVM(ctx context.Context, namespace, name string) (*virt.VirtualMachine, error) { + vm := &virt.VirtualMachine{} + err := c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{ + Namespace: namespace, + Name: name, + }, vm) + if err != nil { + return nil, err + } + return vm, nil +} + +func (c *client) ListVMs(ctx context.Context, namespace string) ([]virt.VirtualMachine, error) { + vmList := &virt.VirtualMachineList{} + opts := []ctrlrtclient.ListOption{} + if namespace != "" { + opts = append(opts, ctrlrtclient.InNamespace(namespace)) + } + + err := c.kubeClient.ControllerRuntimeClient().List(ctx, vmList, opts...) + if err != nil { + return nil, err + } + return vmList.Items, nil +} + +func (c *client) WaitForVMRunning(ctx context.Context, namespace, name string, timeoutSeconds int) error { + return utils.RetrySec(timeoutSeconds, func() error { + vm, err := c.GetVM(ctx, namespace, name) + if err != nil { + return err + } + if vm.Status.Phase != virt.MachineRunning { + return fmt.Errorf("VM %s/%s is not running, phase: %s", namespace, name, vm.Status.Phase) + } + return nil + }) +} + +func (c *client) GetVMIP(ctx context.Context, namespace, name string) (string, error) { + vm, err := c.GetVM(ctx, namespace, name) + if err != nil { + return "", err + } + if vm.Status.IPAddress == "" { + return "", fmt.Errorf("VM %s/%s does not have an IP address yet", namespace, name) + } + return vm.Status.IPAddress, nil +} + +// Helper methods + +func (c *client) getOrCreateClusterVirtualImage(ctx context.Context, imageName, imageURL string) (*virt.ClusterVirtualImage, error) { + // Derive name from image + cvmiName := utils.HashMd5(imageURL)[:8] + + // Try to get existing + cvmi := &virt.ClusterVirtualImage{} + err := c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{Name: cvmiName}, cvmi) + if err == nil { + return cvmi, nil + } + if !apierrors.IsNotFound(err) { + return nil, err + } + + // Create new + cvmi = &virt.ClusterVirtualImage{ + ObjectMeta: metav1.ObjectMeta{ + Name: cvmiName, + }, + Spec: virt.ClusterVirtualImageSpec{ + DataSource: virt.ClusterVirtualImageDataSource{ + Type: "HTTP", + HTTP: &virt.DataSourceHTTP{URL: imageURL}, + }, + }, + } + + err = c.kubeClient.ControllerRuntimeClient().Create(ctx, cvmi) + if err != nil { + return nil, err + } + + return cvmi, nil +} + +func (c *client) createOrGetIPClaim(ctx context.Context, namespace, vmName, ip string) (*virt.VirtualMachineIPAddress, error) { + claimName := fmt.Sprintf("%s-ipaddress-0", vmName) + + claim := &virt.VirtualMachineIPAddress{} + err := c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{ + Namespace: namespace, + Name: claimName, + }, claim) + if err == nil { + return claim, nil + } + if !apierrors.IsNotFound(err) { + return nil, err + } + + claim = &virt.VirtualMachineIPAddress{ + ObjectMeta: metav1.ObjectMeta{ + Name: claimName, + Namespace: namespace, + }, + Spec: virt.VirtualMachineIPAddressSpec{ + Type: virt.VirtualMachineIPAddressTypeStatic, + StaticIP: ip, + }, + } + + err = c.kubeClient.ControllerRuntimeClient().Create(ctx, claim) + if err != nil { + return nil, err + } + + return claim, nil +} + +func (c *client) createOrGetSystemDisk(ctx context.Context, namespace, vmName, storageClass string, diskSize int, cvmi *virt.ClusterVirtualImage) (*virt.VirtualDisk, error) { + diskName := fmt.Sprintf("%s-system", vmName) + + disk := &virt.VirtualDisk{} + err := c.kubeClient.ControllerRuntimeClient().Get(ctx, ctrlrtclient.ObjectKey{ + Namespace: namespace, + Name: diskName, + }, disk) + if err == nil { + return disk, nil + } + if !apierrors.IsNotFound(err) { + return nil, err + } + + var sc *string = nil + if storageClass != "" { + sc = &storageClass + } + + disk = &virt.VirtualDisk{ + ObjectMeta: metav1.ObjectMeta{ + Name: diskName, + Namespace: namespace, + }, + Spec: virt.VirtualDiskSpec{ + PersistentVolumeClaim: virt.VirtualDiskPersistentVolumeClaim{ + Size: resource.NewQuantity(int64(diskSize)*1024*1024*1024, resource.BinarySI), + StorageClass: sc, + }, + DataSource: &virt.VirtualDiskDataSource{ + Type: virt.DataSourceTypeObjectRef, + ObjectRef: &virt.VirtualDiskObjectRef{ + Kind: virt.ClusterVirtualImageKind, + Name: cvmi.Name, + }, + }, + }, + } + + err = c.kubeClient.ControllerRuntimeClient().Create(ctx, disk) + if err != nil { + return nil, err + } + + return disk, nil +} + diff --git a/e2e-tests/internal/logger/logger.go b/e2e-tests/internal/logger/logger.go new file mode 100644 index 0000000..796cb95 --- /dev/null +++ b/e2e-tests/internal/logger/logger.go @@ -0,0 +1,123 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package logger + +import ( + "fmt" + "log" + "os" +) + +// Logger interface for dependency injection +type Logger interface { + Debug(msg string, args ...interface{}) + Info(msg string, args ...interface{}) + Warn(msg string, args ...interface{}) + Error(msg string, args ...interface{}) + Fatal(msg string, args ...interface{}) +} + +// SimpleLogger is a basic implementation of Logger +type SimpleLogger struct { + debug bool + logger *log.Logger +} + +// NewSimpleLogger creates a new simple logger +func NewSimpleLogger(debug bool) *SimpleLogger { + return &SimpleLogger{ + debug: debug, + logger: log.New(os.Stdout, "", log.LstdFlags), + } +} + +func (l *SimpleLogger) Debug(msg string, args ...interface{}) { + if l.debug { + l.logger.Printf("[DEBUG] "+msg, args...) + } +} + +func (l *SimpleLogger) Info(msg string, args ...interface{}) { + l.logger.Printf("[INFO] "+msg, args...) +} + +func (l *SimpleLogger) Warn(msg string, args ...interface{}) { + l.logger.Printf("[WARN] "+msg, args...) +} + +func (l *SimpleLogger) Error(msg string, args ...interface{}) { + l.logger.Printf("[ERROR] "+msg, args...) +} + +func (l *SimpleLogger) Fatal(msg string, args ...interface{}) { + l.logger.Fatalf("[FATAL] "+msg, args...) +} + +// Default logger instance +var Default Logger = NewSimpleLogger(false) + +// SetDefault sets the default logger +func SetDefault(l Logger) { + Default = l +} + +// Convenience functions +func Debug(msg string, args ...interface{}) { + Default.Debug(msg, args...) +} + +func Info(msg string, args ...interface{}) { + Default.Info(msg, args...) +} + +func Warn(msg string, args ...interface{}) { + Default.Warn(msg, args...) +} + +func Error(msg string, args ...interface{}) { + Default.Error(msg, args...) +} + +func Fatal(msg string, args ...interface{}) { + Default.Fatal(msg, args...) +} + +// Debugf formats and logs a debug message +func Debugf(format string, args ...interface{}) { + Debug(format, args...) +} + +// Infof formats and logs an info message +func Infof(format string, args ...interface{}) { + Info(format, args...) +} + +// Warnf formats and logs a warn message +func Warnf(format string, args ...interface{}) { + Warn(format, args...) +} + +// Errorf formats and logs an error message +func Errorf(format string, args ...interface{}) { + Error(format, args...) +} + +// Fatalf formats and logs a fatal message and exits +func Fatalf(format string, args ...interface{}) { + Fatal(fmt.Sprintf(format, args...)) +} + diff --git a/testkit_v2/tests/00_healthcheck_test.go b/e2e-tests/internal/utils/crypto.go similarity index 61% rename from testkit_v2/tests/00_healthcheck_test.go rename to e2e-tests/internal/utils/crypto.go index f56a509..403f422 100644 --- a/testkit_v2/tests/00_healthcheck_test.go +++ b/e2e-tests/internal/utils/crypto.go @@ -14,23 +14,16 @@ See the License for the specific language governing permissions and limitations under the License. */ -package integration +package utils import ( - "testing" - - util "github.com/deckhouse/sds-e2e/util" + "crypto/md5" + "fmt" ) -func TestNodeHealthCheck(t *testing.T) { - cluster := util.EnsureCluster("", "") - - nodeMap := cluster.MapLabelNodes(nil) - for label, nodes := range nodeMap { - if len(nodes) == 0 { - t.Errorf("No %s nodes", label) - } else { - util.Infof("%s nodes: %d", label, len(nodes)) - } - } +// HashMd5 computes MD5 hash of input string and returns hex representation +func HashMd5(input string) string { + hash := md5.Sum([]byte(input)) + return fmt.Sprintf("%x", hash) } + diff --git a/e2e-tests/internal/utils/retry.go b/e2e-tests/internal/utils/retry.go new file mode 100644 index 0000000..726492c --- /dev/null +++ b/e2e-tests/internal/utils/retry.go @@ -0,0 +1,75 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package utils + +import ( + "fmt" + "time" +) + +// RetrySec retries a function with exponential backoff up to maxSeconds +func RetrySec(maxSeconds int, fn func() error) error { + start := time.Now() + attempt := 0 + + for { + err := fn() + if err == nil { + return nil + } + + elapsed := time.Since(start) + if elapsed >= time.Duration(maxSeconds)*time.Second { + return fmt.Errorf("timeout after %d seconds: %w", maxSeconds, err) + } + + attempt++ + backoff := time.Duration(attempt) * time.Second + if backoff > 10*time.Second { + backoff = 10 * time.Second + } + + time.Sleep(backoff) + } +} + +// RetryWithTimeout retries a function until timeout +func RetryWithTimeout(timeout time.Duration, fn func() error) error { + start := time.Now() + attempt := 0 + + for { + err := fn() + if err == nil { + return nil + } + + elapsed := time.Since(start) + if elapsed >= timeout { + return fmt.Errorf("timeout after %v: %w", timeout, err) + } + + attempt++ + backoff := time.Duration(attempt) * time.Second + if backoff > 10*time.Second { + backoff = 10 * time.Second + } + + time.Sleep(backoff) + } +} + diff --git a/e2e-tests/pkg/cluster/interface.go b/e2e-tests/pkg/cluster/interface.go new file mode 100644 index 0000000..a94abcb --- /dev/null +++ b/e2e-tests/pkg/cluster/interface.go @@ -0,0 +1,62 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "context" + + "github.com/deckhouse/sds-e2e-tests/internal/config" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes/deckhouse" + "github.com/deckhouse/sds-e2e-tests/internal/kubernetes/virtualization" +) + +// Cluster is the main interface for interacting with a Kubernetes cluster +type Cluster interface { + // Name returns the cluster name + Name() string + + // Context returns the context associated with this cluster + Context() context.Context // TODO amarkov: I'm not sure how good it is in Go to store context. Should we rather pass it in each call as usual? - Context should better be passed as an argument, do not keep in in struct. + + // Virtualization returns the virtualization client for VM operations + Virtualization() virtualization.Client // TODO amarkov: Is the VirtualizationClient this one? https://github.com/deckhouse/virtualization/blob/main/api/client/kubeclient/client.go#L53 + + // Deckhouse returns the deckhouse client for module operations + Deckhouse() deckhouse.Client + + // Lifecycle + EnsureReady(ctx context.Context) error + Close() error +} + +// Manager manages cluster lifecycle +type Manager interface { + // GetOrCreate retrieves an existing cluster or creates a new one + GetOrCreate(ctx context.Context, configPath, name string) (Cluster, error) + + // Get retrieves an existing cluster by name + Get(name string) (Cluster, bool) + + // CloseAll closes all managed clusters + CloseAll() error +} + +// Builder builds and provisions new clusters +type Builder interface { + // BuildCluster creates a new nested Kubernetes cluster on the base cluster + BuildCluster(ctx context.Context, baseCluster Cluster, cfg *config.DKPClusterConfig) (Cluster, error) +} diff --git a/e2e-tests/pkg/testkit/cluster.go b/e2e-tests/pkg/testkit/cluster.go new file mode 100644 index 0000000..355926c --- /dev/null +++ b/e2e-tests/pkg/testkit/cluster.go @@ -0,0 +1,40 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testkit + +import ( + "context" + + internalCluster "github.com/deckhouse/sds-e2e-tests/internal/cluster" + "github.com/deckhouse/sds-e2e-tests/internal/config" + "github.com/deckhouse/sds-e2e-tests/pkg/cluster" +) + +// GetCluster returns or creates a cluster for testing +func GetCluster(ctx context.Context, configPath, name string) (cluster.Cluster, error) { + manager := internalCluster.NewManager() + return manager.GetOrCreate(ctx, configPath, name) +} + +// BuildTestCluster builds a new test cluster from configuration +func BuildTestCluster(ctx context.Context, baseCluster cluster.Cluster, dkpCfg *config.DKPClusterConfig) (cluster.Cluster, error) { + cfg := config.Load() + manager := internalCluster.NewManager() + builder := internalCluster.NewBuilder(cfg, manager, nil) // SSH factory would be passed here + return builder.BuildCluster(ctx, baseCluster, dkpCfg) +} + diff --git a/e2e-tests/tests/cluster_creation_test.go b/e2e-tests/tests/cluster_creation_test.go new file mode 100644 index 0000000..39099cd --- /dev/null +++ b/e2e-tests/tests/cluster_creation_test.go @@ -0,0 +1,145 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// TODO: Need to add test suite for test package +// + +package integration + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/deckhouse/sds-e2e-tests/internal/config" + "github.com/deckhouse/sds-e2e-tests/pkg/cluster" + "github.com/deckhouse/sds-e2e-tests/pkg/testkit" +) + +var _ = Describe("Cluster Creation", func() { + var ( + baseCluster cluster.Cluster + testCluster cluster.Cluster + clusterCfg *config.DKPClusterConfig + ) + + BeforeEach(func(ctx SpecContext) { + + // Load base cluster + cfg := config.Load() + var err error + baseCluster, err = testkit.GetCluster(ctx, cfg.BaseCluster.KubeConfig, "") + Expect(err).NotTo(HaveOccurred()) + + // Define test cluster configuration + clusterCfg = &config.DKPClusterConfig{ + ClusterDefinition: config.ClusterDefinition{ + Masters: []config.ClusterNode{ + { + Hostname: "master-1", + HostType: config.HostTypeVM, + Role: config.ClusterRoleMaster, + OSType: config.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: config.NodeAuth{ + Method: config.AuthMethodSSHKey, + User: "user", + SSHKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8WyGvnBNQp+v6CUweF1QYCRtR7Do/IA8IA2uMd2HuBsddFrc5xYon2ZtEvypZC4Vm1CzgcgUm9UkHgxytKEB4zOOWkmqFP62OSLNyuWMaFEW1fb0EDenup6B5SrjnA8ckm4Hf2NSLvwW9yS98TfN3nqPOPJKfQsN+OTiCerTtNyXjca//ppuGKsQd99jG7SqE9aDQ3sYCXatM53SXqhxS2nTew82bmzVmKXDxcIzVrS9f+2WmXIdY2cKo2I352yKWOIp1Nk0uji8ozLPHFQGvbAG8DGG1KNVcBl2qYUcttmCpN+iXEcGqyn/atUVJJMnZXGtp0fiL1rMLqAd/bb6TFNzZFSsS+zqGesxqLePe32vLCQ3xursP3BRZkrScM+JzIqevfP63INHJEZfYlUf4Ic+gfliS2yA1LwhU7hD4LSVXMQynlF9WeGjuv6ZYxmO8hC6IWCqWnIUqKUiGtvBSPXwsZo7wgljBr4ykJgBzS9MjZ0fzz1JKe80tH6clpjIOn6ReBPwQBq2zmDDrpa5GVqqqjXhRQuA0AfpHdhs5UKxs1PBr7/PTLA7PI39xkOAE/Zj1TYQ2dmqvpskshi7AtBStjinQBAlLXysLSHBtO+3+PLAYcMZMVfb0bVqfGGludO2prvXrrWWTku0eOsA5IRahrRdGhv5zhKgFV7cwUQ== ayakubov@MacBook-Pro-Alexey.local", + }, + CPU: 4, + RAM: 8, + DiskSize: 30, + }, + }, + Workers: []config.ClusterNode{ + { + Hostname: "worker-1", + HostType: config.HostTypeVM, + Role: config.ClusterRoleWorker, + OSType: config.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: config.NodeAuth{ + Method: config.AuthMethodSSHKey, + User: "user", + SSHKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8WyGvnBNQp+v6CUweF1QYCRtR7Do/IA8IA2uMd2HuBsddFrc5xYon2ZtEvypZC4Vm1CzgcgUm9UkHgxytKEB4zOOWkmqFP62OSLNyuWMaFEW1fb0EDenup6B5SrjnA8ckm4Hf2NSLvwW9yS98TfN3nqPOPJKfQsN+OTiCerTtNyXjca//ppuGKsQd99jG7SqE9aDQ3sYCXatM53SXqhxS2nTew82bmzVmKXDxcIzVrS9f+2WmXIdY2cKo2I352yKWOIp1Nk0uji8ozLPHFQGvbAG8DGG1KNVcBl2qYUcttmCpN+iXEcGqyn/atUVJJMnZXGtp0fiL1rMLqAd/bb6TFNzZFSsS+zqGesxqLePe32vLCQ3xursP3BRZkrScM+JzIqevfP63INHJEZfYlUf4Ic+gfliS2yA1LwhU7hD4LSVXMQynlF9WeGjuv6ZYxmO8hC6IWCqWnIUqKUiGtvBSPXwsZo7wgljBr4ykJgBzS9MjZ0fzz1JKe80tH6clpjIOn6ReBPwQBq2zmDDrpa5GVqqqjXhRQuA0AfpHdhs5UKxs1PBr7/PTLA7PI39xkOAE/Zj1TYQ2dmqvpskshi7AtBStjinQBAlLXysLSHBtO+3+PLAYcMZMVfb0bVqfGGludO2prvXrrWWTku0eOsA5IRahrRdGhv5zhKgFV7cwUQ== ayakubov@MacBook-Pro-Alexey.local", + }, + CPU: 2, + RAM: 6, + DiskSize: 30, + }, + { + Hostname: "worker-2", + HostType: config.HostTypeVM, + Role: config.ClusterRoleWorker, + OSType: config.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: config.NodeAuth{ + Method: config.AuthMethodSSHKey, + User: "user", + SSHKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8WyGvnBNQp+v6CUweF1QYCRtR7Do/IA8IA2uMd2HuBsddFrc5xYon2ZtEvypZC4Vm1CzgcgUm9UkHgxytKEB4zOOWkmqFP62OSLNyuWMaFEW1fb0EDenup6B5SrjnA8ckm4Hf2NSLvwW9yS98TfN3nqPOPJKfQsN+OTiCerTtNyXjca//ppuGKsQd99jG7SqE9aDQ3sYCXatM53SXqhxS2nTew82bmzVmKXDxcIzVrS9f+2WmXIdY2cKo2I352yKWOIp1Nk0uji8ozLPHFQGvbAG8DGG1KNVcBl2qYUcttmCpN+iXEcGqyn/atUVJJMnZXGtp0fiL1rMLqAd/bb6TFNzZFSsS+zqGesxqLePe32vLCQ3xursP3BRZkrScM+JzIqevfP63INHJEZfYlUf4Ic+gfliS2yA1LwhU7hD4LSVXMQynlF9WeGjuv6ZYxmO8hC6IWCqWnIUqKUiGtvBSPXwsZo7wgljBr4ykJgBzS9MjZ0fzz1JKe80tH6clpjIOn6ReBPwQBq2zmDDrpa5GVqqqjXhRQuA0AfpHdhs5UKxs1PBr7/PTLA7PI39xkOAE/Zj1TYQ2dmqvpskshi7AtBStjinQBAlLXysLSHBtO+3+PLAYcMZMVfb0bVqfGGludO2prvXrrWWTku0eOsA5IRahrRdGhv5zhKgFV7cwUQ== ayakubov@MacBook-Pro-Alexey.local", + }, + CPU: 2, + RAM: 6, + DiskSize: 30, + }, + }, + }, + KubernetesVersion: "Automatic", + PodSubnetCIDR: "10.112.0.0/16", + ServiceSubnetCIDR: "10.225.0.0/16", + ClusterDomain: "cluster.local", + LicenseKey: "", + RegistryRepo: "dev-registry.deckhouse.io/sys/deckhouse-oss", + } + }) + + It("should create a nested Kubernetes cluster with all modules enabled", func(ctx SpecContext) { + var err error + + By("Building the test cluster", func() { + testCluster, err = testkit.BuildTestCluster(ctx, baseCluster, clusterCfg) + Expect(err).NotTo(HaveOccurred()) + }) + + By("Verifying the cluster is ready", func() { + err = testCluster.EnsureReady(ctx) // TODO - use Eventually instead of Expect + Expect(err).NotTo(HaveOccurred()) // there must be Eventually + }) + + By("Verifying modules are enabled", func() { + // Check that modules are enabled + dhClient := testCluster.Deckhouse() + + modules := []string{ + "snapshot-controller", + "sds-local-volume", + "sds-node-configurator", + "sds-replicated-volume", + } + + for _, moduleName := range modules { + err = dhClient.EnsureModuleEnabled(ctx, &config.ModuleConfig{ + Name: moduleName, + Version: 1, + Enabled: true, + }) + Expect(err).NotTo(HaveOccurred()) + } + }) + }) + + AfterEach(func() { + if testCluster != nil { + _ = testCluster.Close() + } + }) +}) diff --git a/images/sds-lvm-e2e/src/LICENSE b/legacy/images/sds-lvm-e2e/src/LICENSE similarity index 100% rename from images/sds-lvm-e2e/src/LICENSE rename to legacy/images/sds-lvm-e2e/src/LICENSE diff --git a/images/sds-lvm-e2e/src/funcs/const.go b/legacy/images/sds-lvm-e2e/src/funcs/const.go similarity index 100% rename from images/sds-lvm-e2e/src/funcs/const.go rename to legacy/images/sds-lvm-e2e/src/funcs/const.go diff --git a/images/sds-lvm-e2e/src/funcs/lvm_func.go b/legacy/images/sds-lvm-e2e/src/funcs/lvm_func.go similarity index 100% rename from images/sds-lvm-e2e/src/funcs/lvm_func.go rename to legacy/images/sds-lvm-e2e/src/funcs/lvm_func.go diff --git a/images/sds-lvm-e2e/src/funcs/pod.go b/legacy/images/sds-lvm-e2e/src/funcs/pod.go similarity index 100% rename from images/sds-lvm-e2e/src/funcs/pod.go rename to legacy/images/sds-lvm-e2e/src/funcs/pod.go diff --git a/images/sds-lvm-e2e/src/funcs/pvc.go b/legacy/images/sds-lvm-e2e/src/funcs/pvc.go similarity index 100% rename from images/sds-lvm-e2e/src/funcs/pvc.go rename to legacy/images/sds-lvm-e2e/src/funcs/pvc.go diff --git a/images/sds-lvm-e2e/src/funcs/storage_class.go b/legacy/images/sds-lvm-e2e/src/funcs/storage_class.go similarity index 100% rename from images/sds-lvm-e2e/src/funcs/storage_class.go rename to legacy/images/sds-lvm-e2e/src/funcs/storage_class.go diff --git a/images/sds-lvm-e2e/src/go.mod b/legacy/images/sds-lvm-e2e/src/go.mod similarity index 100% rename from images/sds-lvm-e2e/src/go.mod rename to legacy/images/sds-lvm-e2e/src/go.mod diff --git a/images/sds-lvm-e2e/src/go.sum b/legacy/images/sds-lvm-e2e/src/go.sum similarity index 100% rename from images/sds-lvm-e2e/src/go.sum rename to legacy/images/sds-lvm-e2e/src/go.sum diff --git a/images/sds-lvm-e2e/src/tests/01_create_storage_classes_test.go b/legacy/images/sds-lvm-e2e/src/tests/01_create_storage_classes_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/01_create_storage_classes_test.go rename to legacy/images/sds-lvm-e2e/src/tests/01_create_storage_classes_test.go diff --git a/images/sds-lvm-e2e/src/tests/case_23_25_test.go b/legacy/images/sds-lvm-e2e/src/tests/case_23_25_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/case_23_25_test.go rename to legacy/images/sds-lvm-e2e/src/tests/case_23_25_test.go diff --git a/images/sds-lvm-e2e/src/tests/case_26_test.go b/legacy/images/sds-lvm-e2e/src/tests/case_26_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/case_26_test.go rename to legacy/images/sds-lvm-e2e/src/tests/case_26_test.go diff --git a/images/sds-lvm-e2e/src/tests/case_27_test.go b/legacy/images/sds-lvm-e2e/src/tests/case_27_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/case_27_test.go rename to legacy/images/sds-lvm-e2e/src/tests/case_27_test.go diff --git a/images/sds-lvm-e2e/src/tests/case_28_test.go b/legacy/images/sds-lvm-e2e/src/tests/case_28_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/case_28_test.go rename to legacy/images/sds-lvm-e2e/src/tests/case_28_test.go diff --git a/images/sds-lvm-e2e/src/tests/case_600_test.go b/legacy/images/sds-lvm-e2e/src/tests/case_600_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/case_600_test.go rename to legacy/images/sds-lvm-e2e/src/tests/case_600_test.go diff --git a/images/sds-lvm-e2e/src/tests/init.go b/legacy/images/sds-lvm-e2e/src/tests/init.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/init.go rename to legacy/images/sds-lvm-e2e/src/tests/init.go diff --git a/images/sds-lvm-e2e/src/tests/lvm_func_test.go b/legacy/images/sds-lvm-e2e/src/tests/lvm_func_test.go similarity index 100% rename from images/sds-lvm-e2e/src/tests/lvm_func_test.go rename to legacy/images/sds-lvm-e2e/src/tests/lvm_func_test.go diff --git a/images/sds-lvm-e2e/src/v1alpha1/lvm_logical_volume.go b/legacy/images/sds-lvm-e2e/src/v1alpha1/lvm_logical_volume.go similarity index 100% rename from images/sds-lvm-e2e/src/v1alpha1/lvm_logical_volume.go rename to legacy/images/sds-lvm-e2e/src/v1alpha1/lvm_logical_volume.go diff --git a/images/sds-lvm-e2e/src/v1alpha1/lvm_volume_group.go b/legacy/images/sds-lvm-e2e/src/v1alpha1/lvm_volume_group.go similarity index 100% rename from images/sds-lvm-e2e/src/v1alpha1/lvm_volume_group.go rename to legacy/images/sds-lvm-e2e/src/v1alpha1/lvm_volume_group.go diff --git a/images/sds-lvm-e2e/src/v1alpha1/register.go b/legacy/images/sds-lvm-e2e/src/v1alpha1/register.go similarity index 100% rename from images/sds-lvm-e2e/src/v1alpha1/register.go rename to legacy/images/sds-lvm-e2e/src/v1alpha1/register.go diff --git a/images/sds-lvm-e2e/src/v1alpha1/zz_generated.deepcopy.go b/legacy/images/sds-lvm-e2e/src/v1alpha1/zz_generated.deepcopy.go similarity index 100% rename from images/sds-lvm-e2e/src/v1alpha1/zz_generated.deepcopy.go rename to legacy/images/sds-lvm-e2e/src/v1alpha1/zz_generated.deepcopy.go diff --git a/templates/deployment-and-roles.yaml b/legacy/templates/deployment-and-roles.yaml similarity index 100% rename from templates/deployment-and-roles.yaml rename to legacy/templates/deployment-and-roles.yaml diff --git a/testkit/LICENSE b/legacy/testkit/LICENSE similarity index 100% rename from testkit/LICENSE rename to legacy/testkit/LICENSE diff --git a/testkit/clusterManagement/cluster.go b/legacy/testkit/clusterManagement/cluster.go similarity index 100% rename from testkit/clusterManagement/cluster.go rename to legacy/testkit/clusterManagement/cluster.go diff --git a/testkit/config.yml.tpl b/legacy/testkit/config.yml.tpl similarity index 100% rename from testkit/config.yml.tpl rename to legacy/testkit/config.yml.tpl diff --git a/testkit/createuser.sh b/legacy/testkit/createuser.sh similarity index 100% rename from testkit/createuser.sh rename to legacy/testkit/createuser.sh diff --git a/testkit/funcs/const.go b/legacy/testkit/funcs/const.go similarity index 100% rename from testkit/funcs/const.go rename to legacy/testkit/funcs/const.go diff --git a/testkit/funcs/helper.go b/legacy/testkit/funcs/helper.go similarity index 100% rename from testkit/funcs/helper.go rename to legacy/testkit/funcs/helper.go diff --git a/testkit/funcs/kubeclient.go b/legacy/testkit/funcs/kubeclient.go similarity index 100% rename from testkit/funcs/kubeclient.go rename to legacy/testkit/funcs/kubeclient.go diff --git a/testkit/funcs/namespace.go b/legacy/testkit/funcs/namespace.go similarity index 100% rename from testkit/funcs/namespace.go rename to legacy/testkit/funcs/namespace.go diff --git a/testkit/funcs/replicatedstorage_pool_class.go b/legacy/testkit/funcs/replicatedstorage_pool_class.go similarity index 100% rename from testkit/funcs/replicatedstorage_pool_class.go rename to legacy/testkit/funcs/replicatedstorage_pool_class.go diff --git a/testkit/funcs/service.go b/legacy/testkit/funcs/service.go similarity index 100% rename from testkit/funcs/service.go rename to legacy/testkit/funcs/service.go diff --git a/testkit/funcs/ssh.go b/legacy/testkit/funcs/ssh.go similarity index 100% rename from testkit/funcs/ssh.go rename to legacy/testkit/funcs/ssh.go diff --git a/testkit/funcs/statefulset.go b/legacy/testkit/funcs/statefulset.go similarity index 100% rename from testkit/funcs/statefulset.go rename to legacy/testkit/funcs/statefulset.go diff --git a/testkit/funcs/virtual.go b/legacy/testkit/funcs/virtual.go similarity index 100% rename from testkit/funcs/virtual.go rename to legacy/testkit/funcs/virtual.go diff --git a/testkit/go.mod b/legacy/testkit/go.mod similarity index 100% rename from testkit/go.mod rename to legacy/testkit/go.mod diff --git a/testkit/go.sum b/legacy/testkit/go.sum similarity index 100% rename from testkit/go.sum rename to legacy/testkit/go.sum diff --git a/testkit/main.go b/legacy/testkit/main.go similarity index 100% rename from testkit/main.go rename to legacy/testkit/main.go diff --git a/testkit/ms.yml.tpl b/legacy/testkit/ms.yml.tpl similarity index 100% rename from testkit/ms.yml.tpl rename to legacy/testkit/ms.yml.tpl diff --git a/testkit/resources.yml.tpl b/legacy/testkit/resources.yml.tpl similarity index 100% rename from testkit/resources.yml.tpl rename to legacy/testkit/resources.yml.tpl diff --git a/testkit/tests/sds-node-configurator/00_lvg_create_test.go b/legacy/testkit/tests/sds-node-configurator/00_lvg_create_test.go similarity index 100% rename from testkit/tests/sds-node-configurator/00_lvg_create_test.go rename to legacy/testkit/tests/sds-node-configurator/00_lvg_create_test.go diff --git a/testkit/tests/sds-node-configurator/01_lvg_change_size_test.go b/legacy/testkit/tests/sds-node-configurator/01_lvg_change_size_test.go similarity index 100% rename from testkit/tests/sds-node-configurator/01_lvg_change_size_test.go rename to legacy/testkit/tests/sds-node-configurator/01_lvg_change_size_test.go diff --git a/testkit/tests/sds-node-configurator/02_lvg_remove_test.go b/legacy/testkit/tests/sds-node-configurator/02_lvg_remove_test.go similarity index 100% rename from testkit/tests/sds-node-configurator/02_lvg_remove_test.go rename to legacy/testkit/tests/sds-node-configurator/02_lvg_remove_test.go diff --git a/testkit/tests/stress/00_delete_namespace_test.go b/legacy/testkit/tests/stress/00_delete_namespace_test.go similarity index 100% rename from testkit/tests/stress/00_delete_namespace_test.go rename to legacy/testkit/tests/stress/00_delete_namespace_test.go diff --git a/testkit/tests/stress/01_create_sp_test.go b/legacy/testkit/tests/stress/01_create_sp_test.go similarity index 100% rename from testkit/tests/stress/01_create_sp_test.go rename to legacy/testkit/tests/stress/01_create_sp_test.go diff --git a/testkit/tests/stress/02_create_namespace_test.go b/legacy/testkit/tests/stress/02_create_namespace_test.go similarity index 100% rename from testkit/tests/stress/02_create_namespace_test.go rename to legacy/testkit/tests/stress/02_create_namespace_test.go diff --git a/testkit/tests/stress/03_create_sts_with_logs_test.go b/legacy/testkit/tests/stress/03_create_sts_with_logs_test.go similarity index 100% rename from testkit/tests/stress/03_create_sts_with_logs_test.go rename to legacy/testkit/tests/stress/03_create_sts_with_logs_test.go diff --git a/testkit/tests/stress/04_wait_for_pod_readiness_test.go b/legacy/testkit/tests/stress/04_wait_for_pod_readiness_test.go similarity index 100% rename from testkit/tests/stress/04_wait_for_pod_readiness_test.go rename to legacy/testkit/tests/stress/04_wait_for_pod_readiness_test.go diff --git a/testkit/tests/stress/05_change_pv_size_test.go b/legacy/testkit/tests/stress/05_change_pv_size_test.go similarity index 100% rename from testkit/tests/stress/05_change_pv_size_test.go rename to legacy/testkit/tests/stress/05_change_pv_size_test.go diff --git a/testkit/tests/stress/06_delete_sts_test.go b/legacy/testkit/tests/stress/06_delete_sts_test.go similarity index 100% rename from testkit/tests/stress/06_delete_sts_test.go rename to legacy/testkit/tests/stress/06_delete_sts_test.go diff --git a/testkit/tests/stress/07_create_vm_test.go b/legacy/testkit/tests/stress/07_create_vm_test.go similarity index 100% rename from testkit/tests/stress/07_create_vm_test.go rename to legacy/testkit/tests/stress/07_create_vm_test.go diff --git a/testkit/tests/stress/99_delete_namespace_test.go b/legacy/testkit/tests/stress/99_delete_namespace_test.go similarity index 100% rename from testkit/tests/stress/99_delete_namespace_test.go rename to legacy/testkit/tests/stress/99_delete_namespace_test.go diff --git a/testkit/tests/stress/README.md b/legacy/testkit/tests/stress/README.md similarity index 100% rename from testkit/tests/stress/README.md rename to legacy/testkit/tests/stress/README.md diff --git a/testkit/tests/stress/init.go b/legacy/testkit/tests/stress/init.go similarity index 100% rename from testkit/tests/stress/init.go rename to legacy/testkit/tests/stress/init.go diff --git a/testkit_v2/LICENSE b/legacy/testkit_v2/LICENSE similarity index 100% rename from testkit_v2/LICENSE rename to legacy/testkit_v2/LICENSE diff --git a/testkit_v2/README.md b/legacy/testkit_v2/README.md similarity index 100% rename from testkit_v2/README.md rename to legacy/testkit_v2/README.md diff --git a/legacy/testkit_v2/data/config.yml b/legacy/testkit_v2/data/config.yml new file mode 100644 index 0000000..73e98ec --- /dev/null +++ b/legacy/testkit_v2/data/config.yml @@ -0,0 +1,118 @@ +# General cluster parameters. +# https://deckhouse.io/documentation/v1/installing/configuration.html#clusterconfiguration +apiVersion: deckhouse.io/v1 +kind: ClusterConfiguration +clusterType: Static +# Address space of the cluster's Pods. +podSubnetCIDR: 10.112.0.0/16 +# Address space of the cluster's services. +serviceSubnetCIDR: 10.225.0.0/16 +kubernetesVersion: Automatic +# Cluster domain (used for local routing). +clusterDomain: "cluster.local" +--- +# Settings for the bootstrapping the Deckhouse cluster +# https://deckhouse.io/documentation/v1/installing/configuration.html#initconfiguration +apiVersion: deckhouse.io/v1 +kind: InitConfiguration +deckhouse: + # Address of the Docker registry where the Deckhouse images are located + imagesRepo: dev-registry.deckhouse.io/sys/deckhouse-oss + # A special string with your token to access Docker registry (generated automatically for your license token) + registryDockerCfg: eyJhdXRocyI6eyJkZXYtcmVnaXN0cnkuZGVja2hvdXNlLmlvIjp7ImF1dGgiOiJiR2xqWlc1elpTMTBiMnRsYmpwWU4xbHBaMWhTUWxBM1pscHdOVGxwZW5acFUyWkNWelZFTlZGTmIwaFJlUT09In19fQ== + devBranch: main +--- +# Deckhouse module settings. +# https://deckhouse.io/documentation/v1/modules/002-deckhouse/configuration.html +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: deckhouse +spec: + version: 1 + enabled: true + settings: + bundle: Default + releaseChannel: Stable + logLevel: Info +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: flant-integration +spec: + enabled: false + settings: + kubeall: + host: fake.host + version: 1 +--- +# Global Deckhouse settings. +# https://deckhouse.ru/documentation/v1/deckhouse-configure-global.html#parameters +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: global +spec: + version: 1 + settings: + modules: + publicDomainTemplate: "%s.virtualmachines.local" +--- +# user-authn module settings. +# https://deckhouse.io/documentation/v1/modules/150-user-authn/configuration.html +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: user-authn +spec: + version: 1 + enabled: true + settings: + controlPlaneConfigurator: + dexCAMode: DoNotNeed + # Enabling access to the API server through Ingress. + # https://deckhouse.io/documentation/v1/modules/150-user-authn/configuration.html#parameters-publishapi + publishAPI: + enabled: true + https: + mode: Global + global: + kubeconfigGeneratorMasterCA: "" +--- +# cni-cilium module settings. +# https://deckhouse.io/documentation/v1/modules/021-cni-cilium/configuration.html +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: cni-cilium +spec: + version: 1 + # Enable cni-cilium module + enabled: true + settings: + # cni-cilium module settings + # https://deckhouse.io/documentation/v1/modules/021-cni-cilium/configuration.html + tunnelMode: VXLAN +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: sds-node-configurator +spec: + enabled: true + settings: + enableThinProvisioning: true + version: 1 +--- +# Static cluster settings. +# https://deckhouse.io/documentation/v1/installing/configuration.html#staticclusterconfiguration +apiVersion: deckhouse.io/v1 +kind: StaticClusterConfiguration +# List of internal cluster networks (e.g., '10.0.4.0/24'), which is +# used for linking Kubernetes components (kube-apiserver, kubelet etc.). +# If every node in cluster has only one network interface +# StaticClusterConfiguration resource can be skipped. +internalNetworkCIDRs: +- 10.10.10.0/24 +- 10.211.1.0/24 diff --git a/testkit_v2/data/config.yml.tpl b/legacy/testkit_v2/data/config.yml.tpl similarity index 100% rename from testkit_v2/data/config.yml.tpl rename to legacy/testkit_v2/data/config.yml.tpl diff --git a/legacy/testkit_v2/data/resources.yml b/legacy/testkit_v2/data/resources.yml new file mode 100644 index 0000000..52587e2 --- /dev/null +++ b/legacy/testkit_v2/data/resources.yml @@ -0,0 +1,117 @@ +#--- +#apiVersion: deckhouse.io/v1 +#kind: User +#metadata: +# name: admin +#spec: +# email: admin@cluster.local +# password: JDJ5JDEwJGxTSHpOU25EOXlYUUY1UlVTRTB2V09zektSU29hdGhjOC5XY1o0ckEvblZZL0lvRGdVSmIyCgo= +#--- +#apiVersion: deckhouse.io/v1alpha1 +#kind: Group +#metadata: +# name: admins +#spec: +# name: admins +# members: +# - kind: User +# name: admin +--- +apiVersion: deckhouse.io/v1 +kind: NodeGroup +metadata: + name: worker +spec: + disruptions: + approvalMode: Automatic + kubelet: + containerLogMaxFiles: 4 + containerLogMaxSize: 50Mi + maxPods: 200 + resourceReservation: + mode: "Off" + nodeType: Static +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModulePullOverride +metadata: + name: snapshot-controller +spec: + imageTag: main + scanInterval: 15s + source: deckhouse +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: snapshot-controller +spec: + enabled: true +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModulePullOverride +metadata: + name: sds-local-volume +spec: + imageTag: main + scanInterval: 15s + source: deckhouse +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: sds-local-volume +spec: + enabled: true +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModulePullOverride +metadata: + name: sds-replicated-volume +spec: + imageTag: main + scanInterval: 15s + source: deckhouse +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: sds-replicated-volume +spec: + enabled: true + settings: + logLevel: DEBUG + version: 1 +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModulePullOverride +metadata: + name: sds-node-configurator +spec: + imageTag: main + scanInterval: 15s + source: deckhouse +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleConfig +metadata: + name: sds-node-configurator +spec: + enabled: true +--- +apiVersion: deckhouse.io/v1alpha1 +kind: ModuleSource +metadata: + name: deckhouse + annotations: + meta.helm.sh/release-name: deckhouse + meta.helm.sh/release-namespace: d8-system + labels: + app.kubernetes.io/managed-by: Helm +spec: + registry: + ca: "" + dockerCfg: eyJhdXRocyI6eyJkZXYtcmVnaXN0cnkuZGVja2hvdXNlLmlvIjp7ImF1dGgiOiJiR2xqWlc1elpTMTBiMnRsYmpwWU4xbHBaMWhTUWxBM1pscHdOVGxwZW5acFUyWkNWelZFTlZGTmIwaFJlUT09In19fQ== + repo: dev-registry.deckhouse.io/sys/deckhouse-oss/modules + scheme: HTTPS + releaseChannel: "" diff --git a/testkit_v2/data/resources.yml.tpl b/legacy/testkit_v2/data/resources.yml.tpl similarity index 100% rename from testkit_v2/data/resources.yml.tpl rename to legacy/testkit_v2/data/resources.yml.tpl diff --git a/testkit_v2/e2e_test.sh b/legacy/testkit_v2/e2e_test.sh similarity index 100% rename from testkit_v2/e2e_test.sh rename to legacy/testkit_v2/e2e_test.sh diff --git a/testkit_v2/go.mod b/legacy/testkit_v2/go.mod similarity index 100% rename from testkit_v2/go.mod rename to legacy/testkit_v2/go.mod diff --git a/testkit_v2/go.sum b/legacy/testkit_v2/go.sum similarity index 100% rename from testkit_v2/go.sum rename to legacy/testkit_v2/go.sum diff --git a/testkit_v2/runme.sh b/legacy/testkit_v2/runme.sh similarity index 100% rename from testkit_v2/runme.sh rename to legacy/testkit_v2/runme.sh diff --git a/legacy/testkit_v2/tests/00_healthcheck_test.go b/legacy/testkit_v2/tests/00_healthcheck_test.go new file mode 100644 index 0000000..b3d8d36 --- /dev/null +++ b/legacy/testkit_v2/tests/00_healthcheck_test.go @@ -0,0 +1,120 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package integration + +import ( + "testing" + + util "github.com/deckhouse/sds-e2e/util" +) + +// Modules definition for the cluster deployment +var testModules = []util.ModuleConfig{ + { + Name: "snapshot-controller", + Version: 1, + Enabled: true, + Settings: map[string]any{}, + Dependencies: []string{}, + }, + { + Name: "sds-node-configurator", + Version: 1, + Enabled: true, + Settings: map[string]any{ + "enableThinProvisioning": true, + }, + Dependencies: []string{"sds-local-volume"}, + }, + { + Name: "sds-local-volume", + Version: 1, + Enabled: true, + Settings: map[string]any{}, + Dependencies: []string{"snapshot-controller"}, + }, + { + Name: "sds-replicated-volume", + Version: 1, + Enabled: true, + Settings: map[string]any{}, + Dependencies: []string{"sds-node-configurator"}, + }, +} + +// Cluster definition for healthcheck test +var testClusterDefinition = util.ClusterDefinition{ + Masters: []util.ClusterNode{ + { + Hostname: "master-1", + HostType: util.HostTypeVM, + Role: util.ClusterRoleMaster, + OSType: util.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: util.NodeAuth{ + Method: util.AuthMethodSSHKey, + User: "user", + SSHKey: "", // Public key that will be deployed to the node - value or filepath + }, + CPU: 4, + RAM: 8, + DiskSize: 30, + }, + }, + Workers: []util.ClusterNode{ + { + Hostname: "worker-1", + HostType: util.HostTypeVM, + Role: util.ClusterRoleWorker, + OSType: util.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: util.NodeAuth{ + Method: util.AuthMethodSSHKey, + User: "user", + SSHKey: "", // Public key that will be deployed to the node - value or filepath + }, + CPU: 2, + RAM: 6, + DiskSize: 30, + }, + { + Hostname: "worker-2", + HostType: util.HostTypeVM, + Role: util.ClusterRoleWorker, + OSType: util.OSTypeMap["Ubuntu 22.04 6.2.0-39-generic"], + Auth: util.NodeAuth{ + Method: util.AuthMethodSSHKey, + User: "user", + SSHKey: "", // Public key that will be deployed to the node - value or filepath + }, + CPU: 2, + RAM: 6, + DiskSize: 30, + }, + }, +} + +func TestNodeHealthCheck(t *testing.T) { + cluster := util.EnsureCluster("", "") + + nodeMap := cluster.MapLabelNodes(nil) + for label, nodes := range nodeMap { + if len(nodes) == 0 { + t.Errorf("No %s nodes", label) + } else { + util.Infof("%s nodes: %d", label, len(nodes)) + } + } +} diff --git a/testkit_v2/tests/01_sds_nc_test.go b/legacy/testkit_v2/tests/01_sds_nc_test.go similarity index 100% rename from testkit_v2/tests/01_sds_nc_test.go rename to legacy/testkit_v2/tests/01_sds_nc_test.go diff --git a/testkit_v2/tests/03_sds_lv_test.go b/legacy/testkit_v2/tests/03_sds_lv_test.go similarity index 100% rename from testkit_v2/tests/03_sds_lv_test.go rename to legacy/testkit_v2/tests/03_sds_lv_test.go diff --git a/testkit_v2/tests/05_sds_node_configurator_test.go b/legacy/testkit_v2/tests/05_sds_node_configurator_test.go similarity index 100% rename from testkit_v2/tests/05_sds_node_configurator_test.go rename to legacy/testkit_v2/tests/05_sds_node_configurator_test.go diff --git a/testkit_v2/tests/99_finalizer_test.go b/legacy/testkit_v2/tests/99_finalizer_test.go similarity index 100% rename from testkit_v2/tests/99_finalizer_test.go rename to legacy/testkit_v2/tests/99_finalizer_test.go diff --git a/testkit_v2/tests/data-exporter/base_test.go b/legacy/testkit_v2/tests/data-exporter/base_test.go similarity index 100% rename from testkit_v2/tests/data-exporter/base_test.go rename to legacy/testkit_v2/tests/data-exporter/base_test.go diff --git a/testkit_v2/tests/tools.go b/legacy/testkit_v2/tests/tools.go similarity index 100% rename from testkit_v2/tests/tools.go rename to legacy/testkit_v2/tests/tools.go diff --git a/testkit_v2/util/env.go b/legacy/testkit_v2/util/env.go similarity index 100% rename from testkit_v2/util/env.go rename to legacy/testkit_v2/util/env.go diff --git a/testkit_v2/util/filter.go b/legacy/testkit_v2/util/filter.go similarity index 100% rename from testkit_v2/util/filter.go rename to legacy/testkit_v2/util/filter.go diff --git a/testkit_v2/util/kube.go b/legacy/testkit_v2/util/kube.go similarity index 100% rename from testkit_v2/util/kube.go rename to legacy/testkit_v2/util/kube.go diff --git a/testkit_v2/util/kube_cluster.go b/legacy/testkit_v2/util/kube_cluster.go similarity index 100% rename from testkit_v2/util/kube_cluster.go rename to legacy/testkit_v2/util/kube_cluster.go diff --git a/legacy/testkit_v2/util/kube_cluster_definitions.go b/legacy/testkit_v2/util/kube_cluster_definitions.go new file mode 100644 index 0000000..ee3de19 --- /dev/null +++ b/legacy/testkit_v2/util/kube_cluster_definitions.go @@ -0,0 +1,124 @@ +/* +Copyright 2025 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package integration + +import "time" + +// HostType represents the type of host (VM or bare-metal) +type HostType string + +const ( + HostTypeVM HostType = "vm" + HostTypeBareMetal HostType = "bare-metal" +) + +// ClusterRole represents the role of a node in the cluster +type ClusterRole string + +const ( + ClusterRoleMaster ClusterRole = "master" + ClusterRoleWorker ClusterRole = "worker" + ClusterRoleSetup ClusterRole = "setup" // Bootstrap node for DKP installation +) + +// OSType represents the operating system type +type OSType struct { + Name string + ImageURL string + KernelVersion string +} + +var ( + OSTypeMap = map[string]OSType{ + "Ubuntu 22.04 6.2.0-39-generic": { + ImageURL: "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img", + KernelVersion: "6.2.0-39-generic", + }, + "Ubuntu 24.04 6.8.0-53-generic": { + ImageURL: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img", + KernelVersion: "6.8.0-53-generic", + }, + } +) + +// AuthMethod represents the authentication method +type AuthMethod string + +const ( + AuthMethodSSHKey AuthMethod = "ssh-key" + AuthMethodSSHPass AuthMethod = "ssh-password" +) + +// NodeAuth contains authentication information for a node +type NodeAuth struct { + Method AuthMethod `yaml:"method"` + User string `yaml:"user"` + SSHKey string `yaml:"sshKey"` // Public key (value like "ssh-rsa ...", path to .pub file, or empty for default ~/.ssh/id_rsa.pub) + Password string `yaml:"password"` // Password (if using password auth) + // Internal: resolved private key path for SSH authentication (not in YAML) + privateKeyPath string +} + +// ClusterNode defines a single node in the cluster +type ClusterNode struct { + Hostname string `yaml:"hostname"` + IPAddress string `yaml:"ipAddress,omitempty"` // Required for bare-metal, optional for VM + OSType OSType `yaml:"osType,omitempty"` // Required for VM, optional for bare-metal + HostType HostType `yaml:"hostType"` + Role ClusterRole `yaml:"role"` + Auth NodeAuth `yaml:"auth"` + // VM-specific fields (only used when HostType == HostTypeVM) + CPU int `yaml:"cpu"` // Required for VM + RAM int `yaml:"ram"` // Required for VM, in GB + DiskSize int `yaml:"diskSize"` // Required for VM, in GB + // Image field removed - osType is used as image definition + // Bare-metal specific fields + Prepared bool `yaml:"prepared,omitempty"` // Whether the node is already prepared for DKP installation +} + +// ClusterDefinition defines the complete cluster configuration +type ClusterDefinition struct { + Masters []ClusterNode `yaml:"masters"` + Workers []ClusterNode `yaml:"workers"` + Setup *ClusterNode `yaml:"setup,omitempty"` // Bootstrap node (can be nil, will use master for VM clusters or first worker for bare-metal if not set) +} + +// ModuleConfig defines a Deckhouse module configuration +type ModuleConfig struct { + Name string + Version int + Enabled bool + Settings map[string]any + Dependencies []string // Names of modules that must be enabled before this one +} + +// DKPClusterConfig defines the Deckhouse Kubernetes Platform cluster configuration +type DKPClusterConfig struct { + ClusterDefinition ClusterDefinition + KubernetesVersion string + PodSubnetCIDR string + ServiceSubnetCIDR string + ClusterDomain string + LicenseKey string + RegistryRepo string +} + +const ( + HostReadyTimeout = 10 * time.Minute // Timeout for hosts to be ready + DKPDeployTimeout = 30 * time.Minute // Timeout for DKP deployment + ModuleDeployTimeout = 10 * time.Minute // Timeout for module deployment +) diff --git a/testkit_v2/util/kube_deckhouse_modules.go b/legacy/testkit_v2/util/kube_deckhouse_modules.go similarity index 100% rename from testkit_v2/util/kube_deckhouse_modules.go rename to legacy/testkit_v2/util/kube_deckhouse_modules.go diff --git a/testkit_v2/util/kube_deploy.go b/legacy/testkit_v2/util/kube_deploy.go similarity index 100% rename from testkit_v2/util/kube_deploy.go rename to legacy/testkit_v2/util/kube_deploy.go diff --git a/testkit_v2/util/kube_modules.go b/legacy/testkit_v2/util/kube_modules.go similarity index 100% rename from testkit_v2/util/kube_modules.go rename to legacy/testkit_v2/util/kube_modules.go diff --git a/testkit_v2/util/kube_node.go b/legacy/testkit_v2/util/kube_node.go similarity index 100% rename from testkit_v2/util/kube_node.go rename to legacy/testkit_v2/util/kube_node.go diff --git a/testkit_v2/util/kube_secret.go b/legacy/testkit_v2/util/kube_secret.go similarity index 100% rename from testkit_v2/util/kube_secret.go rename to legacy/testkit_v2/util/kube_secret.go diff --git a/testkit_v2/util/kube_storage.go b/legacy/testkit_v2/util/kube_storage.go similarity index 100% rename from testkit_v2/util/kube_storage.go rename to legacy/testkit_v2/util/kube_storage.go diff --git a/testkit_v2/util/kube_tester.go b/legacy/testkit_v2/util/kube_tester.go similarity index 100% rename from testkit_v2/util/kube_tester.go rename to legacy/testkit_v2/util/kube_tester.go diff --git a/testkit_v2/util/kube_vm.go b/legacy/testkit_v2/util/kube_vm.go similarity index 100% rename from testkit_v2/util/kube_vm.go rename to legacy/testkit_v2/util/kube_vm.go diff --git a/testkit_v2/util/kube_vm_cluster.go b/legacy/testkit_v2/util/kube_vm_cluster.go similarity index 100% rename from testkit_v2/util/kube_vm_cluster.go rename to legacy/testkit_v2/util/kube_vm_cluster.go diff --git a/testkit_v2/util/log.go b/legacy/testkit_v2/util/log.go similarity index 100% rename from testkit_v2/util/log.go rename to legacy/testkit_v2/util/log.go diff --git a/testkit_v2/util/ssh.go b/legacy/testkit_v2/util/ssh.go similarity index 100% rename from testkit_v2/util/ssh.go rename to legacy/testkit_v2/util/ssh.go diff --git a/testkit_v2/util/tools.go b/legacy/testkit_v2/util/tools.go similarity index 100% rename from testkit_v2/util/tools.go rename to legacy/testkit_v2/util/tools.go