Codex/aip 4614 pr153 review fixes

.github/workflows/cluster-setup-tests.yml

@@ -20,5 +20,8 @@ jobs:
       - name: Run cluster_setup unit tests
         run: bash tools/cluster_setup/test_k0s_cluster_with_stack.sh
 
+      - name: Run prerequisite installer unit tests
+        run: bash tools/cluster_setup/test_installer_prerequisites.sh
+
       - name: Run Splunk-optional installer dry-run tests
         run: bash tools/cluster_setup/test_installer_dry_run.sh

docs/deployment/k0s-quick-reference.md

@@ -7,6 +7,18 @@ explanations, diagrams, and edge cases see
 
 All commands run from `tools/cluster_setup/` unless noted otherwise.
 
+Clone or download the **complete repository** to the admin workstation first;
+do not copy only `k0s_cluster_with_stack.sh`. The installer also needs its
+adjacent library, version lock, configuration templates, and helper scripts.
+
+```bash
+git clone https://github.com/splunk/splunk-ai-operator.git
+cd splunk-ai-operator/tools/cluster_setup
+```
+
+If Git is unavailable, use GitHub's **Code → Download ZIP** action, extract the
+archive locally, and change into its `tools/cluster_setup` directory.
+
 ## Table of Contents
 
 1. [Prerequisites](#1-prerequisites)

tools/cluster_setup/DEPLOYMENT_GUIDE.md

@@ -502,7 +502,7 @@ graph TD
 
     subgraph BIN["📁 binaries/"]
         K0S["k0s binary\nlatest or --k0s-version"]
-        YQ["yq v4.44.1\nYAML processor"]
+        YQ["yq v4.53.3\nYAML processor"]
     end
 
     subgraph IMG["📁 images/  ⭐ pre-loaded OCI image bundles"]

tools/cluster_setup/K0S_README.md

@@ -70,7 +70,7 @@ The script installs everything needed for the AI Platform:
 
 - **Two-phase parallel installation** — Independent components install concurrently for faster deployments
 - **Helm retry with exponential backoff** — Automatic retries on transient errors (timeouts, TLS handshake failures)
-- **Preflight validation** — Checks tools, config, SSH connectivity, and disk space before starting
+- **Prerequisite bootstrap and preflight validation** — Installs missing admin-workstation tools on supported hosts, then checks config, SSH connectivity, and disk space before starting
 - **Safety gate** — Refuses to wipe a cluster that has Ready nodes (prevents accidental data loss)
 - **Session logging** — All stdout/stderr captured to `tools/cluster_setup/logs/k0s-install-YYYY-MM-DD_HH-MM-SS.log`
 - **Existing cluster detection** — `useExisting` flag (auto/force/never) to skip k0s install and deploy stack only
@@ -95,24 +95,61 @@ AIPlatform CR → AIService → Job/RayCluster → Pods
 
 ### Required Tools (on Admin Workstation)
 
+The `install` command manages its local tool prerequisites automatically. It
+checks the admin workstation, installs any missing tools, checks again, and
+then continues with the existing configuration and cluster preflight flow.
+Tools that are already available are left unchanged.
+
+The managed tools are `curl`, `ssh`/`scp`, `git`, `jq`, `yq`, `kubectl`, Helm,
+`tar`, and the GNU `timeout` utility used by the installer. On macOS, Homebrew
+`coreutils` supplies `gtimeout`; the prerequisite module exposes it as
+`timeout` from the user-local tool directory.
+Model-staging and storage-provider helpers have additional conditional tools
+such as Python, `wget`, Git LFS, the AWS CLI, or `mc`; those helpers validate
+their own requirements only when the related workflow is enabled.
+Downloaded `kubectl`, Helm, `jq`, and `yq` assets are pinned and checksummed in
+`prerequisites.lock`. A custom `PREREQ_KUBECTL_VERSION` must be paired with the
+matching `PREREQ_KUBECTL_SHA256`.
+Automatic installation is supported on these **installer hosts** (this is
+separate from the operating-system requirements for cluster nodes):
+
+| Installer host | Installation method |
+|---|---|
+| Ubuntu / Debian | `apt` |
+| RHEL / Rocky Linux / AlmaLinux / Fedora / Amazon Linux | `dnf` or `yum` |
+| macOS | An existing Homebrew installation; the installer does not install Homebrew |
+
+The script itself requires **Bash 4.4 or newer**. Bash cannot be bootstrapped by
+the prerequisite installer because the script must start before that installer
+can run. In particular, macOS ships an older `/bin/bash`; install a current
+Bash manually and use it to launch the script:
+
 ```bash
-# Install required tools on macOS
-brew install kubectl helm git jq yq
+# macOS only: one-time bootstrap
+brew install bash
+"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh prereqs check
+```
+
+Because this repository keeps the portable `/bin/bash` shebang used by existing
+Linux automation, use the Homebrew Bash prefix shown above for every installer
+invocation on macOS.
 
-# Install required tools on Ubuntu/Debian
-sudo apt-get update
-sudo apt-get install -y kubectl helm git jq
-wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq
-chmod +x /usr/local/bin/yq
+You can inspect or prepare a workstation before an install:
 
-# Verify installations
-kubectl version --client
-helm version
-git --version
-jq --version
-yq --version
+```bash
+# Report missing prerequisites without changing the host
+./k0s_cluster_with_stack.sh prereqs check
+
+# Install missing prerequisites and verify them
+./k0s_cluster_with_stack.sh prereqs install
 ```
 
+Interactive runs may prompt for local `sudo` when the package manager needs
+elevated privileges. Silent/non-interactive runs use non-interactive sudo and
+fail with remediation instructions if passwordless sudo is unavailable; they
+never wait for a password prompt. If host packages are managed separately, use
+`install --no-install-prereqs` to retain the fail-on-missing-tools behavior.
+
 ### Hardware Requirements
 
 | Node Type | Min CPU | Min RAM | Min Disk | Notes |
@@ -158,11 +195,22 @@ You must provide an external S3-compatible object storage endpoint:
 
 ### 1. Clone the Repository
 
+Download the **complete repository** to the admin workstation before running
+the installer. Do not copy only `k0s_cluster_with_stack.sh`: it uses the
+adjacent prerequisite library, version lock, configuration templates, and
+helper scripts from this repository.
+
+If Git is already available:
+
 ```bash
 git clone https://github.com/splunk/splunk-ai-operator.git
 cd splunk-ai-operator/tools/cluster_setup
 ```
 
+Otherwise, use GitHub's **Code → Download ZIP** action, extract the archive
+locally, and change into its `tools/cluster_setup` directory. Run the remaining
+quick-start commands from that directory.
+
 ### 2. Create Configuration File
 
 ```bash
@@ -179,6 +227,13 @@ vi my-cluster.yaml
 CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install
 ```
 
+This checks, installs, and rechecks local prerequisites before continuing. To
+disable host changes on a centrally managed workstation:
+
+```bash
+CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs
+```
+
 ### 4. Verify Installation
 
 ```bash
@@ -566,6 +621,15 @@ imagePullSecrets:
 # Install cluster and full AI Platform stack
 CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh install
 
+# Check prerequisites only (does not install anything)
+./k0s_cluster_with_stack.sh prereqs check
+
+# Install any missing prerequisites, then verify them
+./k0s_cluster_with_stack.sh prereqs install
+
+# Install without changing workstation prerequisites
+CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs
+
 # Stage model artifacts only (download from HF + upload to object store)
 CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh stage-artifacts
 
@@ -584,6 +648,13 @@ CONFIG_FILE=./my-config.yaml ./k0s_cluster_with_stack.sh join-workers
 
 > **Air-gap uses these same commands.** With `cluster.airgap: true` in the config, `install` and `join-workers` stage the offline artifacts first and then continue; every other subcommand runs immediately, unchanged. See [Air-Gapped Deployment](#air-gapped-deployment).
 
+For a normal `install`, prerequisite handling is `check` → install only what is
+missing → `check` again. A failure to install or verify a tool stops the run
+before cluster changes begin. `validate` and other read-only commands do not
+install host packages. `--silent`, `-s`, and `AUTO_APPROVE=true` keep the
+prerequisite step non-interactive as well: any required `sudo` is attempted
+without a password prompt and fails promptly if authorization is unavailable.
+
 ### Environment Variables
 
 #### General
@@ -1309,7 +1380,7 @@ cd tools/cluster_setup
 
 | Category | Contents |
 |---|---|
-| Binaries | `k0s` (latest stable or `--k0s-version`), `yq v4.44.1` |
+| Binaries | `k0s` (latest stable or `--k0s-version`), `yq v4.53.3` |
 | **Image bundles** (`images/`) | **`k0s-images.tar`** — k0s control-plane images (pause, Calico, kube-proxy, CoreDNS, metrics-server); **`addon-images.tar`** — add-on component images (cert-manager, kube-prometheus-stack, kuberay, MetalLB, OTel, NVIDIA device plugin, busybox). Both built automatically and staged to `/var/lib/k0s/images/` on every node at install time. |
 | Manifests | `cert-manager v1.13.0`, `local-path-provisioner v0.0.24`, `nvidia-device-plugin v0.17.3` |
 | Helm charts | `kube-prometheus-stack` (version captured at download time), `opentelemetry-operator` (version captured at download time), `kuberay-operator 1.2.2`, `metallb 0.14.8` |

tools/cluster_setup/TROUBLESHOOTING.md

@@ -64,16 +64,82 @@ that has Ready nodes.
 
 ### "Required tool not found: \<tool\>"
 
-The install machine is missing a required binary.
+The install machine is missing a required binary. A normal `install` first
+checks prerequisites, attempts to install anything missing, and checks again.
+This message therefore means either automatic installation was disabled with
+`--no-install-prereqs` or the installation/recheck did not succeed.
+
+Inspect the workstation without making changes, or retry only the prerequisite
+step:
+
+```bash
+./k0s_cluster_with_stack.sh prereqs check
+./k0s_cluster_with_stack.sh prereqs install
+```
 
 | Tool | Install |
 |---|---|
 | `kubectl` | `brew install kubectl` / [kubernetes.io/docs/tasks/tools](https://kubernetes.io/docs/tasks/tools/) |
 | `helm` | `brew install helm` / [helm.sh/docs/intro/install](https://helm.sh/docs/intro/install/) |
-| `yq` | `brew install yq` / `wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq && chmod +x /usr/local/bin/yq` |
+| `yq` | Prefer `./k0s_cluster_with_stack.sh prereqs install` (pinned and checksummed) / `brew install yq` on a manually managed Mac |
 | `jq` | `brew install jq` / `apt-get install jq` / `dnf install jq` |
 | `ssh` | `apt-get install openssh-client` / `brew install openssh` |
 | `curl` | `apt-get install curl` / `brew install curl` |
+| `tar` | Normally provided by the base OS; `apt-get install tar` / `dnf install tar` |
+| `timeout` | `apt-get install coreutils` / `dnf install coreutils` / `brew install coreutils` |
+
+---
+
+### "Unsupported installer host" or no supported package manager
+
+Automatic prerequisite installation is supported on Ubuntu/Debian (`apt`),
+RHEL/Rocky Linux/AlmaLinux/Fedora/Amazon Linux (`dnf` or `yum`), and macOS with
+Homebrew already installed. This support matrix applies to the admin
+workstation, not the remote cluster nodes.
+
+On another host, install the tools listed by `prereqs check` manually and run:
+
+```bash
+./k0s_cluster_with_stack.sh prereqs check
+CONFIG_FILE=./my-cluster.yaml ./k0s_cluster_with_stack.sh install --no-install-prereqs
+```
+
+On macOS, install Homebrew yourself before using `prereqs install`; the script
+does not bootstrap a package manager.
+
+---
+
+### Bash is older than version 4.4
+
+The installer requires Bash 4.4 or newer before prerequisite management can
+start, so it cannot upgrade Bash for itself. macOS's system `/bin/bash` is too
+old. Install a current Bash manually, then invoke the installer with it:
+
+```bash
+brew install bash
+"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh prereqs check
+"$(brew --prefix)/bin/bash" ./k0s_cluster_with_stack.sh install
+```
+
+---
+
+### Prerequisite installation cannot obtain sudo
+
+Linux package installation may require local `sudo`. An interactive install
+may display the normal sudo password prompt. A silent/non-interactive install
+(`--silent`, `-s`, or `AUTO_APPROVE=true`) uses non-interactive sudo and never
+waits for a password; it fails promptly when passwordless authorization is not
+available.
+
+Choose one of these fixes:
+
+1. Run `prereqs install` interactively from a terminal and authorize sudo.
+2. Have an administrator install the tools reported by `prereqs check`.
+3. Configure appropriately scoped passwordless sudo for automation, then retry.
+
+After a manual fix, run `prereqs check` again. Use
+`install --no-install-prereqs` only when you intentionally want missing tools
+to fail preflight without an installation attempt.
 
 ---
 

tools/cluster_setup/airgap_install.sh

@@ -17,7 +17,7 @@
 set -euo pipefail
 
 # ── Versions (keep in sync with k0s_cluster_with_stack.sh) ─────────────────
-YQ_VERSION="v4.44.1"
+YQ_VERSION="v4.53.3"
 CERT_MANAGER_VERSION="v1.13.0"
 LOCAL_PATH_PROVISIONER_VERSION="v0.0.24"
 NVIDIA_DEVICE_PLUGIN_VERSION="v0.17.3"

tools/cluster_setup/k0s-cluster-config.yaml

@@ -9,14 +9,23 @@
 #   3. Replace all values marked with "CHANGE THIS"
 #   4. Run: CONFIG_FILE=./my-k0s-config.yaml ./k0s_cluster_with_stack.sh install
 #
-# Prerequisites (must be satisfied BEFORE running the installer):
+# Prerequisites:
 #
 # Local machine:
-#   ssh, kubectl, helm, git, jq, yq, docker — see the Prerequisites section of
-#   docs/deployment/k0s-quick-reference.md for install instructions per OS.
-#   macOS: brew install kubectl helm git jq yq
-#   RHEL 9: none of kubectl/helm/docker/yq are in the default dnf repos —
-#     install each via its own official method (git/jq ARE available via dnf).
+#   - Bash 4.4+ must be installed before this script can run. macOS's /bin/bash
+#     is too old; install a current Bash manually (for example, brew install bash).
+#   - During 'install', missing ssh/scp/curl/git/jq/yq/kubectl/helm/tar/timeout
+#     prerequisites are installed on supported installer hosts, then verified again.
+#   - Supported installer hosts: Ubuntu/Debian (apt), RHEL/Rocky/Alma/Fedora/
+#     Amazon Linux (dnf/yum), and macOS with Homebrew already installed.
+#   - Check or install them separately with:
+#       ./k0s_cluster_with_stack.sh prereqs check
+#       ./k0s_cluster_with_stack.sh prereqs install
+#   - Use 'install --no-install-prereqs' when the workstation is managed manually.
+#   - Interactive package installs may prompt for local sudo. Silent installs use
+#     non-interactive sudo and fail instead of waiting for a password prompt.
+#   - Docker is needed only for workflows that manually mirror container images;
+#     see docs/deployment/k0s-quick-reference.md for OS-specific instructions.
 #
 # All remote nodes (controllers + workers):
 #   - SSH access using the key at cluster.sshKeyPath as cluster.sshUser

tools/cluster_setup/k0s_aws_provision.sh

@@ -833,33 +833,32 @@ AWSCLI
     log "AWS CLI ready on installer"
   fi
 
-  log "Installing prerequisites on installer (yq, kubectl, helm, jq)..."
-  ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" 'bash -s' <<'PREREQ'
-set -e
-export PATH="$PATH:/usr/local/bin"
-sudo dnf install -y git jq curl unzip 2>/dev/null || sudo yum install -y git jq curl unzip
-command -v yq &>/dev/null || {
-  sudo curl -sSL -o /usr/local/bin/yq \
-    "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64"
-  sudo chmod +x /usr/local/bin/yq
-}
-command -v kubectl &>/dev/null || {
-  K8S_VER=$(curl -sSL https://dl.k8s.io/release/stable.txt 2>/dev/null || echo v1.32.0)
-  sudo curl -sSL -o /usr/local/bin/kubectl \
-    "https://dl.k8s.io/release/${K8S_VER}/bin/linux/amd64/kubectl"
-  sudo chmod +x /usr/local/bin/kubectl
-}
-command -v helm &>/dev/null || \
-  curl -sSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
-echo "Prerequisites ready."
-PREREQ
-
   log "Copying k0s cluster scripts to installer..."
   ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" \
-    'mkdir -p ~/cluster_setup'
+    'mkdir -p ~/cluster_setup/lib'
   scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \
     "${SCRIPT_DIR}/"*.sh "${SCRIPT_DIR}/"*.yaml \
     "ec2-user@${eip}:~/cluster_setup/" 2>/dev/null || true
+  scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \
+    "${SCRIPT_DIR}/prerequisites.lock" \
+    "ec2-user@${eip}:~/cluster_setup/" \
+    || err "Failed to copy prerequisites.lock to installer"
+  scp -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no \
+    "${SCRIPT_DIR}/lib/installer_prereqs.sh" \
+    "ec2-user@${eip}:~/cluster_setup/lib/" \
+    || err "Failed to copy installer_prereqs.sh to installer"
+
+  log "Installing verified prerequisites on installer..."
+  ssh -i "${KEY_LOCAL}" -o StrictHostKeyChecking=no "ec2-user@${eip}" \
+    'sudo bash -s' <<'PREREQ'
+set -euo pipefail
+export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
+export PREREQ_INSTALL_DIR="/usr/local/bin"
+source /home/ec2-user/cluster_setup/lib/installer_prereqs.sh
+prereq_ensure_profile cluster noninteractive
+echo "Verified prerequisites ready."
+PREREQ
+
   # Copy artifacts_download_upload_scripts (sibling of cluster_setup) — required by model staging step
   local artifacts_dir="${SCRIPT_DIR}/../artifacts_download_upload_scripts"
   if [[ -d "${artifacts_dir}" ]]; then