Skip to content

operations.md

Latest commit

 

History

History
263 lines (199 loc) · 10.2 KB

operations.md

File metadata and controls

263 lines (199 loc) · 10.2 KB
Raw

Operations

Windows Host Access

Use WinRM over HTTPS for ADLER-WHITE-1W. Do not use raw SSH for normal Windows host administration.

$cred = Import-Clixml -LiteralPath 'C:\Users\KRT\.codex\secrets\adler-winrm.credential.xml'
Invoke-Command -ComputerName ADLER-WHITE-1W -UseSSL -ConfigurationName PowerShell.7 -Credential $cred -Authentication Negotiate -ScriptBlock {
    hostname
    whoami
    $PSVersionTable.PSVersion.ToString()
}

Expected host identity is ADLER-WHITE-W1\KRT, PowerShell 7.6.5 Core, FullLanguage, and an administrator token. SSH is retained only for bootstrap/recovery of WinRM itself.

Deploy

Запуск HTTPS после изменения сети VM

nginx слушает все адреса гостя и не должен ждать конкретный IPv4 на eth0. Роль управляет /etc/systemd/system/nginx.service.d/20-wait-lan.conf: порядок запуска через network-online.target и Docker, повтор при ошибке, без цикла ожидания старого адреса. Такой цикл блокировал HTTPS после переноса VM в отдельную подсеть, хотя контейнер Frigate уже работал.

После применения проверьте nginx -t, systemctl is-active nginx, отсутствие адресного ExecStartPre в systemctl cat nginx и HTTPS /review через имя сервиса. Ответ 401 без реквизитов подтверждает доступность TLS и защиты; проверка входа и камер выполняется штатным smoke-тестом.

  1. Copy ansible/inventory.example.yml to ansible/inventory.yml.
  2. Copy ansible/group_vars/all.example.yml to ansible/group_vars/all.yml.
  3. Put real camera credentials in ansible/group_vars/all.yml, or encrypt them:
ansible-vault encrypt ansible/group_vars/all.yml

Before the first Ansible connection, obtain the VM SSH host key with ssh-keyscan, compare its fingerprint with ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the VM console, and only then append the verified public key to the operator's known_hosts. The sample inventory uses StrictHostKeyChecking=yes and will not learn a key silently.

  1. Run:
ansible-playbook -i ansible/inventory.yml ansible/playbooks/site.yml --ask-become-pass

Hyper-V Host

Current production note (2026-09-14): ADLER-WHITE-W1 no longer contains the Tesla P40. Do not run this host setup with -AssignGpu against White unless a new compatible device and its exact current location path have first been verified. frigate-ubuntu is intentionally off with autostart disabled; the Pi kiosk streams directly from the cameras through Red.

Run against the Windows Server host through WinRM HTTPS:

$cred = Import-Clixml -LiteralPath 'C:\Users\KRT\.codex\secrets\adler-winrm.credential.xml'
Invoke-Command -ComputerName ADLER-WHITE-1W -UseSSL -ConfigurationName PowerShell.7 -Credential $cred -Authentication Negotiate -FilePath .\scripts\hyperv-host-setup.ps1

Use -AssignGpu only when intentionally assigning the Tesla P40 DDA device to the VM. That operation changes host PCI device state.

Trust Local Certificate

Copy the public certificate over an authenticated channel and run on a Windows client:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\install-frigate-local-ca.ps1 `
  -FrigateUrl https://frigate.adler-white-w1.lan `
  -CaCertPath C:\secure-transfer\fullchain.pem

If the Frigate service uses the local root CA from the server, copy the public CA and CRL to the Windows client first, then install both explicitly:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\install-frigate-local-ca.ps1 -CaCertPath $env:TEMP\KRT-Frigate-Local-Root-CA-2026.pem -CrlPath $env:TEMP\KRT-Frigate-Local-Root-CA-2026.crl

Alternatively, pass an out-of-band SHA-256 with -ExpectedSha256Thumbprint. Bare remote capture is rejected; -TrustOnFirstUse is an explicit bootstrap-only escape hatch.

Config Backups

Backups are configuration-only and are retained only on the home server file storage. The policy is documented in backup-policy.md, and the persistent registry is registries/backup-registry.csv.

The production scheduled task on ADLER-WHITE-1W is:

WinHome Config Backup

It runs daily at 03:20 as SYSTEM and stores verified archives under:

F:\Files\Backups\win-home-configs

Run the same backup manually through WinRM:

$cred = Import-Clixml -LiteralPath 'C:\Users\KRT\.codex\secrets\adler-winrm.credential.xml'
Invoke-Command -ComputerName ADLER-WHITE-1W -UseSSL -ConfigurationName PowerShell.7 -Credential $cred -Authentication Negotiate -FilePath .\scripts\invoke-config-backup.ps1

Camera Presence Watchdog

A camera that is powered off or unplugged keeps Frigate in a restart loop: ffmpeg retries every few seconds and fills the log with Error opening input file and method DESCRIBE failed: 404 (Not Found). krt-camera-watchdog.timer runs /usr/local/sbin/krt-camera-watchdog.sh once a minute and keeps the camera list in step with reality, so switching a camera off and on needs no manual step:

  • it opens TCP 554 on each camera, with a 3-second timeout;
  • after frigate_vm_camera_watchdog_offline_threshold consecutive failures (default 10, so ten minutes) it disables the camera;
  • after frigate_vm_camera_watchdog_online_threshold consecutive successes (default 2) it enables it again;
  • it toggles cameras.<name>.enabled through PUT /api/config/set with requires_restart: 0, so the change applies live and is written to config.yml; recording on the other cameras is never interrupted;
  • it does nothing at all unless the frigate container is running|healthy and the API answers, and it never disables the last enabled camera.

State lives in /run/krt-camera-watchdog, so the counters restart from zero after a reboot. Follow its decisions with journalctl -u krt-camera-watchdog.service.

enabled: in ansible/group_vars/all.yml is therefore only the state a deploy starts from - the watchdog corrects it within minutes either way. Set watchdog: false on a camera to keep it out of the watchdog's hands, and camera_watchdog_enabled: false to turn the watchdog off entirely.

cameras:
  - name: shed_hikvision
    host: 192.168.50.33
    detect_width: 640
    detect_height: 360
    detect_fps: 5
    enabled: false
    watchdog: true

Smoke Test

For the current recorder-only production profile, run after deploy and after host reboots:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke-test-recorder.ps1

It checks the 2-vCPU/4-GB VM, autostart, zero DDA devices, at least 150 GB free on host drive F:, Frigate health, copy-mode recording, the expected number of configured cameras, VM CPU headroom, three-day retention, disabled analytics/Ollama/ASR and no GPU runtime. An offline camera is reported in camera_fps but does not fail the other camera recordings. -ExpectedCameraCount counts the cameras in the config, including the ones the watchdog has disabled, so it does not move when a camera goes away.

For the retained gpu_analytics profile, run:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke-test.ps1

All LAN APIs are protected by nginx basic auth. Provide the shared credentials with the -FrigateAuth*, -OllamaAuth*, and -AsrAuth* parameters or the matching FRIGATE_BASIC_*, OLLAMA_BASIC_*, and ASR_BASIC_* environment variables.

The test writes scripts\logs\frigate-vm-smoke-latest.json unless a custom -ReportPath is supplied. It verifies Hyper-V autostart, DDA GPU assignment, trusted TLS, Frigate health, ONNX GPU detector, CUDA ffmpeg, camera FPS, recordings, Ollama service/API, Frigate-to-Ollama network path and Ollama text GPU execution. It also checks the separate ASR HTTPS API and container.

To include a real ASR transcription check, pass a local audio sample:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke-test.ps1 -AsrSamplePath "C:\path\audio.m4a" -SkipOllamaGenerate

LAN API Usage

Install and verify the local certificate with install-frigate-local-ca.ps1 before using these endpoints. The examples intentionally do not disable TLS verification.

Frigate is published on the VM through its LAN name:

curl.exe -u "$env:FRIGATE_BASIC_USER`:$env:FRIGATE_BASIC_PASSWORD" `
  https://frigate.adler-white-w1.lan/api/version

Ollama is published on the VM LAN address:

curl.exe -u "$env:OLLAMA_BASIC_USER`:$env:OLLAMA_BASIC_PASSWORD" https://192.168.1.138:11443/api/version

ASR is published on the VM LAN address over HTTPS:

curl.exe -u "$env:ASR_BASIC_USER`:$env:ASR_BASIC_PASSWORD" https://192.168.1.138:9443/health

Transcribe audio with the OpenAI-compatible endpoint:

curl.exe -u "$env:ASR_BASIC_USER`:$env:ASR_BASIC_PASSWORD" -X POST "https://192.168.1.138:9443/v1/audio/transcriptions" `
  -F "file=@C:\path\audio.m4a" `
  -F "language=ru" `
  -F "response_format=json"

Run the installed gpt-oss model from this workstation:

$body = @{
  model = "huihui_ai/gpt-oss-abliterated:20b"
  prompt = "Напиши одно короткое предложение по-русски."
  stream = $false
  think = "low"
  options = @{ num_predict = 256; num_ctx = 2048 }
} | ConvertTo-Json -Depth 4
curl.exe -u "$env:OLLAMA_BASIC_USER`:$env:OLLAMA_BASIC_PASSWORD" `
  --max-time 600 -H "Content-Type: application/json" `
  --data-binary $body https://192.168.1.138:11443/api/generate

For this gpt-oss model, keep num_predict at 256 or higher for normal visible answers; lower limits may be spent on thinking tokens. Cold start on the Tesla P40 takes about 5 minutes, so use a 600 second timeout for the first request.

Rollback

The recorder transition script preserves camera definitions and existing credentials while removing GPU/analytics services:

sudo python3 /tmp/apply-recorder-profile.py --check
sudo python3 /tmp/apply-recorder-profile.py
sudo docker compose -f /opt/frigate/docker-compose.yml up -d --force-recreate