[JBRes-9717] Allow users to specify more specs for pods

[vpoliakov-pixel]

Why

Today StartServerRequest only lets callers customize a narrow slice of the server pod: resources (cpu/mem/ephemeral-storage), node_selector, runtime_class_name, and run_as_root. Pipelines that run agents routinely need to mount secrets and volumes (credentials, configmaps, shared data) and occasionally other pod-level knobs.

What

Adds a hybrid customization surface to StartServerRequest, threaded end to end (client → orchestrator → deploy_server → V1PodSpec):

• Typed, validated fields for the common cases: volumes, volume_mounts, env_from, service_account_name.
• A generic pod_overrides escape hatch (partial V1PodSpec, deep-merged) for everything else — tolerations, hostAliases, dnsConfig, pod-level securityContext, sidecars, …

All k8s-shaped inputs use native camelCase manifest shape (secretName, mountPath, readOnly) so users can paste straight from k8s docs.

How it works

• deserialize_k8s (new helper in kubernetes_client.py) converts camelCase dicts to V1* models via the kubernetes client's own deserializer, reusing the cached ApiClient (no extra aiohttp session).
• pod_overrides is layered last via a sanitize_for_serialization → deep_merge(concat_lists=True) → deserialize roundtrip: scalars override, list fields concatenate, so the managed server container survives and node-pool tolerations / regcred pull-secret are preserved. Can add sidecars; can't replace server.
• deep_merge added to common-utils/.../dict.py as a reusable utility.
• ServiceAccount precedence: normal start/restore → caller SA wins; snapshot preparation → snapshot SA only (it carries snapshot permissions). New runtime-only fields are intentionally excluded from the snapshot-reuse hash.

Example

await client.start_server(
    image_tag="my-env:latest",
    volumes=[{"name": "agent-creds", "secret": {"secretName": "agent-creds"}}],
    volume_mounts=[{"name": "agent-creds", "mountPath": "/etc/creds", "readOnly": True}],
    env_from=[{"secretRef": {"name": "agent-creds"}}],
    service_account_name="agent-runner",
    pod_overrides={"tolerations": [{"key": "dedicated", "operator": "Exists", "effect": "NoSchedule"}]},
)

Tests

• Unit (uv run pytest -m unit — all green, 521 passed):
  • test_dict.py — deep_merge (recurse, list concat vs replace, scalar override, no input mutation).
  • test_kubernetes_client.py (new) — deploy_server builds the pod spec with volumes/volume_mounts/env_from, and pod_overrides merges pod-level fields / concatenates lists / keeps the managed container / can add a sidecar; plus StartServerRequest field defaults & camelCase acceptance.
• E2E (uv run pytest -m e2e, needs a cluster — added, not run in this PR): secret volume mount + env_from, service_account_name, and pod_overrides hostAliases → /etc/hosts.

Notes

• Validation of the k8s shapes is delegated to the API server (the client deserializer silently drops unknown keys); typed fields are the safe, discoverable path and pod_overrides is the escape hatch.
• New fields are additive optionals, so stored-request (de)serialization and snapshot reuse-matching are unaffected.

[Copilot]

Pull request overview

This PR expands StartServerRequest’s Kubernetes pod customization surface so callers can mount volumes/secrets, import env via envFrom, set a ServiceAccount, and apply a deep-merged pod_overrides escape hatch that’s threaded from client → orchestrator → deploy_server → V1PodSpec.

Changes:

• Add typed “native Kubernetes shape” models (idegym.api.pod_spec) and new StartServerRequest fields: volumes, volume_mounts, env_from, service_account_name, pod_overrides.
• Implement Kubernetes camelCase deserialization + deep-merge override application in backend-utils deploy_server.
• Add unit + e2e tests and supporting e2e Kubernetes helpers (secret + ServiceAccount lifecycle).

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
unit-tests/test_pod_spec_models.py	Unit tests for new typed pod-spec models and camelCase round-tripping.
unit-tests/test_kubernetes_client.py	Unit tests verifying deploy_server applies typed pod fields and merges pod_overrides.
unit-tests/test_dict.py	Adds unit tests for the new deep_merge helper.
orchestrator/src/idegym/orchestrator/snapshot_pipeline.py	Threads new pod customization fields through snapshot pipeline deploy.
orchestrator/src/idegym/orchestrator/router/server.py	Threads new pod customization fields through normal server start; adjusts SA precedence logic.
e2e-tests/utils/k8s_client.py	Adds e2e helpers to create/delete Secrets and ServiceAccounts.
e2e-tests/test_pod_spec_overrides.py	Adds e2e coverage for secret volume mounts, envFrom, ServiceAccount, and pod_overrides.
common-utils/src/idegym/utils/dict.py	Introduces reusable deep_merge utility (with optional list concatenation).
client/src/idegym/client/operations/servers.py	Exposes new pod customization parameters through client operations.
client/src/idegym/client/client.py	Exposes new pod customization parameters on the public client APIs.
backend-utils/src/idegym/backend/utils/kubernetes_client.py	Adds deserialize_k8s, applies volumes/mounts/envFrom, and deep-merges pod_overrides into V1PodSpec.
api/src/idegym/api/pod_spec.py	Adds new typed “native Kubernetes shape” models with camelCase aliases and extra passthrough.
api/src/idegym/api/orchestrator/servers.py	Extends StartServerRequest with new pod customization fields.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.