Skip to content

Add nodeSelector, tolerations and imagePullSecrets for the EPP under spec.advanced.inferencePool #176

Description

@tylerpotts

Problem

InferencePoolAdvancedSpec (operator/api/v1alpha1/llmmodel_types.go) exposes only schedulerConfig. nodeSelector, affinity and tolerations exist only under spec.advanced.vllm and apply to the model pod. There is no way to place the EPP Deployment or give it imagePullSecrets.

Customer report: a two-node cluster (one GPU node, one CPU node) where the EPP image became unpullable (https://github.com/Concepts-NREC/Nebari-GitOps, see the llm-serving-pack default-image issue). One EPP replica ran on the GPU node with the image already present; the other sat in ImagePullBackOff on the CPU node. They could not pin the EPP to the node that had the image, and hand-patching the EPP Deployment was reverted by the operator within ~90s.

Proposal

Add to InferencePoolAdvancedSpec:

NodeSelector     map[string]string             `json:"nodeSelector,omitempty"`
Tolerations      []corev1.Toleration           `json:"tolerations,omitempty"`
Affinity         *corev1.Affinity              `json:"affinity,omitempty"`
ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
Replicas         *int32                        `json:"replicas,omitempty"`

and thread them through buildEPPDeployment in reconcilers/inferencepool.go. Replicas is included because a single-node or two-node cluster has no reason to run two EPPs; today the count is hardcoded.

Also consider an operator-level default (defaults.epp.imagePullSecrets, defaults.epp.nodeSelector) alongside defaults.epp.image, since a private mirror is a cluster-wide decision rather than a per-model one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    Priority

    None yet

    Start date

    None yet

    Target date

    None yet

    Size

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions