Pressure-based autoscaling with Infrastructure Providers, what your recommended patterns?

[Cliftonz]

I'm building [omni-infra-provider-truenas](https://github.com/bearbinary/omni-infra-provider-truenas) which is an infrastructure provider that provisions Talos Linux VMs on TrueNAS SCALE via its JSON-RPC 2.0 API.

Because TrueNAS virtualizes hardware on demand (zvols + VMs), it behaves more like a cloud provider than a static bare metal pool.

So We can actually create new machines dynamically in response to cluster pressure.

This puts us in an interesting position: MachineClasses + autoprovision already handles the fulfillment side beautifully, but there's a gap on the trigger side for pressure-based autoscaling (i.e. unschedulable pods → new node).

The Problem

Omni's model is intentionally declarative. You set MachineSet replica counts and Omni fulfills them.

That's great for GitOps, but doesn't close the loop on pod-scheduling pressure automatically. Classic Cluster Autoscaler solves this for CAPI via MachineDeployment annotations, but that integration doesn't exist for Omni's infrastructure provider model.

Since our provider runs outside the cluster (alongside Omni, not inside K8s), we can't simply deploy an in-cluster controller to watch for FailedScheduling events and bump replica counts. The options I have looked at identified:

Option A: Embed a K8s watcher goroutine in the provider binary

The provider uses the Omni-vended kubeconfig to watch v1.Event for FailedScheduling, then calls the Omni API to scale the MachineSet replica count. No in-cluster dependency, single binary.
However, this feels a bit like bolting two concerns onto the provider, and this gave me pause.

Option B: Karpenter custom cloud provider

Implement Karpenter's CloudProvider interface where Create() calls the Omni API → MachineRequest → provider → TrueNAS. Karpenter runs inside the cluster, handles bin-packing and scheduling pressure natively. Build cost is high and introduces an in-cluster dependency.

Option C: Something in Omni itself???

Does Omni have, or are there plans for, a native scheduling-pressure signal that could trigger MachineSet scaling via the infrastructure provider API? Something like a ScalingPolicy resource that watches for unschedulable pods and adjusts replica counts automatically?

Questions for the Team

1. Is there a recommended pattern for pressure-based autoscaling with Omni infrastructure providers today?
2. Is native autoscaling (Omni watching for unschedulable pods and driving MachineSet replicas) on the roadmap?
3. For virtualized providers (like ours, or the Proxmox/cloud providers) where machines can be created on-demand, does the Omni team see a path toward a first-class autoscaler integration?
4. Would a Karpenter cloud provider shim that talks to the Omni API be a worthwhile community contribution, or is there a simpler primitive we're missing?

We're happy to prototype whichever direction the team thinks is most aligned with Omni's architecture. Linking the provider repo in case it's a useful context for the discussion.

Reference: [bearbinary/omni-infra-provider-truenas](https://github.com/bearbinary/omni-infra-provider-truenas) — TrueNAS SCALE infrastructure provider (JSON-RPC 2.0, ZFS-native, SideroLink)

[Unix4ever]

We wanted to use https://github.com/kubernetes/autoscaler
But autoscaling feature never had enough interest, so we never got to it.
It's a low hanging fruit now with all things we have implemented already.
See this epic for more details.

[Cliftonz]

I understand, how do you plan to gauge interest in this?
demos, sales calls, etc

Is there anything I can do to help, test, etc?

[rothgar]

We've had enough people ask for it that it was worth spending a day building it. Getting feedback on how people actually use it and if they want it integrated into Omni is the next step.

A lot of our Omni customers don't use any autoscaling because they're running in fix capacity environments with a known size of customers. Autoscaling doesn't save any money when they own the compute.

[Cliftonz]

@rothgar I want to share my perspective because I think the hypervisor use case is fundamentally different from bare-metal, and it changes how autoscaling should work.

On bare-metal, autoscaling makes intuitive sense — you have a rack of machines, the bare-metal provider adds them to the cluster when provisioned.

On a hypervisor, it's different. The machines don't exist until the provider creates them.

When the autoscaler says "add a node," my TrueNAS box has to carve out RAM, CPU, and a zvol from a shared pool that other clusters and other workloads on the NAS are also using.

And when it says "remove a node," those resources genuinely go back into the pool for someone else.

This means hypervisor-backed autoscaling is closer to cloud autoscaling than bare-metal.

So you're creating and destroying compute on demand. But unlike AWS, you own the box.

You know exactly how much capacity is left and, in my experience, you're probably running multiple clusters on the same hardware.

My goal is to run several clusters on a single TrueNAS box.

Right now they're all statically sized, I set replicas: 3 on a MachineClass of the minimum spects and that's what runs 24/7.

At 3am when nothing is happening, I have 6 worker VMs idling, eating RAM that could be used for ZFS ARC cache or other services.

During the day when CI kicks off, CD runs a deployment, or I am running AI building systems, pods sit pending because I sized my workers for "average" not "peak."

The worst part is I can see the waste.

My provider already reports host RAM, CPU, and pool space as OTEL metrics.

I watch Grafana and see 40GB of RAM allocated to idle VMs while my NAS struggles with ZFS operations because ARC got squeezed.

What I want my clusters to breathe.

Scale up when there's work, scale down when there isn't.

And I want the hypervisor to be the capacity referee.

For example:
Night shift / day shift. My batch processing cluster should scale up to 8 workers at 10pm when the ETL jobs run, then drop back to 1 by 6am. My dev cluster should do the opposite — 5 workers during business hours, 1 overnight.
They're on the same host. They should take turns using the capacity. The Kubernetes Cluster Autoscaler already handles this naturally when pods create pressure or nodes get added.
Pods finish, nodes drain. I just need the infra provider to respond and the system to be smart about capacity.

Right-sizing. Not all workloads need the same VM shape. A build job needs 8 CPUs and 4GB RAM. A database needs 2 CPUs and 16GB RAM. A model inference job needs a GPU.
Today I'd have to create separate MachineClasses for each and manually size them.
I'd rather the autoscaler pick the right class based on what's pending the way EKS Auto picks instance types.

Capacity awareness. This is the big one, and it's where hypervisors diverge from cloud.
AWS has effectively infinite capacity and the autoscaler never needs to ask "is there room?".
On my TrueNAS box, there absolutely is a ceiling.
If I have 256GB of RAM and 200GB is allocated, the autoscaler shouldn't try to provision a 64GB worker that would OOM the host.
My provider already does a memory overcommit pre-check and it blocks VMs requesting more than 80% of host RAM. But that's a hard wall, not a conversation.
The autoscaler should know before it asks that there's only room for 2 more 8GB workers, not 10.

So here is how I think it should work

1. Pods are unschedulable → Cluster Autoscaler triggers scale-up
2. Autoscaler tells Omni "add 1 node to MachineSet X"
3. Omni updates the MachineSet → provider gets the request → creates a VM
4. VM boots Talos, joins the cluster, pods schedule

Scale-down is the reverse, so when node is underutilized, autoscaler drains it, Omni updates the MachineSet, provider stops and deletes the VM.

This works the same whether the provider is TrueNAS, Proxmox, Nutanix, or VMware. The only difference is how each provider creates/destroys VMs and what capacity signals it can report back.

On the infra provider side, I can add a capacity endpoint that answers "how many more VMs of this MachineClass can this host support?" based on real-time host metrics. My provider already has all the data:

• Available RAM (host total minus allocated)
• Pool free space (for zvols)
• CPU headroom

That information should flow back to the autoscaler so it can make informed decisions instead of blindly requesting nodes until the provider starts failing.

Ideally, this becomes a first-class Omni feature rather than a sidecar, but even as a sidecar it's useful today.

Happy to test against the TrueNAS provider and share what I learn.

[rothgar]

I don't understand how what you described is different from how things work.

The autoscaler is responsible for watching for unscheduled pods and requesting an appropriate amount of resources. If you have multiple machine sets it should pick the correct one to scale (eg GPU machine set). That's all it does.

The infrastructure provider reacts to the request by calling the providers APIs to add or remove machines. That's all it does.

If the provider doesn't have resources it's supposed to respond to the API requests with a failure. This is how it works on a single server and in AWS. When AWS doesn't have capacity of a specific instance type or in a specific AZ (it happens frequently) it will respond with an error and the cluster autoscaler will try a different machine set. The autoscaler is not designed to know if this request will succeed or fail before it sends the request to the machine set. It doesn't know about infrastructure providers or where the capacity comes from. It only does math to make sure the pending resources are matched with available resources.

Omni is just the API for where the request state is stored and schedules nodes to be removed from the cluster. Talos actually handles draining.

You could avoid a lot of this complexity by using your single server to run raw containers without an orchestration layer. Linux and BSD are surprisingly good at balancing resource constraints on a single node and adding a VM layer for manual resource allocation will never be as efficient or as fast.

[Cliftonz]

You're 100% correct, I was trying to clarify the use cases I would use this for so that I could help give a reason to build a case for this feature.

I wasn't trying to propose changes to how the autoscaler, Omni, or providers work or interact.

What I was trying to communicate is why autoscaling with talos/omni was meaningful from the hypervisor side.

In my opinion, the use cases I listed above that make this worth building into Omni as a first-class feature rather than leaving it as a controller.

To me, it makes total sense that on bare-metal, people often have fixed capacity and known workload and since this is the majority of your customers, I get why you have the pause.

What I want to communicate is that on hypervisors, it's different.

Right now, I am running multiple clusters on shared hardware, Because frankly, I can't afford to run actual clusters on dedicated hardware yet.

On this shared infrastructure, the Talos VMs don't need to exist until they're needed.

When they're removed, the resources actually go back to the host for ZFS, other clusters, or other services.

For me, Autoscaling on a hypervisor isn't about saving money on a cloud bill; it's about not wasting physical resources that I have already paid for.

I'd love to be running these clusters on bigger, dedicated hardware and eventually the goal, and unfortunately, right now I'm working with what I have and trying to squeeze the most out of it.

If I'm just being too much, let me knew.

I think this would be a genuinely valuable feature, and I'm happy to help in any way I can.