Does this actually work when unet is in cpu?

[Symbiomatrix]

My goal is not actually to run on cpu, but to temporarily move a model there in order to free memory for other programs (eg ollama) and quickly move it back once they're done.
As far as I can tell, this is only possible when the model is fully connected to an actual gen workflow, even a basic dummy one with 1 step - anything less and the model is completely unloaded.
I tried such a dummy workflow with a flux FP8 finetune as unet on cpu, and clip + t5x fp8 on gpu. The move is performed successfully, however when I reach the ksampler stage, I consistently receive the following errors:

NotImplementedError: No operator found for `memory_efficient_attention_forward` with inputs:
     query       : shape=(1, 4352, 24, 128) (torch.float32)
     key         : shape=(1, 4352, 24, 128) (torch.float32)
     value       : shape=(1, 4352, 24, 128) (torch.float32)
     attn_bias   : <class 'NoneType'>
     p           : 0.0
`decoderF` is not supported because:
    device=cpu (supported: {'cuda'})
    attn_bias type is <class 'NoneType'>
`[email protected]` is not supported because:
    device=cpu (supported: {'cuda'})
    dtype=torch.float32 (supported: {torch.bfloat16, torch.float16})
    operator wasn't built - see `python -m xformers.info` for more info
`cutlassF-pt` is not supported because:
    device=cpu (supported: {'cuda'})
`smallkF` is not supported because:
    max(query.shape[-1] != value.shape[-1]) > 32
    device=cpu (supported: {'cuda'})
    unsupported embed per head: 128

The model can be moved back to the gpu quickly at any point afterwards (so for a separate process it works), but due to the error, this dummy offloading is inadequate as a step in a bigger workflow. Running something else seems to trigger a full reload, though the model doesn't seem to be removed from ram.

What do you suggest?

[Mescalamba]

Well, if you load it into CPU, its simply on CPU and it doesnt work there.

But you can load model into GPU and let it offload 100% of it via virtual vram size setting.

And for example SDXL based models work even when offloaded fully to system memory. FLUX should too, if you have fast system ram, it can probably work really well.

[Symbiomatrix]

@Mescalamba Perhaps I'm missing something - isn't the virtual vram setting available only for distorch, ie gguf? As stated, the model's an FP8 finetune. I'd love to convert it but city's process as it stands is far from trivial.
That's a neat trick though, hadn't thought of it.

[Mescalamba]

fp8 finetune can be probably transformed directly to Q8, just there are no tools for that, think I will make one, cause it happens often that I want to GGUF something and cant cause its fp8

Virtual ram works only on GGUF, you are right. I wasnt exactly sure what you aiming for with loading diffusion model into CPU/RAM. That obviously wont work.