Regarding SSD endurance and KV Caching

[celestial-rose]

Hi oMLX team,

I’ve been following the development of this project and it looks super promising. The benchmarks are impressive, and the way you've implemented SSD KV Caching seems like a game-changer for handling large contexts on Apple Silicon for agentic harnesses.

I haven’t had the chance to test it out yet, because I have one lingering concern: SSD wear-and-tear.

Since high-context coding sesh involves significant data movement, I’m worried that constant read/write operations to the SSD for KV Caching might significantly impact the drive's lifespan. Given that Mac SSDs are soldered, I’m being a bit cautious before diving in.

I’d love to hear your thoughts on:

• How "heavy" are the writes during a typical session?

• Are there any optimizations in place to reduce the frequency of disk writes?

• Would you recommend using an external SSD for the cache to protect the internal drive ? Is that a possibility ?

I’m really excited about the speed and efficiency oMLX offers; I just want to make sure I’m being mindful of my expensive hardware. Thanks for the amazing work !

NB : this is maybe totally irrelevant and maybe the SSD operations are lower than playing opening 100 Chrome tabs but I thought asking could help others too in their decisions

[kuanjames]

I bought an external SSD drive just for this. Solves all problems.

[kuanjames]

Other people use hot cache (RAM instead of SSD), so you could try that for maximum speed.

[celestial-rose]

Other people use hot cache (RAM instead of SSD), so you could try that for maximum speed.

sure but that's how regular KV cache works !

[moritzchow]

Other people use hot cache (RAM instead of SSD), so you could try that for maximum speed.

sure but that's how regular KV cache works !

Well if you have used other inferencers you know that the KV cache is not working quite well on MLX because the shift of attention invalidates immediately the kv cache and the model will have to recalculate each time for that, that’s where oMLX fixes by applying prefixed cache so the shift of attention won’t invalidates immediately invalidates the cache. This does not happen to other engines like llama.cpp

[celestial-rose]

Other people use hot cache (RAM instead of SSD), so you could try that for maximum speed.

sure but that's how regular KV cache works !

Well if you have used other inferencers you know that the KV cache is not working quite well on MLX because the shift of attention invalidates immediately the kv cache and the model will have to recalculate each time for that, that’s where oMLX fixes by applying prefixed cache so the shift of attention won’t invalidates immediately invalidates the cache. This does not happen to other engines like llama.cpp

Yeh I'm following every MR around KV Cache for MLX on mlx-lm for example and other servers especially to get more insights about it.
I'm interested in oMLX exactly because it does things differently but I still wonder how it works.
You're saying that oMLX allows for KV Cache on RAM too ? It's not only SSD ? How do we disable it ? Thanks for your answer

[kuanjames]

I bought an external SSD drive just for this. Solves all problems.

but it's then slower so it kinda defies the purpose

EDIT: also, how do we pick which drive to use for caching ?

You can change the path in the General Settings page