[Feature]: AutoDeploy: Allow for specifying mamba cache dtype #8763
Description
🚀 The feature, motivation and pitch
For accuracy reasons, mamba caches need to kept in float32. In vllm users specify cache dtype using --mamba_ssm_cache_dtype flag
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status