Encoder-decoder inference with fast kernels

[saikrishb]

Hi, I'm interested in using deepspeed to run inference on a BART model, with kernel injection. I see this PR open for some time now. Any ETA on when this feature would be available?
#2451

@RezaYazdaniAminabadi

[RezaYazdaniAminabadi]

Hi @saikrishb

Thanks for your interest. I am actively working on this. It should be out most likely by beginning or middle of december.
Best,
Reza

[drxmy]

Hi @saikrishb

Thanks for your interest. I am actively working on this. It should be out most likely by beginning or middle of december. Best, Reza

I am new to inference optimization but trying to deploy a mt5-xxl. Is there something else i could do after using this awesome kernel injection?
Really appreciate it if you could give any advice.