[Question] [Usage]: MoriEP is equivalent to which mode in DeepEP: `throughput` or `low_latency` mode ?

[tjtanaavllm]

I saw that MoriEP supports intra-node and inter-node ops, and I have a few questions:

1. MoriEP is equivalent to which mode in DeepEP: throughput or low_latency mode?
2. Is MoriEP beneficial in the case no non-PD inferencing?
3. In PD inferencing, is MoriEP suitable to be used in Prefill instance , Decode instance or both?

[TianDi101]

I saw that MoriEP supports intra-node and inter-node ops, and I have a few questions:

1. MoriEP is equivalent to which mode in DeepEP: throughput or low_latency mode?
2. Is MoriEP beneficial in the case no non-PD inferencing?
3. In PD inferencing, is MoriEP suitable to be used in Prefill instance , Decode instance or both?

Hi @tjtanaavllm , thanks for these good questions.

1. It's more similar to low_latency mode. Current design aims to reduce redundant data transfer as much as possible. We're adding another version to reduce memory footage.
2. For non-PD inferencing, it depends. For single-node case, since the intra-node bandwidth is large enough, using AllReduce / AllGather instead of MoriEP should not have a big performance impact. But for multi-node case, MoriEP is expected to perform better than CCL's ring algorithm for two reasons: 1) fewer communication round trips 2) fewer data size (for AR/AG/A2A padding is required)
3. It works for both. One consideration is that it allocates a relatively large buffer, but for prefill instances in PD disaggregation, I guess this should not be a problem (I'm not an expert in PD disaggregation so please correct me if I'm wrong).

[alokprasad]

@TianDi101 can we say for PD disaggregation MORI-IO is better choice than MORI-EP?

[TianDi101]

@TianDi101 can we say for PD disaggregation MORI-IO is better choice than MORI-EP?

@alokprasad MORI-IO and MORI-EP serve completely different purposes. The former is used for P2P communication (KVCache Transfer) and the latter is used for EP communication (dispatch/combine).

They are indeed orthogonal, you can use them both in PD disaggregation.