about delayed parameter update in ZeRO-Offload

[simey1128]

I want to see the code of DPU(Delayed Parameter Update) in ZeRO-Offload
but I can't find it.

please let me know where the location of that part.

thank you

[minjiaz]

Hi @simey1128,

DPU has not been committed to DeepSpeed main repo because it introduces 1-step staleness in parameters and changes loss slightly in each training step. Because of this, it cannot pass the unit tests in DeepSpeed that we added for checking the correctness of system optimizations.

To enable delay parameter update, the files that need to be changed can be found in the repo https://github.com/jren73/delay_param_update. Also, note that this implementation was based on DeepSpeed v0.3.0.

[simey1128]

thank you!!!! It is very helpful to me :)

[taehyunzzz]

@simey1128 Hi, did you manage to see the parallelization of weight update with forward+backward passes? CUDA streams did not do the job for me.

[simey1128]

@taehyunzzz hi, It seems that there's no overlapping gradient offload with BWD pass on single GPU
but on multiple GPU, the overlapping is done in gradient reduction stage

[taehyunzzz]

@simey1128 Thank you for sharing your experience!