How to gather loss across GPUs?

[tnq177]

I'm trying to calculate perplexity. To do this, I need to sum all the loss across GPUs. It seems to me the loss return from engine(input) is for each GPU as when I print it out:

print(f"Rank = {engine.global_rank} ,", nll_loss.item())

I got:

Rank = 1 , 1.7060546875
Rank = 2 , 1.169921875
Rank = 0 , 1.052734375
Rank = 6 , 0.240478515625
Rank = 4 , 1.2724609375
Rank = 7 , 0.81884765625
...

How do I gather the loss from all GPUs please?

[tjruwase]

You can do a reduce to sum the losses, such as this.