For this reason, fp16 is the default for reduction operations because the loss is minimal when averaging gradients.