Explained with code and visuals.
There is a typo in the loss computation of mixed precision. It must be loss calculated in float16 (same as weight’s precision) and then upscale it up float32.
There is a typo in the loss computation of mixed precision. It must be loss calculated in float16 (same as weight’s precision) and then upscale it up float32.