Discussion about this post

User's avatar
Abhishek Pan's avatar

Weight decay is the knob people forget when they cut the learning rate for Lion. Decoupled decay is applied as lr * λ, so dropping lr 3 to 10x also cuts regularization 3 to 10x unless λ goes up by the same factor, which is what the Lion paper's authors recommend.

Nan Deng's avatar

The cost I would log next to step time is the optimizer state, not just the parameter count. At one billion parameters the post's own arithmetic is two FP32 moment buffers, about 8 GB, before the weights and gradients, while Adafactor keeps roughly n plus m second-moment statistics per matrix instead of n times m. A method that needs fewer steps can still lose if those buffers force a smaller batch or spill the update off the GPU. I would not copy an AdamW learning rate onto Lion either: Lion updates with the sign of a momentum blend and asks for a smaller rate, so a speed comparison that reuses the AdamW schedule is measuring a mistuned run.

Ready for more?