In today’s newsletter:
Your AI coding agents should get smarter every time you use them.
12 optimizers for training neural networks, clearly explained.
Your AI coding agents should get smarter every time you use them
AI coding agents write good code now, but they still don’t learn from experience.
Say you spend 30 minutes helping Claude Code debug a tricky issue. It explores your codebase, tries several approaches, hits dead ends, and eventually finds the fix.
Tomorrow you start a new session, and the agent makes the exact same mistakes.
Switch to Codex or Cursor and it gets worse. They have no idea what your previous agent figured out.
Every session produces useful knowledge: debugging breakthroughs, failed approaches, architectural decisions, and patterns specific to your codebase.
But most of it disappears when the session ends.
Beacon, built by Asymptote Labs, is an open-source memory layer that lets your coding agents learn from past sessions.
It works across Claude Code, Cursor, Codex, OpenCode, Cline, and 20+ other agent harnesses.
Beacon launched on Product Hunt today.
Check out the launch and show your support with an upvote →
Instead of treating every session as an isolated conversation, Beacon captures what your agents actually do and turns that experience into reusable knowledge.
Here’s how it works:
↳ Capture every session. Beacon records execution traces across your coding agents, including commands, file changes, errors, and debugging paths.
↳ Extract reusable knowledge. It picks out successful fixes, recurring workflows, and lessons worth remembering.
↳ Improve future sessions. Those lessons become skills and context your agents can use, even in a different coding tool.
↳ Close the feedback loop. Later traces show when saved knowledge stops working, so it can be updated or retired.
Beacon Cloud continuously finds ways to improve your agents
Capturing sessions is only the first step.
Every hour, Beacon Cloud reviews your agents’ new activity and turns it into findings in three categories:
↳ Security: risky things your agents did. Secrets pasted into prompts, destructive commands, permission bypasses, or connections to unknown MCP servers. Beacon flags the action, shows which session it happened in, and suggests a fix.
↳ Skill: workflows worth reusing. When you keep walking agents through the same debugging steps, Beacon notices and drafts a reusable skill. You download it and make it available to all your coding agents.
↳ Behavior: patterns in how you work with agents. Retry loops, abandoned sessions, unnecessary approvals, or expensive models used for simple tasks. Beacon points out the waste and recommends faster, cheaper ways to work.
The findings keep updating as your agents generate new traces.
One memory layer for every coding agent
Beacon sits across the agent harness layer.
So your Claude Code debugging lessons can help your next Codex session, and your Cursor discoveries can become skills in OpenCode.
With beacon handoff, you can also resume work in a different agent when you hit a rate limit or need to switch tools. You don’t re-explain the repo or retrace failed approaches.
It matters even more across a team. One engineer spends an hour debugging an infrastructure issue, Beacon captures the approach that worked, and the next engineer can reuse it as a skill in whichever coding agent they use.
The loop looks like this:
Run agents → Capture traces → Generate findings → Reuse skills → Improve → Repeat
Beacon runs locally without a hosted backend. Beacon Cloud adds hosted trace storage, cross-harness search, session analytics, and the findings above.
A smarter model only gets you so far. The coding agent that keeps getting better is the one that learns from experience.
Support Beacon on Product Hunt →
Stop teaching your agents the same lessons twice.
Thanks to Asymptote Labs for partnering today!
12 optimizers for training neural networks, clearly explained!
An optimizer can reduce the number of training steps and still make your run slower. Each update has its own compute and memory cost, which becomes part of the training budget.
That tradeoff gets easier to reason about once you look at what each optimizer stores and how it transforms the gradient.
The 12 optimizers in the visual approach this differently. Let’s break them down through the engineering decisions they change, starting with how much gradient history survives each update.
Smoothing gradients and adapting step sizes
SGD with momentum keeps a running accumulation of gradients. Directions that persist across batches build up, while inconsistent directions tend to cancel.
This helps reduce oscillation, but momentum alone doesn’t give each parameter its own adaptive step size.
AdaGrad adds that adaptation by accumulating squared gradients for each parameter. Frequently updated parameters receive progressively smaller effective steps, which can help with sparse features.
Its limitation follows directly from that accumulation: the denominator keeps growing, so effective learning rates can become too small.
RMSprop replaces the full accumulation with an exponential moving average of squared gradients. Older observations fade, allowing the scaling to respond to more recent gradients.
Adam combines momentum with this second-moment scaling. One moving average tracks gradient direction; another tracks squared gradient magnitude.
That second quantity isn’t a direct curvature measurement. Adam adapts to gradient statistics without estimating the Hessian.
Changing regularization, memory, and gradient history
AdamW separates weight decay from the adaptive gradient update. Adding an L2 penalty to Adam’s loss lets its adaptive scaling affect that penalty; decoupled weight decay applies parameter shrinkage separately.
That distinction matters when you’re tuning regularization. Weight decay becomes easier to reason about independently of the gradient scaling.
Memory introduces another constraint. Standard Adam maintains two parameter-sized state arrays, in addition to the model weights and gradients.
With one billion parameters and FP32 moment buffers, those two arrays alone require about 8 GB before sharding or compression.
Adafactor reduces the second-moment storage for matrices using row and column statistics. For an n × m matrix, that replaces nm stored second-moment values with roughly n+m.
You’re trading a full per-parameter estimate for a factored approximation. That’s useful when optimizer state limits the model or batch size you can fit.
Lion keeps momentum without a second-moment buffer. Its gradient update uses the sign of a momentum-gradient blend, giving coordinates the same update magnitude within a parameter group.
That changes the update scale, so copying AdamW’s learning rate into Lion isn’t a fair comparison. The Lion paper specifically calls for smaller learning rates.
AdEMAMix takes another approach: it combines fast and slow moving averages of gradients alongside second-moment scaling. Recent gradients remain influential while older directional information survives longer, at the cost of additional state.
Using matrix structure and curvature
Adam scales coordinates individually. Shampoo uses matrix-valued statistics to transform updates across parameter dimensions, capturing relationships that elementwise scaling misses.
Those richer statistics bring additional storage and matrix computation.
SOAP connects this approach back to Adam. It rotates gradients into the eigenbasis of Shampoo’s preconditioners, applies Adam there, and rotates the update back.
The coordinate system changes how Adam’s elementwise scaling acts on the update. Computing and refreshing that basis also adds overhead.
Muon approximately orthogonalizes momentum updates for matrix parameters using Newton–Schulz iterations. It changes the matrix update’s singular-value structure rather than scaling every coordinate independently.
Parameter selection matters here. Muon is typically applied to hidden weight matrices, with another optimizer handling embeddings, output layers, and vector parameters.
Sophia uses a diagonal curvature estimate to scale momentum, then clips updates elementwise. Its curvature estimate is refreshed periodically, limiting the average overhead of obtaining that information.
Comparing optimizers on your workload
I’d use AdamW as the reference run for a transformer experiment, then choose alternatives against a specific constraint.
For memory pressure, investigate Adafactor or Lion. For matrix updates, evaluate Muon, Shampoo, or SOAP while accounting for their parameter coverage and implementation costs.
Give each optimizer a learning-rate search. Keep the data, model, and evaluation protocol consistent, and report both validation quality at a fixed compute budget and time to a target quality.
Also measure peak memory and step time. An optimizer that reaches the target in 20% fewer steps but makes each step 30% slower takes about 4% longer overall.
The useful result is a reproducible improvement under your training constraints, including the cost of tuning it.
If you want to build these foundations properly, our Classical ML and Deep Learning roadmap covers the full stack in order, from the assumptions behind linear models to neural network training and the architectures built on top:
Start the Classical ML and Deep Learning roadmap →
👉 Over to you: Which optimizer has improved your training runs enough to justify changing your default?
Good day!