In today’s newsletter:
Image-aware agent memory!
k-means vs. e-means, clearly explained.
Why self-improving agent harnesses break when the model changes.
Image-aware agent memory!
Agent memory has mostly been built around text, since that’s what agents read and write the most.
Oracle just released image-native memory in Oracle AI Agent Memory, where your agent stores an image once and retrieves it later with a plain text query:
Each image is saved with a short description and scoped to a user, an agent, or a thread.
The description is embedded like any other memory, and the raw image bytes are stored in Oracle AI Database alongside it.
At query time, a vector search runs over those descriptions to find images similar to the specified queries.
The search returns the description and metadata first, without the image bytes.
The app loads the original image only when it asks for it, so image data stays out of the context until a step needs the pixels.
Messages work the same way. A turn can hold text and image parts in order, so a message with a screenshot comes back exactly as the user sent it.
This avoids the usual tradeoff of either dropping images from memory or passing them into every future prompt and paying for those image tokens on each turn.
So your agent now remembers what users show it as well as what they say, and it spends image tokens only on the turns that need them.
You can find the full walkthrough here →
Thanks to Oracle for partnering today!
k-means vs. e-means, clearly explained
k-means has been the default way to cluster data for decades, yet it moves its centroids only once per full pass over the data.
A new paper from Meta FAIR introduces e-means, which reaches the same result up to 45x faster on GPUs.
Both algorithms have the same goal. The only real difference is how often they move the centroids.
How standard k-means moves the centroids
The standard version is Lloyd’s k-means, and it runs in two steps.
In the assign step, every point in the dataset joins the group of its closest centroid.
In the update step, each centroid snaps to the mean of all the points in its group, and then the loop repeats.
The problem is the order of these steps. k-means assigns every point before it moves anything, so one full pass over the data buys exactly one centroid move.
On a dataset with 10 billion embeddings, that is a lot of reading for one small step.
Why one move per pass is slow
e-means flips the order. It assigns a small batch of points, moves the centroids a little, and then goes on to the next batch.
For example, your data might have 10,000 points, with 100 points in each batch.
k-means looks at all 10,000 points before it moves the centroids once. e-means moves them after every 100 points, so it makes 100 moves in the time k-means makes 1.

The work per pass is about the same, since both still compare every point against the centroids. The difference is that e-means puts that work to use 100 times instead of once.
That is why e-means reaches the loss that k-means gets after 25 epochs in about 2 epochs.
How e-means nudges the centroids
e-means does not snap a centroid to the mean of a batch. Instead, it keeps each centroid as a running average (an exponential moving average) and nudges it toward every new batch.
The size of that nudge changes over the run. Early batches move the centroids a lot, and later batches move them less, following a fixed cosine schedule.

Big steps early let the centroids travel across the space quickly. Small steps late let them settle without being thrown off by one noisy batch.
How e-means revives empty clusters
Every k-means implementation has a known failure, where a centroid ends up with no points at all and never moves again.
e-means checks for these dead clusters every 5 batches. It revives each one by splitting a crowded “donor” cluster in two, which is the same procedure Faiss uses.
In the paper’s ablations, this revival step had the biggest effect of all the changes to the update rule.
Issues with small-batch k-means
Small-batch k-means is not a new idea. MacQueen described a sequential version in 1967, and Sculley’s mini-batch k-means came out in 2010.
Neither one replaced the standard version, because both had three problems that e-means now fixes:
↳ Centroids slowed down too early. e-means starts with big nudges and shrinks them on a fixed schedule, so the centroids keep moving until the end.
↳ Empty clusters stayed dead. e-means checks every 5 batches and revives each empty one by splitting a crowded cluster.
↳ You had to guess the settings. e-means sets the batch size and the step schedule from scaling rules, so there is nothing to tune.
The authors fitted these scaling rules on 75,000 optimization runs over a single dataset (DEEP), and the same formulas held on the other datasets without any hyperparameter sweep.
Benchmarks
On a wide range of embedding datasets, e-means beats the loss of Lloyd’s k-means at 25 epochs in just 1.4 to 2.6 epochs.
On GPUs, that turns into 10x to 45x wall-clock speedups over state-of-the-art k-means implementations.

The largest test is DINO10B, a dataset of 10 billion DINO embeddings. e-means split it into 16 million clusters in 2.5 epochs and still finished below the loss that Lloyd’s k-means reaches at 25 epochs.
It also often lands at a lower loss than k-means++, which is the usual fix for bad starting centroids.
Running e-means
e-means ships inside Faiss, and a CPU run needs only faiss-cpu and NumPy:
import numpy as np
from faiss.contrib.e_means import NumpyEmeans
x = np.random.rand(200_000, 64).astype("float32") # 200,000 vectors of dimension 64
em = NumpyEmeans(64, 500) # 500 clusters, 5 passes over the data by default
centroids, labels = em.train(x) # (500, 64) centroids, and each vector's cluster
x_new = np.random.rand(100_000, 64).astype("float32")
distances, new_labels = em.assign(x_new) # nearest centroid for new vectorsOn GPUs, it runs through PyTorch with Triton or CuTe kernels, and the assignment_backend argument picks between them.
This matters most in places where k-means quietly sits inside bigger systems, like building IVF indexes for vector search, quantizing embeddings, and curating large training sets.
In all of them, the clustering step is not doing anything smarter. It is simply getting 100 times more use out of every pass over the data.
If you want to go beyond k-means, we have covered clustering and its close cousin, dimensionality reduction, from first principles:
Gaussian Mixture Models, where every point gets a soft, probabilistic cluster assignment →
HDBSCAN, which finds clusters of different densities without needing k →
Why self-improving agent harnesses break when the model changes
A new paper from Google looks at what happens to recursive self-improvement (RSI) when the model under an agent changes.
In RSI, an optimizer agent improves an agent harness on its own. It reads failure traces, edits the prompts, tools and completion checks, and keeps whatever scores better on a benchmark.
This works until the model underneath changes. A harness evolved on Claude Opus and then moved to Gemini scored below Gemini’s plain starting harness in every domain Google tested.
Why the gains do not transfer
When the harness moves to a new model, the only thing carried over is the evolved harness code, meaning the prompts, checks and rules the optimizer added.
The record of how each piece got there stays behind, including which failure it targeted and whether it fixed that failure when tested.
So the optimizer working on the new model inherits a set of changes with no reasons attached. It ends up retrying experiments that already failed, or removing fixes it does not understand.
How Self-Improvement Memory works
Google’s fix is Self-Improvement Memory (SIM), a log kept for the optimizer itself rather than for the agent that solves the tasks.
Each entry is a recurring problem, like an agent declaring a task done before its commands ran, and every fix attempted under it stores:
The edit and the reasoning behind it
The tasks it was predicted to fix, written down before testing
The tasks it actually fixed, failures included
The model and benchmark it was tested on

The whole log goes into the optimizer’s prompt every round, and new attempts get appended as optimization continues.
New direction with SIM
The paper shows the difference on one inherited check that stops the agent from finishing early. With SIM, the optimizer read why the check existed and extended it.
Without SIM, it rewrote the same check four times, and twice that was to fix deadlocks it had created itself.

On the Opus-to-Gemini workspace tasks, both runs started from a score of 68.41. The run with SIM reached 84.75 while the run without it reached 73.42, which is about 3x more improvement from the same start.
SIM also finished ahead in 9 of 10 independent coding runs, by 2.36 points on average.
So when an optimizer edits your harness, the reasons behind each edit are worth keeping just as much as the edits themselves.
We covered what goes into a production agent harness, and why it matters as much as the model, in this deep dive: The Anatomy of an Agent Harness →
👉 Over to you: Where does k-means lie in your stack today, and how long does a full clustering run take?
Good day!









