Agentic RL: Environments, Trajectories, and the Training Loop
The full RL nanodegree, covered with implementation.
Instant H100 access. Fully on demand for AI teams.
Most GPU clouds sell you a VM and hope the abstraction holds. Lightning AI just shipped something that inverts that.
Their AI Cloud is based on the idea that instead of leasing hardware from someone else and building software on top, own the entire stack: GPUs, datacenter fabric, hypervisor, and scheduler as one system.
Why that matters mechanically:
Topology stops being a guess. Because Lightning controls the hypervisor, the guest sees the real hardware topology instead of a flattened abstraction. NCCL and PyTorch work correctly without manual tuning.
Multi-node placement becomes deterministic. Because they own the datacenter, the scheduler knows the physical switch fabric in advance and will land jobs on well-connected machines instead of scattered leased capacity.
Provision has no leased layer sitting between a request and the hardware, so there’s one operator accountable for the machine actually showing up.
The AI Cloud is now available as guaranteed on-demand or spot capacity, on the same platform behind PyTorch Lightning.
Infrastructure ownership determines whether your placement, your performance, and your provisioning are things you can actually reason about, whereas when you rent your infra, you’re hoping someone (or a series of vendors) get it right in their cloud.
Request access to the Lightning Cloud now →
Thanks to Lightning for partnering today!
Agentic RL: Environments, Trajectories, and the Training Loop
Part 12 of the Reinforcement Learning course is available now.
The previous chapters covered GRPO, verifiable rewards, and LLM-as-a-judge. This one extends all of that into a working training loop for an agent that takes multi-step actions with tools.
Read Part 12 of the RL course here →
It covers:
What an RL environment means for an LLM agent
Trajectories as the unit of training
The environment spectrum from state-changing to static worlds
Outcome vs. process-informed scoring
RULER as a packaged judge with prefix deduplication
The credit assignment problem in long-horizon episodes
Hands-on: training a SQL agent with ART and RULER on free Colab
Read Part 12 of the RL course here →
Why care?
Right now, most agents in production are shaped entirely through prompt engineering. The system prompt tells the agent what tools it has, how to reason, and when to stop.
All the intelligence comes from the base model’s pre-existing capabilities and whatever structure the prompt imposes.
That works up to a point, but they cannot reward the behavior you want and penalize the behavior you do not want, across thousands of rollouts, the way RL can.
This chapter covers the complete RL loop.
It defines what an RL environment looks like when the agent is an LLM with tools, formalizes the multi-step trajectory as the training object, introduces RULER as the scoring layer, and runs the full rollout-score-update cycle end to end.
The hands-on section trains a 3B parameter model to answer database questions using only tool calls, on a free Colab GPU, with no hand-written reward function beyond a one-line correctness bonus.
Here’s what we have covered so far:
👉 Over to you: What topics would you like us to cover in this RL series?
Skill bundles in Hermes agent
We found skill bundles to be the most underrated feature in Hermes.
Real workflows need clusters of skills together, not one at a time.
For example, writing code might need a code review skill, a testing skill, and a PR workflow skill.
Every time, you’re loading the same group manually with three slash commands in sequence, or asking the agent in natural language and hoping it picks the right ones.
A skill bundle fixes this with a single YAML file that groups multiple skills under one slash command.
When you invoke it, every listed skill loads at once, plus any custom instruction you’ve baked in.
The image below shows the anatomy of a bundle file and how it expands at invocation with one command that has a shared instruction baked in.
But the design choices underneath are what make it practically useful:
1) Missing skills are skipped, and it’s not fatal.
→ If one skill in the bundle is uninstalled, the rest still load. You don’t lose the whole workflow because of one missing piece.
2) Bundles beat skills on name collisions
→ If a skill and a bundle share a name, the bundle wins since you opted into it.
3) It works everywhere
→ CLI, TUI, dashboard, Telegram, Discord, Slack. one YAML definition, all platforms.
4) No cache invalidation
→ Bundles generate a fresh user message at invocation, the same as individual skill loading, and at no performance cost.
Once your bundles are stable, the natural next step is team sharing.
Bundle YAMLs are just file so you can just put them in a Git repo and have each team member symlink into the ~/.hermes/skill-bundles/ folder.
If you update the repo, everyone gets the update.
This is what turns a personal agent into something a small team can standardize around.
We wrote a full deep dive covering Hermes agent’s self-evolving skills, three-tier memory, GEPA optimization, and setting up multiple specialized agents.
Good day!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn everything about MCPs in this crash course with 9 parts →
Learn how to build Agentic systems in a crash course with 14 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.