Redis built a cache that cuts LLM costs by 90%!
Production LLM apps do not receive completely new questions every time.
A customer-support assistant might receive all three of these:
“Can I get a refund after buying the monthly plan?”
“Is the monthly subscription refundable?”
“Can I cancel the plan and get my money back?”
The wording is different, but the underlying question and its answer remain the same.
Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated.
Prefix caching reduces part of these repeated calls.
When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix.
But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded.
So even with a prefix-cache hit, there’s another generation call involved.
To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it.
When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again.
A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call.
In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches.
If you want to use this, Redis already implements it as a managed service called Redis LangCache.
Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs.
Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud.
We built an interface to compare it against direct LLM inference:
For the paraphrased question in our run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens.
Redis returned the earlier response in 0.373 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run.
Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload.
You can try Redis LangCache here: https://redis.io/langcache/
Thanks to Redis for partnering today!
Why multi-turn agents need more than a task graph
Most agent frameworks represent one run as a graph. A call enters the graph, moves through its nodes, and returns an answer. That model works well for bounded tasks such as parsing a document and writing a summary.
A conversation does not follow the same lifecycle.
Every message starts another graph run, while the session must retain enough context to connect that run to the previous ones.
The same flow object can hold conversation data and execution bookkeeping, even though they need different lifetimes. Conversation history should survive across turns. Completed nodes and cached method outputs should reset.
CrewAI’s conversational flows add a layer around the graph to separate those lifetimes.
Let’s examine the failure first, then build a support agent with session continuity, routing, clean outputs, and session-level tracing.
Let’s begin!
Why repeated kickoffs replay previous output
A flow keeps two kinds of information during a run:
Application state holds data such as the current message.
Execution state records which nodes finished and what they returned.
Those records can share a lifecycle for a one-shot task because both become irrelevant when the run ends.
A conversation needs different behavior. Message history and domain data must survive for the next turn, while completed nodes and cached outputs must reset.
If both remain attached to the same flow instance, the second call may find its nodes already marked complete. It can then return the previous output without processing the new message.
A conversational layer separates these. It preserves session data, clears per-run execution records, and places the latest message into the new run.
State persistence across repeated kickoffs
A task run has a defined end. A conversation stays open for another message, so the second turn must begin a new run without losing the session around it.
Persisting the first run seems like the direct fix. It also creates the bug. Consider this small CrewAI flow:
@start()marks the step that opens the run.@listen(x)marks a step that runs after the stepxfinishes.@persist()on the class saves a snapshot of state, so a later run with the same session ID can pick it up.
from crewai.flow import Flow, listen, start, persist
@persist()
class SupportFlow(Flow[ChatState]):
@start()
def read_message(self):
return self.state.message
@listen(read_message)
def answer(self, message):
return f"answering: {message}"
flow = SupportFlow()
session_id = "customer-4471"
first = flow.kickoff(inputs={"id": session_id, "message": "Where is order 4471?"})
second = flow.kickoff(inputs={"id": session_id, "message": "Has that arrived yet?"})The two calls share a session ID but contain different questions. The expected output is:
answering: Where is order 4471?
answering: Has that arrived yet?The actual output repeats the first answer:
answering: Where is order 4471?
answering: Where is order 4471?What survives into the second turn
The answer step never runs during the second call. The same Python instance still holds the set of completed methods and their outputs.
When the graph checks that record, it treats both steps as finished and returns the earlier result.
A second leak sits underneath the first:
kickoffreadsinputs={"id": ...}as a checkpoint restore, so the first run’s snapshot comes back and brings the old message with it.When you clear the completed-method record without addressing that, the answer runs, reads state, and finds the previous turn’s message sitting there.
The completed-method record leaks execution progress. The restored snapshot leaks data. Both come from a finished run but remain active during the next one.
Class-level @persist() adds another edge case. It writes a snapshot after every method, so the latest row can represent the middle of a run rather than its final state. A later turn may restore that incomplete snapshot and miss updates made by the handler.
Run state, session state, and memory
Run state tracks the nodes and method outputs completed during one execution. It should disappear when that run finishes.
Conversation history stores messages exchanged during one session. It must persist across graph runs.
Memory stores facts that may outlive a conversation and appear in later sessions. Its retention policy is different from both.
The broken flow keeps run state alive on the instance while restoring conversational data from a snapshot. Without separate lifecycles, each mechanism can carry stale data into the next turn.
Requirements for multi-turn agent execution
A conversational flow must connect graph runs without carrying one run’s execution into the next. That requires four pieces.
Session identifies the conversation and restores the history needed by each new run.
Router sends the message to the appropriate execution path. A database lookup, research request, and clarification need different amounts of work.
Output layer separates internal agent work from the visible conversation. Tool calls and intermediate results remain traceable without filling the history.
Session trace connects otherwise independent graph runs into one interaction.
The session gives an otherwise independent run its conversational context.
Session persistence across runs
Every message carries a session ID. Messages from different customers resolve to different histories, even when the same deployment handles both.
Each turn follows the same cycle:
restore the conversation → process the new message → save the updated session
Restoration connects the new run to earlier turns. Without it, a follow-up reaches the model without the context that gives the message meaning.
Three kinds of data belong to the session rather than one graph run:
Conversation history, so the next run knows what was already said.
Domain data produced by earlier turns, so a finding from turn two remains available at turn six.
One-time setup that belongs to the session, such as permissions, cache warming, or loading customer records.
Execution progress is absent from that list. Carrying it forward caused the agent to replay its first answer.
Session storage remains an application decision. In-process storage stops being reliable once multiple workers handle requests because consecutive turns can land on different processes.
Session continuity tells a run which conversation it belongs to. Routing decides what the latest message needs.
Intent routing per turn
One agent can hold every tool, but that design makes the model decide when to search during the run. Weak results can trigger more tool calls and more model round trips.
A router makes the first decision before the expensive path begins and sends the message to a narrower handler.
An order question goes to a lookup handler with database access.
A question about a carrier delay goes to a research handler with web access.
A clarification goes to a handler that reads the history and calls nothing else.
Clarifications and follow-ups often need history but no new research. Sending them through the same research loop adds work without adding evidence.
Routing cost and fallback behavior
The router adds a model call to every turn, so its value depends on the execution paths it avoids. It makes sense when costly routes are uncommon. If every message requires research, the router becomes pure overhead.
An LLM also should not be the only routing mechanism. Deterministic rules are cheaper and easier to test when the intent is explicit. The router still needs defaults for failed calls and invalid route labels.
Traditional systems made a similar decision with intent classifiers. An LLM makes the route catalog easier to extend, but the cost and failure modes remain.
Separating agent outputs from message history
A research run produces more than its final answer. It may contain queries, tool results, failed branches, and intermediate findings. The next conversational turn rarely needs all of that material.
Writing the full execution record into message history forces later turns to process it again. The run therefore needs two destinations:
Intermediate work stays in the execution record and trace.
The final answer enters the conversation history read by the next turn.
The model receives the user-visible exchange, while engineers can still inspect the full execution when a run fails.
Excluding scratch work slows history growth but does not stop it. Long sessions still need a policy, such as a recent-turn window, a token budget, or a rolling summary that compresses older turns while preserving recent messages.
That summary belongs to conversation history. Durable facts extracted for later sessions belong to memory. Once the records are separate, tracing must reconnect them for debugging.
Session-level tracing
A per-run trace can report success even when the conversation drifts. One turn may answer a slightly different question, and later turns can build on that answer without raising an exception.
A session-level trace keeps routing decisions, handlers, tool calls, and replies in order across turns. It shows whether the interaction progressed correctly, not merely whether each graph run completed.
A deferred trace must eventually close. Otherwise, the batch may never export the conversation it contains. The application therefore needs a finalization path for explicit exits, timeouts, dropped connections, and abandoned sessions.
The implementation now has a clear contract: create a fresh run for every message, preserve the session around those runs, and close the trace when the session ends.
Implementing the conversational runtime
CrewAI implements that contract with conversational flows. Instead of calling the same flow directly for every message, handle_turn() wraps each kickoff with session restoration, per-turn execution reset, message handling, routing, and deferred tracing.
This API is experimental in CrewAI v1.15.6 and lives under
crewai.experimental, so pin that version if you use the code below.
The example is an e-commerce support assistant with one session and three routes that perform different amounts of work.
Session state and turn continuity
A conversational flow receives a message and session ID. The ID selects the conversation, while the state holds the history and domain fields needed by later turns.
from uuid import uuid4
from crewai.experimental.conversational import ConversationState
class SupportState(ConversationState):
last_order_id: str | None = None
...
session_id = str(uuid4())
flow = SupportFlow(
# The second message has no order number in it, so it resolves
# against what the first turn left in state.
flow.handle_turn("Where is order 4471?", session_id=session_id)
flow.handle_turn("Has that arrived yet?", session_id=session_id)ConversationState already contains message history and the last routing decision. The subclass adds domain fields required by the handlers.
The second message contains no order number. Its handler falls back to last_order_id, which the first turn stored in the session.
Route handlers
The router reads the current message and selects a handler. Each capability is an ordinary flow listener with a narrow tool set and execution path.
from crewai.experimental.conversational import RouterConfig
from crewai.flow import listen
router_config = RouterConfig(
prompt=(
"You route messages for an e-commerce support assistant. "
"Pick the route that matches what the customer is asking for right now."
),
llm=ROUTER_LLM,
default_intent="converse", # router call failed, or no LLM configured
fallback_intent="converse", # router returned a label that does not exist
)
# Each label becomes a route. Each docstring line becomes its description,
# which is the text the router LLM reads when deciding where a message goes.
@listen("ORDER_LOOKUP")
def handle_order_lookup(self) -> str:
"""Status, tracking, or delivery date for an order."""
...CrewAI includes three built-in routes:
conversefor ordinary chat and follow-ups,endfor goodbyes,an optional
answer_from_history, and you can override any of them by defining a handler with the same name.
The configuration prompt supplies domain context and business rules. CrewAI builds the route catalog from handler docstrings, which keeps route definitions tied to executable listeners.
Output visibility
The research route runs a two-agent crew:
The researcher queries live sources and records its findings.
The summarizer converts those findings into a customer-facing answer.
from crewai import Crew, Process
# Sequential: the researcher runs first, the summarizer reads its output.
result = Crew(
agents=[researcher, summarizer],
tasks=[research_task, summarize_task],
process=Process.sequential,
).kickoff()
# The researcher's raw findings: every search, dead end, and retry.
self.append_agent_result(
"researcher", result.tasks_output[0].raw, visibility="private"
)
# The summarizer's two or three sentences.
# This is the only part the next turn reads.
self.append_assistant_message(result.raw)Both outputs come from the same crew run but serve different consumers.
append_agent_result(..., visibility="private")writes a structured event tostate.eventsand a thread understate.agent_threads.append_assistant_message()puts the concise reply in the canonical conversation history.
Deferred trace finalization
Conversational flows can keep one trace batch open across turns. The resulting trace contains each routing decision, handler, tool call, and reply in session order.
from crewai import Flow
from crewai.experimental.conversational import ConversationConfig
@ConversationConfig(
llm=CONVERSATION_LLM,
router=router_config,
# Hold one trace batch open across turns instead of closing per kickoff.
defer_trace_finalization=True,
)
class SupportFlow(Flow[SupportState]):
conversational = True
...
flow = SupportFlow()
try:
for message in conversation:
# Nested crews append to the open batch rather than closing it
# So their tool calls stay inside the session.
reply = flow.handle_turn(message, session_id=session_id)
finally:
flow.finalize_session_traces()defer_trace_finalization defaults to True, so the application must close the batch when the session ends. The finally block guarantees that explicit loop exits still export the trace.
Conversational flows do not turn a chat into one endless graph run. They preserve the session while resetting and executing the graph once per message.
End-to-end support flow
The complete turn lifecycle now looks like this:
The turn opens with a message and session ID.
Per-turn execution tracking resets, so the graph runs instead of replaying cached outputs.
The session restores conversation history and persisted domain fields.
The router reads the new message and selects a route.
The handler records private work and produces a visible reply.
The reply enters conversation history while the session trace remains open.
Model and service dependencies
The order service wraps an SQLite database behind an order-management API. Return policies live in documents queried through retrieval.
The flow uses two models for two different workloads. A small routing model classifies short messages, while the conversation model handles the agent work.
from crewai import LLM
ROUTER_LLM = LLM(model="gpt-4o-mini")
CONVERSATION_LLM = LLM(model="gpt-4o")This keeps classification on the cheaper model and reserves the larger model for routes that need it.
from crewai import Flow
from crewai.experimental.conversational import ConversationConfig
from crewai.flow import listen
# Models, router config, services, and crews are set up above.
@ConversationConfig(llm=CONVERSATION_LLM, router=router_config)
class SupportFlow(Flow[SupportState]):
conversational = True
@listen("ORDER_LOOKUP")
def handle_order_lookup(self) -> str:
"""Order status, tracking, or delivery date."""
msg = self.state.current_user_message
order_id = extract_order_id(msg) or self.state.last_order_id
self.state.last_order_id = order_id
self.append_assistant_message(reply)
return reply
@listen("RETURN_POLICY")
def handle_return_policy(self) -> str:
"""Returns, refunds, and damaged items."""
reply = return_agent.kickoff(self.state.current_user_message).raw
self.append_assistant_message(reply)
return reply
@listen("RESEARCH")
def handle_research(self) -> str:
"""Live carrier delays and outside conditions."""
question = self.state.current_user_message
result = research_crew.kickoff(inputs={"question": question})
findings = result.tasks_output[0].raw
self.append_agent_result("researcher", findings, visibility="private")
self.append_assistant_message(result.raw)
return reply(Full code is provided below)
Exposing the flow through an API
The earlier loop runs in a terminal. The AG-UI endpoint exposes the same flow to a chat frontend by enabling conversational mode.
from ag_ui_crewai.endpoint import add_crewai_flow_fastapi_endpoint
from fastapi import FastAPI
from support_flow import SupportFlow
app = FastAPI(title="Support Flow Agent Server")
flow = SupportFlow()
add_crewai_flow_fastapi_endpoint(
app=app,
flow=flow,
path="/conversation",
conversational=True,
)Running a multi-turn conversation
Each message below exercises a different part of the flow. Every handle_turn() call creates a separate graph run under the same session ID.
conversation = [
"Where is order 4471?", # plain lookup, no agent
"Has that arrived yet?", # no order number in the text
"Can I return it if it shows up damaged?", # needs the policy for that category
"Is Bluedart running late this week?", # needs live external information
"Sorry, what was the order number again?", # already answered on turn one
]
session_id = str(uuid4())
flow = SupportFlow()
try:
for message in conversation:
# A chat UI makes these calls across HTTP requests instead of loop
reply = flow.handle_turn(message, session_id=session_id)
print(f"customer: {message}")
print(f" route: {flow.state.last_intent}")
print(f" agent: {reply}\n")
finally:
flow.finalize_session_traces()
Production properties
Together, these pieces keep each graph run independent without breaking the conversation around it:
Multi-turn continuity. Conversation data survives across runs, while the graph’s completed-method record resets before each turn.
Controlled execution. A session containing fifteen clarifications and two research questions can run the research path twice rather than seventeen times.
Clean context. Later turns receive the visible conversation instead of the research log behind it. History still grows, but intermediate tool activity no longer accelerates that growth.
Session-level debugging. One trace follows routing, execution, and replies across turns, matching the interaction the customer experienced.
An ordinary flow decides what one run should execute. A conversational flow also identifies the session, restores the right history, resets execution progress, and preserves the result for the next turn.
CrewAI handles that lifecycle. The application still decides where sessions live, what each handler does, how long history can grow, and when an inactive session should close.
Good day!