Logo
All posts

This is how Lyft reduced agent hallucinations by 20%.

In this blog we will see how Lyft routes, builds and evaluates the AI agents that answer its support requests.

Gaurav Sen/7 min read

In May 2026, Lyft wrote that the hallucination and contradiction rates of its support agents had fallen by 20%.[1] Lyft completed 945.5 million rides in 2025, about 2.6 million a day.[2]

Engineers building AI agents for customer support face three problems:

  1. A complaint has to reach the team that can resolve it.
  2. Building one agent takes months.
  3. Nobody can tell whether an agent's answers are right.

In this blog we will see how Lyft routes, builds and evaluates the AI agents that answer its support requests.

1. A meta agent routes the complaint

Who decides which team gets a complaint? Before LLMs, the user did, and didn't always pick right.

Many of these rides have issues which are raised to customer support. Its job is to quickly and accurately resolve customer complaints. These complaints can be broken into multiple types.

Earlier, a user would need to choose the category and type of their complaint to correctly route their complaint to the right team. But now with the use of LLMs, Lyft has a meta agent which dynamically routes complaints to the right team, reducing user effort.

Support, and four kinds of complaint that reach it. Damages is the first kind on the video's list.
Support, and four kinds of complaint that reach it. Damages is the first kind on the video's list.

Lyft runs one meta agent for riders and another for drivers. A driver's complaint is sorted into earnings, account access or damage claims.

A driver with a damage claim reaches the damage claim agent. A subagent can also hand the driver back to its parent, mid-conversation.

2. An agent built from a configuration

What does the damage claim agent know? Only what its team gives it. It doesn't learn the rest on its own.

Each team has its own agent. It's specialized in managing its kind of complaints. The agent has a system prompt, access to relevant documents, and a configuration. That helps it manage the particular type of complaint effectively.

One agent and the three parts the video lists under it: a config, documents and a system prompt.
One agent and the three parts the video lists under it: a config, documents and a system prompt.

The configuration is JSON, kept in an internal config service, and a domain expert writes the prompt. No engineer changes code to launch an agent.

That configuration is what made agents fast to build. Lyft's first driver agent took about six months, and a new configurable agent takes about two weeks.

3. Memory and tracing

How does an agent remember the earlier turns of a conversation? It stores them.

Each agent also has memory, meaning the conversation of every agent is stored in durable storage. In the case of Lyft, that is DynamoDB.

The agents also have tracing, where every incoming request is logged along with the agent's actions and the result of that request.

What it stores is used twice. A trace records which nodes ran, what the LLM saw and which tools were called. Engineers replay the stored conversations to debug an agent in production.

4. An LLM judges the answer

Who says whether an agent's answer was good? A trace shows only what the agent did.

To evaluate the agent, the engineers at Lyft used an LLM as a judge evaluation pipeline. Sample traces in production are sent to an LLM. It judges the responses based on groundedness, answer relevancy and domain specific metrics.

Evals result in measurable performance, and only if you can measure something can you really improve it.

An eval as the video draws it. A test case goes to the agent, and a second LLM judges the output: Pass or Fail.
An eval as the video draws it. A test case goes to the agent, and a second LLM judges the output: Pass or Fail.

A new agent first serves 5% to 10% of production traffic, and its sampled traces go to the judge. Lyft's advice is to make the judge answer Pass or Fail, instead of giving a score.

5. The system prompt matters most

Which part of an agent did the judge's scores improve most? It wasn't the documents or the tools.

In practice, the Lyft engineers realized that the prompt quality, which is basically the system prompt for the LLM, matters most for good agent performance.

So amongst the possibilities of system prompt, document retrieval and tool usage, improving the system prompt provided the most bang for the buck.

A line runs from the developer's gear to the words System Prompt. The prompt is the part Lyft found hardest.
A line runs from the developer's gear to the words System Prompt. The prompt is the part Lyft found hardest.

Domain experts' prompts handled the usual case and fell apart on edge cases. One told the agent to be empathetic, without saying what that means.

So Lyft wrote a prompt framework with five required parts. Scope is one of them: what the agent handles, and what it must route elsewhere.

This structured approach reduced the time to develop agents from six months to just two weeks, and reduced the chance of hallucinations by 20% when answering user queries.

Summary

  1. A meta agent routes each complaint to the team's agent that resolves it.
  2. Each agent has a system prompt, documents and a configuration. A new agent takes two weeks instead of six months.
  3. Every conversation is stored in DynamoDB, and every request is traced.
  4. An LLM judges sampled production traces on groundedness, answer relevancy and domain metrics.
  5. Improving the system prompt gave the most bang for the buck.
  6. Hallucination and contradiction rates fell by 20%.

Sources

  1. How Lyft Built a Self-Serve AI Agent Platform for Customer Support with LangGraph and LangSmith, Akshay Sharma, LangChain blog, 27 May 2026.
  2. Lyft Reports Record Q4 and Full-Year 2025 Results, Lyft, 10 Feb 2026.
  3. Testing Agent Skills Systematically with Evals, OpenAI Developers.
WhatsApp