How do I evaluate an LLM app when there is no right answer?
In this blog we will see how to evaluate an LLM app with no single right answer, from the expert to the LLM judge.
When a customer asks your support bot "Where is my order?", how do you know its answer was right?
Ask the bot again and it answers in different words. Both answers can be right. A test that expects one exact string fails.
That puzzles engineers too. In our AI engineering cohorts, they asked about evals 60 times, and in 18 of those the first answer didn't land.
In this blog we will see how to evaluate an LLM app with no single right answer, from the expert to the LLM judge.
1. Is an eval a unit test?
It is close. If you call evals testing, you aren't very wrong.
A lot of people say evals are just unit testing, but evals is an overloaded term. It also checks for behavior.
We can also give it subjective scores, like relevance. Was it relevant? Was it not relevant?
A unit test asks whether the bot's code works. A behaviour check asks whether its answer was relevant to the customer.
An eval does both. It is a test of an AI system's behaviour, run against a fixed set of inputs whose correct answers are known or judged.
2. Is the eval score the same as the loss?
No. Loss says how close the model is to the data it was trained on. An eval score says how it does on data you set aside.
Score 100 out of 100 on the textbook quizzes and 95 on the final exam. The number that matters is 95.
The final exam is marked against an answer key. For an eval, that key is called the ground truth.
3. I have no ground truth. Where do I start?
Your support bot has no answer key. So you start with a person who knows what a good answer is.
Now we forward it to a subject matter expert. So, for example, if you're building a healthcare app and this app is going to face customers, then you probably want a nurse or a doctor or a healthcare practitioner to look into this.
They're not going to read code. They're going to create 20 to 50 user queries, commonly asked user queries or important user queries.
For the support bot, the expert is someone who has answered customers for years. Writing 20 to 50 queries takes them about a day.
Then you run the queries through the bot. The expert marks each answer pass or fail, like an exam scorecard.
The expert marks an answer by its meaning, not its words. Two answers to "Where is my order?" that give the same delivery date both pass.
4. An expert can't mark every release. What then?
Your bot changes every week. Can the expert mark 50 answers for every release?
They can't. So an LLM marks them, after the expert has marked them once.
Have the judge mark the same queries, and compare its marks with the expert's. Every disagreement is a false pass or a false fail.
Then you fix the judge's prompt until the two agree. Nothing is trained here. You are editing a prompt.
5. What does the judge see?
Does the judge need to know how your bot works? No. It sees the bot's answer and its own prompt, never the bot's prompt.
It's like I created an agent. It has a system prompt S1. I created a judge. It has a system prompt S2. S1 is not going to be passed to the judge.
The judge is going to have only S2, and it's going to have the output O1.
So the judge doesn't know how the answer was produced. It marks only what the customer would read.
Some checks need more than the answer. A retrieval check also gets the query and the retrieved chunks, and an agent check gets the tool calls.
6. Should the judge score 1 to 5, or pass and fail?
Pass and fail. A score from 1 to 5 sounds richer, but the judge won't use the whole range.
At the end of the day, you want the answer to be relevant, not highly relevant or low relevance, and LLMs suck at giving scores.
They are never going to give a 5 out of 5. They're going to give you a three or four.
So most marks come back as 3 or 4, and the average tells you little. A pass or a fail tells you which answers to look at.
That works best when each judge is asked one question. "Is it relevant?" and "Is it grounded?" go to two different judges.
7. Which model should be the judge?
A model from a different family, where you can. A model is lenient on answers that read like its own.
The LLM, which is DeepSeek, is going to look at DeepSeek's response and say, "Wow, this is a great answer." Because the format, the way in which it answers, the language is going to be similar.
So the same family of models prefers itself.
That leniency nearly cost one team a bug. Claude wrote C++ code with a use-after-free bug, and a Claude review missed it.
If you are tied to one provider, the leniency stays. So let the larger model judge the smaller one.
8. An eval failed. What do I fix?
When the judge fails the bot's answer to "Where is my order?", it doesn't say why. First find the part of the bot that failed.
| Part you control | Metric that points at it |
|---|---|
| Retrieval | Are the fetched chunks relevant to the query? |
| Generation | Is the answer grounded in those chunks? |
| Tools | Was the right tool called with the right arguments? |
Good chunks with a poor answer means the LLM is the culprit. Poor chunks means retrieval is.
Each failure then belongs to one part. Group the failures by part, and fix the biggest group first. A score of 70% on a first version is good.
After a fix, run every query again. A prompt change that fixes one case can break others.
Summary
- An eval tests behaviour on a fixed set of inputs, with answers that are known or judged.
- Loss measures fit to training data. An eval score measures performance on a held-out set.
- The first eval set is 20 to 50 queries marked by a human expert.
- An LLM judge is aligned to the expert by editing its prompt, and it never sees the app's prompt.
- Judges give pass or fail on one question each.
- A judge from another model family is less lenient.
- A failed eval is traced to one part of the system through the metric for that part.
