What are Agents?

For example, ChatGPT started off as a product that answered questions in a chat window. It now searches the web, runs the code it writes and reads the files you upload.

All of this runs on a large language model (LLM). An LLM can only output tokens, so it can't search the web or run code by itself.

So the engineers who build these products face three problems:

1. The model can't call your servers. It can only write the call.
2. A task takes many steps, and each step depends on the result of the last.
3. The model decides those steps, and you have no control over the model.

In this post, we will learn what an agent is, and how it turns a model's tokens into actions. By the end of it, you will be able to tell a single LLM call from an agent, and name the four parts of one.

1. The model on its own

Let's start with a single LLM call and give it a real task. Say you run a travel app, and a user types "Book my flight from Bengaluru to Mumbai."

Your server forwards the user's request to the model. The model is a neural network trained to predict the next token of an input sequence, and tokens are all it produces. So who books the flight?

The model asks for details. The user replies: tomorrow, 5 to 10k, no hops.

Now the model names an airline's website and tells the user to book there. The user says "Book it."

The model answers "I cannot perform actions on your behalf."

So the user got some advice, and the flight isn't booked.

This is a fundamental limitation of the model. It can only output tokens, and it can't take actions.

2. The model with tools

The model still outputs only tokens. This time some of them are a tool call.

To the model, a tool call looks like a function call. The model writes the name of a function and its arguments.

The model can't run it. The server does.

The user sends the same request. This time the model replies with a tool call, get_details.

The server shows the user an HTML form. The user fills in the date, the budget and a seat preference.

The server passes the form to the model. The model replies with a second tool call, search_flights.

The server calls the airline's server and gets JSON back. It forwards the JSON to the model and asks what to do next.

The model reports that flight 123 is available. The user says "Book it", and the model calls book_flight.

The first time, "Book it" got a refusal. This time it gets a tool call.

The model made three tool calls, each chosen after the result of the last. This is the agentic loop.

A system that uses a language model to plan and make decisions, while using tools to execute commands, is called an agent.

A tool is a capability an agent can invoke to act on the world outside the model.

The four parts of an agent

Your travel app is an agent now. An agent has four parts, and your server is the first.

1. Control is the agent code. You write it, and it runs the application.
2. The brain is the LLM. You reach it through an API and have no control over it.
3. Tools perform the actions. You can define them, but most are external systems.

The fourth part is memory.

Memory is the record of an agent's own past decisions, tool calls, results and feedback.

If a user asked for a window seat three months ago, the agent can default to one.

Of the four parts, the agent's brain is the one you have no control over. The smartness of the agent depends completely on it.

What does the brain decide?

You don't control the brain, but you can know what it decides. The brain is the model, and it decides three things about a tool.

1. When a tool should be used.
2. Which tool should be used.
3. What arguments are passed to the tool.

Go back to the booking. The model called get_details first, and search_flights only after it had the form. Those were the first two decisions: when, and which tool.

The arguments are the third decision. Take a currency tool with a base and a target.

A user flies from Bengaluru to Austria, so the model fills in INR and the euro. For Switzerland it fills in CHF.

That basic intelligence is expected of the model.

One response from the model can carry two things. The first is text to show the user, such as "I will book the flight for you".

The second thing in the response is a tool call. It has an id, the tool's name and the input for the tool.

The agent code runs the tool and sends the result back with that id.

So, who books the flight?

Your server does. The model's last tool call is book_flight, and the server runs it.

The model did the same thing in both attempts. It produced tokens.

On its own, the tokens were advice. With tools, they were three tool calls, and the server ran each one.

The main difference between a simple LLM call and an agent is that an agent can perform actions.

An agent isn't a single call to an LLM, it's multiple calls.

Agents are not a big thing. They are simple LLMs with some wrappers and some scaffolding.

Summary

1. A large language model predicts the next token. Tokens are all it produces.
2. Asked to book a flight, a model alone gives advice and then refuses to act.
3. To the model, a tool call looks like a function call. The model writes it, and the server runs it.
4. With tools, the same booking takes three tool calls: get_details, search_flights and book_flight.
5. A system that uses a language model to plan and make decisions, while using tools to execute commands, is called an agent.
6. A tool is a capability an agent can invoke to act on the world outside the model.
7. An agent has four parts. The brain is the LLM, and you don't control it.
8. The model decides when to use a tool, which tool, and what arguments to pass.
9. A response can carry text for the user and a tool call. The tool call has an id, and the result goes back with that id.

Sources

1. Intro to Agentic Systems, the slides of the session.
2. Function calling, OpenAI API documentation.
3. Define tools, Claude API documentation.