How Andrew Ng's OpenWorker Handles Memory
what happens when a session ends? Does the agent remember anything when you start a new conversation?
How Andrew Ng's OpenWorker Handles Memory
In my previous post, I explained how OpenWorker uses a separate context window for exploration.
But what happens when a session ends? Does the agent remember anything when you start a new conversation?
OpenWorker actually has a persistent memory system, and I found a few interesting design decisions in its implementation.
First, where does it store memories?
OpenWorker uses a local SQLite database (coworker.db) to store memories.
Each memory contains information such as its content, a short summary, a scope, and an identifier.
There are two main scopes:
Global: Information that should be available across workspaces, such as user preferences.
Workspace: Information relevant to a particular project or repository.
For example, suppose you tell the agent:
"Whenever you prepare reports for me, keep the executive summary short."
That could become a global memory.
But something specific to the project you're currently working on would be stored as workspace memory.
Now, an important detail: OpenWorker doesn't simply save every conversation as memory.
How does the agent decide what to remember?
OpenWorker exposes four memory tools to the agent:
remember→ Save a new memory.memory_update→ Correct an existing memory.memory_forget→ Delete a memory.memory_read→ Retrieve a memory by its ID.
The agent is instructed to save durable information, such as explicit user preferences, corrections and project context that cannot easily be recovered from the repository.
It is also instructed to avoid saving temporary details, near-duplicates or information already available in the codebase, and to ask permission before saving sensitive information.
So the decision to remember something is largely delegated to the model, while the actual storage is handled by the harness.
One thing worth mentioning here is that these are instructions given to the model, not deterministic checks that guarantee every saved memory is useful.
OpenWorker also allows users to disable memory saving. This prevents new memories from being written, but doesn't delete or hide previously saved memories.
How are memories retrieved?
This is the part I found more interesting.
When the agent engine is initialized, OpenWorker loads global memories and the memories associated with the current workspace.
These are inserted into the agent's system instructions.
But what happens when the user has accumulated hundreds of memories?
OpenWorker has a threshold of 8,000 characters for the fully rendered memory block.
Below that threshold, all applicable memories are included in full.
Once the threshold is exceeded, it switches to an index-based approach:
The 10 newest memories (ordered by memory ID) remain available in full.
Older memories are represented by one-line summaries.
The agent can call
memory_readto retrieve the complete content of a specific memory.
The model still has an index of the older memories, without needing their complete contents in its context window.
And unlike a typical vector-search-based memory system, the implementation doesn't require embeddings or semantic similarity search. It uses scoped memory listings and ID-based retrieval.
An important point:
Memories are loaded when the agent engine is initialized, not continuously refreshed in the system instructions during the conversation.
For example, if a memory is updated or deleted during a session, that change is immediately persisted in SQLite.
However, the memory content already injected into the system instructions isn't automatically rewritten.
The memory_read tool can still retrieve the latest version directly from the database, and a newly initialized agent engine will load the updated memories.
So there is a difference between what the model initially received as context and what is currently stored in the memory database.
What I take from this
OpenWorker separates three things that are often grouped together:
Conversation history keeps track of what happened during a session.
Context compaction summarizes older conversation turns when the context becomes too large.
Persistent memory stores selected information that should remain available across sessions.
They solve different problems.
OpenWorker also keeps explicit user rules separate from learned memories. User-defined rules are treated as higher-priority instructions when they conflict with stored memories.
And OpenWorker's design shows that a useful memory system doesn't necessarily need a vector database or a complicated retrieval pipeline.
The more important decisions are what gets remembered, how it is scoped, and when it becomes available to the agent.
