Memory That Can Show Its Work

Memory That Can Show Its Work

How Busy Family Remembers

Overview

In February we wrote about how Busy Family remembers. The design was simple: three narrative files per family, loaded into the system prompt when a conversation started. It worked, and it taught us what an assistant needs from memory once families depend on it every day.

This post describes what we built next. Memory is now a collection of small markdown files, one per claim, in a shape borrowed from Google's Open Knowledge Format. Each claim says what it is, how strongly it should be obeyed, when it stops being true, and which conversation it came from. A nightly lifecycle keeps the collection current. The agent starts every conversation with a compact brief, receives the memories closest to each message, and can ask for more when it needs it. Families can see, correct, and remove any single claim, and trace it back to the words that produced it.

The rest of this post covers why we changed, the principles behind the new design, the file format, how memories form and stay true over time, how the agent reads them, and how families stay in control.

1. What the First Design Taught Us

The three-file design was easy to build, easy to reason about, and a real improvement over an assistant that forgot everything between sessions. Living with it showed us where the shape of the data was working against us.

The memory was rewritten as a whole. Each summarizer merged new facts into a running narrative. Every merge was a chance for detail to blur. Two facts about the same person could fuse into one sentence, and a fact could slip out when the narrative was reworded. Nothing in the file said where a statement came from, so there was no way to check it.

There was no unit to act on. If one line in a family's knowledge was wrong, the choices were to leave it or to overwrite the file. Edit replaced the whole narrative. Forget emptied it. Families wanted to correct one thing, and the design only offered all or nothing.

Everything loaded every time. The agent received the full narrative at the start of each conversation, whatever the family was about to ask. Across a sample of 27 families, the combined files were about 1.2k tokens at the median and about 5.2k at the largest. Memory that grows without a bound has to compete with the rest of the prompt for the model's attention.

Time wore facts down. A sentence such as "Monday is a holiday" or "pizza night is tonight" made sense on the day it was written and read as a puzzle a week later. Narrative prose has no place to record when a fact stops being true.

Different kinds of memory blurred together. A standing instruction like "always call before booking" sat in the same paragraph as trivia about a favorite restaurant. The model had to guess which sentences were rules and which were colour.

Every surface had its own copy of the problem. The calendar look-ahead, web search, and shared household chat each read the same files and each had to decide what was safe to use. Household chat in particular had to be careful not to repeat the last private conversation in a shared thread.

We could not measure it. There were no numbers for how often a stored fact was right, how often the assistant followed a standing instruction, or how much of what a family had told us made it into memory. We built an evaluation harness for the new design, and it gave us the first baseline for the old one.

These observations point the same way. Memory should be made of small, checkable pieces that know their own age, their own source, and their own weight. The agent should carry a small summary and be able to find the rest.

2. Design Principles

Six principles guided the new design.

One claim per file. The smallest thing we can remember is the smallest thing we can edit, forget, expire, or cite.

Every claim has a source. A memory records the conversation turns it came from. The quote is checked against the transcript when the memory forms, so a claim that cannot be traced does not become a memory.

Files are the truth; the index is a projection. Memories live in S3 as markdown. The search index is built from them and can be rebuilt from them. We never have to ask which copy is right.

Carry little, find the rest. The agent starts with a bounded brief and discloses more as the conversation needs it. The size of the prompt does not grow with the size of the memory.

The agent reads; it does not write. In chat, the assistant can look things up and cannot create or delete memories. Memory changes come from the formation pipeline, the lifecycle pass, and the family's own actions.

Fix forward. We archived the old files as a safety net and built the new system to be correct on its own terms, not to lean on a rollback.

3. OKF: The Memory File

Where the format comes from

In June 2026, Google Cloud published the Open Knowledge Format, an open specification for representing knowledge for AI agents. It formalizes a pattern many teams had arrived at independently: a directory of markdown files, each carrying a small block of YAML front matter. Only one field, type, is required. Everything else is optional, and producers are free to add their own keys. The specification describes index.md files for progressive disclosure, per-claim provenance, trust tiers, and lifecycle fields such as an absolute stale_after instant.

That is the shape we wanted. Markdown is readable by people and models alike. Front matter is readable by machines without parsing prose. Because the format is plain files, S3 versioning, diffs, and ordinary tools all work on it.

We adopted OKF's conventions and extended them with the fields a family memory needs. A memory file is OKF-shaped, with domain keys of our own, and the sections below say how our fields map to the questions OKF cares about.

One memory, one file

This is a memory in the new store. The family and the people in it are invented for this post.

---
id: 01K6N1R0A0B0C0D0E0F0G0H010
type: directive
about: Soccer team group chat
summary: Ask before sending anything to the soccer team group chat.
when_to_use: Any request to message, post to, or reply in the soccer group chat
applies_to:
  - 01K6N1R0A0B0C0D0E0F0ENT002
strength: hard
pin: false
authority: user_explicit
confidence: 1.0
durability: durable
formed_at: 2026-08-14T19:02:00Z
last_reinforced_at: 2026-10-02T23:40:00Z
reinforcement_count: 3
valid_until: null
status: active
supersedes: null
superseded_by: null
related: []
sensitivity: normal
revision: 1
needs_enrichment: false
---

Always show the draft and wait for a yes before anything goes to the soccer
team group chat. This applies to replies as well as new posts.

What each group of fields is for

What kind of memory is this? type is one of identity, directive, preference, fact, constraint, or relationship. about names the subject. applies_to points at the people, places, or groups involved, so the system can tell that two memories concern the same person.

How strongly should the agent follow it? strength separates hard rules from soft leanings. pin marks a memory that must always appear in the brief.

Who said it, and how sure are we? authority records whether the family stated it explicitly, said it in passing, we inferred it from behaviour, or the system created it. confidence is the extractor's confidence. A directive can only be hard or explicit if a turn written by the user supports it.

When does it stop being true? valid_until is an absolute date, never a phrase like "next week". durability says whether the fact is lasting, seasonal, or passing. status runs through active, dormant, expired, superseded, and retracted. A memory is never deleted when it stops being current. It changes state, and its history stays readable.

What replaced it? supersedes and superseded_by link an old version of a claim to the new one, so a correction produces a lineage and not a silent overwrite.

Who should see it surfaced? sensitivity tags topics such as health, finances, or minors. Sensitive memories stay out of the always-on brief and are available when a task calls for them.

How do we change it safely? revision lets concurrent writers detect that a file changed under them.

Evidence lives next to the claim

Each memory has a small evidence ledger beside it, written as one JSON record per line:

{"ref":"conv://{family}/{session}#turn=12-14","quote":"Please check with me before anything goes to the team chat","recordedAt":"2026-08-14T19:02:00Z"}

The conv:// reference names a session and a range of turns. Keeping the ledger in a sidecar keeps the memory file small and lets each new reinforcement append a line without rewriting the claim.

The same document, two places

A memory exists once as a file in S3, and its front matter is embedded in the search index alongside a 512-dimension embedding of the memory's subject, summary, and when-to-use text. Search can filter on status, valid_until, type, and applies_to and can match on meaning as well as on words. If the index is ever lost, we rebuild it from the files, and the rebuilt index contains the same documents.

What the format makes possible

A shape this small and explicit does a lot of work for us:

• Edit, forget, expire, and cite can each operate on one claim.
• Search can exclude anything that is no longer current without reading its text.
• The files are readable and diffable by people, with S3 versioning behind them.
• Both services share a written contract, with golden fixtures that each side's tests parse, so the agent and the pipeline cannot drift apart.
• New fields can be added without breaking readers, since the format tolerates keys it does not know.

4. How a Memory Forms

The old pipeline fanned a finished session out to three summarizers, each rewriting its own file. The new pipeline has two stages with a clear handoff, and the lifecycle in section 5 keeps the result current.

Extraction. When a session ends, an extractor reads the transcript with each turn numbered and a clock stamped in, so that relative dates are written as absolute ones. It proposes candidate memories, each with a type, a summary, a body, a when-to-use line, a durability, an expiry date when one applies, and the turns that support it.

Candidates are then checked by code, not by judgment. The cited turn must exist. A fact about the user must cite a turn the user wrote. The supporting quote must appear in that turn. Anything that fails is dropped. Hard rules and explicit instructions must cite a user-authored turn, and nothing that came from a tool's output can become a hard rule. There is also a cap on how many new candidates one session can contribute, with a higher bar for inferences than for statements.

Reconciliation. Each candidate is compared with what the family already has. The reconciler searches for similar memories, checks the family's suppression rules, and chooses one of: add a new memory, reinforce an existing one, supersede an older version, record a contradiction, or do nothing. A statement the family made explicitly outranks one we inferred. Writes are applied under a short per-family lease with revision checks, so formation, the lifecycle pass, and a family's own edits never overwrite one another. Replaying the same session produces no further changes.

5. Dreaming: How Memory Stays True Over Time

A store that only grows is not memory, it is an archive. Facts expire, habits fade, repeated mentions deserve more weight, and a prompt that quotes last season's plans misleads the model. Something has to look after the collection as time passes. We call that process dreaming, since it does its work in the background and not in the conversation.

What dreaming does

A dream pass is deterministic. It does not invent facts and does not rewrite a family's words. It changes the state of a memory and recompiles the brief.

• Expire. A memory whose valid_until date has arrived moves to expired. A fact marked valid until the 17th is retired when the 17th begins, so it never lingers into the day it no longer applies.
• Promote. A passing fact that the family keeps confirming, three reinforcements by default, becomes lasting. Pinned memories and explicit instructions are left as they are.
• Decay. An inferred memory that has not been reinforced for about six months becomes dormant. Something we guessed from behaviour should not persist forever without fresh evidence.
• Recompile. After the sweep, the pass rebuilds the family's brief from the memories that are currently active, and writes a dated change log.

Dates are the family's own. Expiry is judged against the family's local calendar date, not the clock of the data centre running the pass, so a fact about "tonight" does not retire at five in the afternoon Pacific.

What triggers it

Dreaming runs for one family at a time, in three situations:

• When memory changes. After the reconciler applies at least one change from a session, or after a family adds a memory from a card, that family's brief is recompiled. A session that changes nothing does not cause a dream.
• Every night. A scheduled pass walks every family and sweeps all of their memories, including ones nobody touched that day. This is what lets expiry and decay happen for quiet families, whose memories would not otherwise be revisited.- After a migration. When a family's history is replayed, a dream follows so the brief reflects the result.

Recompiling the brief in the same place that changes the memories means the brief is always a function of the store. The agent can trust that what it is handed was compiled from the current, active memories.

Lifecycle, not deletion

Every transition keeps the memory. An expired memory is still on disk and still part of the family's history. It is excluded from search by default and from the brief, and it can be read through the lineage and evidence links. When a correction arrives, the old memory is marked superseded and points to the new one.

6. The Brief: What the Agent Starts With

Every conversation begins with a brief: a compact, precompiled summary of what matters most about a family. It is built by the dream pass and stored as a single file, brief.json.

A small, fixed budget

The brief has a hard cap of 2,000 tokens and fixed allocations for each section:

Section	What it holds
Identity	Who is in the household and how to address them
Directives	Standing instructions and explicit memories, hard rules first
Topic index	A map of what the family has told us about, with counts, so the agent knows what it can look up
Last conversation	A short summary of the most recent session

Each line has a character limit, and the compiler orders lines by weight: pinned first, then explicit instructions, then statements, then inferences. When the budget runs out, the lowest-weight lines are the ones left out. Because the brief is bounded, a family with fifty memories and a family with five thousand cost the same to carry.

Every line ends with a reference to the memory it summarizes. We come back to that in section 7.

Compiled the same way every time

The brief contains no timestamps and no counters, and its contents are ordered deterministically, so the same memories compile to the same bytes. That matters for prompt caching. The agent's prompt is assembled as a stable instruction block, then the family's brief, then the parts that change from turn to turn. Because the brief only changes when memory changes, the model provider can reuse the work it did on earlier calls in the same conversation, including the tool-use loops that make several calls for one message.

Always current

The agent reads brief.json on every request with a conditional fetch, so a request pays for a download only when the file has changed. Two small mechanical adjustments keep it honest between compiles:

• Lines whose expiry date has passed are filtered when the brief is loaded.
• When a family forgets or edits a memory, the matching line is removed or replaced at once.

A family that forgets something in the memory card therefore does not see it in the next reply.

Safe by construction

Sensitive memories do not appear in the brief. The brief is presented to the model as data, not as instructions, and every line is delimited and length-capped.

7. Progressive Disclosure and References

The brief is the table of contents, not the book. We arrange memory in three tiers, and the agent moves down them only as far as the conversation requires.

1. The brief. Always present, small, and built for the questions families ask most.
2. Turn-level retrieval. For each message, the system looks up the memories closest to what the user just said and places a few short lines in the turn.
3. Tools. When the agent needs the full text of a memory, or the conversation it came from, it asks.

The tiers are connected by references. A memory has an address of the form memory://{family}/{memoryID}, and a conversation excerpt has an address of the form conv://{family}/{session}#turn=12-14. A brief line carries the reference of the memory it summarizes. A memory's evidence carries the reference of the turns that support it. The agent can follow the chain from a one-line summary to the full memory and on to the exact words the family used.

The references are for the agent's use. The system prompt tells the model to open a reference when it needs the detail, and never to show one in a reply. Families see ordinary sentences.

8. Turn-Level Retrieval

The brief holds what is most likely to matter. A family's next message might be about something that is not in it, such as a coach's name, last spring's camp, or which pharmacy they use. For that, every message triggers a lookup before the model starts to answer.

Two searches, run together. One search matches on meaning, using the embedding of the user's message. The other matches on words, with tolerance for small spelling differences. A message about "the dentist" finds a memory that says "orthodontist," and a message that names an unusual word finds the memory that uses it.

Merged and ranked. The results are interleaved: the top few matches by meaning lead, keyword matches follow, and the rest of the semantic list fills in behind them, with duplicates removed. Only memories that are active and unexpired are eligible.

Capped on purpose. The lookup contributes at most a small number of items and a fixed token budget. It places each memory's summary, its age, and its reference in the turn, and labels the section as candidates, so the model knows these are lookups to weigh and not facts it must repeat.

Retrieval does not expand a memory into the memories related to it. The index only returns current memories, so a hit is already the current version of that fact. When the agent wants more context, it can open the memory and ask again, which keeps each step small and the prompt predictable.

9. The Tools: search_memory, read_memory, and read_conversation_excerpt

Three read-only tools give the agent the third tier.

search_memory runs the same search the turn-level lookup uses, on a query the model writes. It returns short lines with references. The system prompt tells the model that when neither the brief nor the retrieved lines state a fact, it should search before telling the family that it does not know.

read_memory takes a memory:// reference and returns the whole memory, front matter included. It is the step from the summary to the full body: the exact condition under which a rule applies, the context behind a preference, the date a fact was recorded.

read_conversation_excerpt takes a conv:// reference and returns the cited turns. This is how the agent, or a family asking "why do you think that?", can reach the words the memory came from.

These tools stay available when the agent is running a saved skill, so a procedure does not lose the ability to check what the family has said. In shared household chat, where several people read the same thread, the tools are not offered, since memories belong to individuals.

There is no tool for writing or deleting a memory. A conversation can inform memory through the formation pipeline, and a family can change memory through the card. The assistant cannot change either on its own, which keeps a mistaken or manipulated reply from rewriting what a family has told us.

10. User Control and Trust

Remembering things about a family carries a responsibility to show what is remembered, make it easy to change, and be careful with it. These capabilities come from the claim-per-file design.

Seeing what we know

When a family asks to see what the assistant has learned, it shows a memory card in chat on the web and in the native app. The card lists memories one claim at a time, with the newest first and more available as the family scrolls. Each item shows the summary, what it concerns, and how long it has been remembered.

Correcting one claim

Edit does not overwrite. The family's new wording becomes a new memory marked as an explicit statement, and the old memory becomes superseded and points to it. The next reply uses the new text, and the history of the correction stays on file.

Forgetting a claim for good

Forget retires the memory and records a suppression rule. The rule matters because the next conversation may mention the same fact again, and without it formation would simply learn it back. Formation checks the family's suppression rules before it writes anything, so a forgotten fact stays forgotten until the family says otherwise.

A family can also forget a group, such as everything about one person or one topic. The card shows how many memories the action will affect and asks for confirmation before it commits.

Showing the source

Every memory can show where it came from. Opening the evidence for an item shows the quote from the conversation that supported it, with its date. A family that wonders why the assistant believes something can see the exact words, and fix it if we misunderstood.

Careful with sensitive topics

Memories about health, finances, and children are tagged sensitive. They stay out of the always-on brief, are reachable by search when a task needs them, and the agent is told to use them only for that task and never to volunteer them in anything others will read, such as a message, a shared calendar entry, or a shared note.

Memory text is data

Stored memory sits in a privileged place in the prompt, so we treat it as untrusted text. It is presented to the model as data and is delimited and escaped, each line has a length limit, and the extractor will only form a hard rule or an explicit instruction when the family's own words support it. Text that arrives through a tool, such as a web page or an email, cannot become a hard rule.

Households keep private things private

In the shared household chat, the assistant draws on the same brief with the "last conversation" section removed. A summary of one person's private chat does not appear in the shared thread.

11. One Brief, Many Surfaces

The same compiled brief serves every place the assistant draws on what it knows about a family.

• Chat. The agent receives the brief on each request and the per-turn lookup alongside it.
• Household chat. Shared threads use the brief without the last-conversation section.
• Web search. When a family asks for recipes, dietary needs recorded in memory shape the query.
• Look-ahead. The daily and weekly look-ahead reads the family's brief once and uses it as background. What is on the calendar takes priority over memory, and memory fills in context around it, such as who handles which activity or which preferences apply.

Because these surfaces share one source, a correction made in the card is reflected in all of them. Where a family has not yet been compiled, each surface falls back to the earlier files, so nothing goes dark during a transition.

12. Moving a Live System

A new design is only useful once the families already using Busy Family have it. We migrated in three steps.

Replay history. We did not start families from empty. Each family's past conversations were run through the same extractor and reconciler that handle new sessions, in chronological order, so a later statement supersedes an earlier one in the right order. Standing instructions from the earlier "explicit memories" file were reconciled as explicit directives, and a report compared that file with the new memories so nothing the family had asked for was lost in the move.

Start with one household. We migrated our own family first and reviewed what came out line by line. The review covered 849 memories: we accepted 717, rejected 118, and rewrote 14 summaries to be clearer. That left us with a collection we had read and trusted, and a reference point to measure the rest of the migration against.

Then everyone else. A resumable script walked each remaining family, one at a time, through enumerate, map, reduce, explicit-memory reconcile, and report, and kept a ledger so a stopped run could continue where it left off. The whole wave took about an hour and a quarter. We copied the earlier files to an archive in each family's folder and left the originals in place, so the earlier summarizers could keep working until we retire them.

We then turned the brief on for every family and moved the look-ahead, web search, and household readers to it.

13. Measuring It

We hold the memory to explicit standards, so "better" is something we can check:

• At least 90 percent of extracted memories are correct and cited, and at most 2 percent are unsupported.
• At least 80 percent of facts we labelled by hand appear in the brief or can be reached through search, and every hard constraint does.
• Duplicate clusters stay at 5 percent or less of active memories.
• Standing instructions are followed at least as reliably as they were with the earlier design.
• The brief stays at or under its 2,000-token cap.

The evaluation harness checks these against labelled examples, including directive compliance (for example, "address me by first name"), constraint recall such as allergies, and retrieval cases where the answer lives only in a memory that is not in the brief. It also tracks how well the agent's prompt cache is used, since the brief's stable byte layout is meant to improve it. We reviewed the first full migration by hand, which taught us as much about the extractor as the automated checks did.

14. Conclusion

The earlier design loaded a family's whole narrative and hoped the model found what mattered. The new one does the reverse. It carries a small, current brief, retrieves the right memory when a message needs it, and keeps the full story one reference away.

The design rests on a few ideas that will outlast any one implementation: memory made of small claims with their own age, weight, and source; a lifecycle that keeps those claims true over time; progressive disclosure so the agent's context stays small however much we remember; and control that lets a family correct or remove any single thing we know.

Because every memory is a plain file with explicit fields, the system has room to grow. New kinds of memory, richer views for families, and new places for the assistant to use what it knows can each be added without redesigning the store. We expect to keep learning how families want the assistant to remember, and the design is built to make each lesson an adjustment and not a rebuild.

If you want to read more, the earlier post, Memory Architecture, describes the first design and the formation pipeline it relied on. The Open Knowledge Format specification is here.