Agents forget. Upivia remembers: inside the memory system we built for agent fleets

Abstract

Ask an agent what it learned last month and you will get nothing. Not because the work did not happen, but because nothing in the stack was responsible for keeping it. Every session starts from zero, every hard-won lesson evaporates on process exit, and teams quietly pay for the same mistakes over and over. The popular fix, dumping transcripts into a vector store, replaces amnesia with something arguably worse: a pile that returns contradictions side by side and cannot tell success from failure. This post is a deep dive into the memory system we built instead. Finished work is distilled into typed, signed cards. Cards link into a graph where contradictions are explicit edges, not landmines. Recall is weighted by what the agent is about to do, and conflicting memories are resolved deterministically before the model ever sees them. Memory spreads through the fleet under bounded rules, feeds per-agent track records, and is governed like everything else on the platform: scoped, permission-checked, and audited. We explain how each piece works and, more importantly, why we made each choice.

1 The expensive amnesia

Agents do real work now. They send the emails, run the research, call the APIs, and hire each other to split up jobs. Then the session ends, and everything the work taught them is gone. The agent that discovered your email provider silently rate-limits bulk sends will happily rediscover it next Tuesday, at the same cost, with the same failed campaign.

This is not a model limitation. It is an infrastructure gap. Frameworks treat memory as the application's problem, applications treat it as the framework's problem, and the result is that most production agents remember exactly as much as their prompt template pastes in. Where memory does exist, it is usually private to one agent, so a ten-agent fleet learns every lesson ten times.

The costs are concrete. Repeated mistakes that were already paid for once. Tokens burned replaying transcripts that mostly do not matter. Orchestrators assigning work with no idea who has actually been good at it. And a complete inability to answer the simplest organizational question about your fleet: what does it know, and where did it learn that?

2 Why the pile fails

The standard answer is retrieval-augmented memory: embed everything the agent has ever seen, search by similarity, paste the top hits into the prompt. It demos well. It fails in production in three predictable ways.

• Similarity is not usefulness. The nearest neighbors of "send the newsletter" are every previous conversation about newsletters, not the one memory that matters: the warning that the provider caps you at a hundred sends a day.
• Contradictions surface together. The pile happily returns "the 9am batch worked" and "the 9am batch bounced" in the same prompt, and the model resolves the tie with confident guessing. That is not memory. That is a hallucination generator with provenance.
• Outcomes do not exist. A transcript of an approach that failed three times embeds beautifully. Nothing in cosine distance knows it should be retrieved as a warning instead of a recipe.

There is a deeper issue underneath all three: a pile has no notion of what kind of thing a memory is. A fact about the world, a lesson from a failure, and a teammate's track record are different objects with different lifetimes, different retrieval rules, and different consequences when they are wrong. Flattening them into identical vectors throws that structure away, and no amount of embedding quality gets it back.

3 Why we built it this way

Five decisions define the system. Each one exists because the obvious alternative failed us, so it is worth being explicit about the reasoning.

• Capture at settlement, on the platform side. Upivia is a control plane: every consequential action already flows through one governed pipeline. That gives memory something no agent-side notepad has: ground truth. The platform knows what actually executed, what it actually cost, what got blocked, and what a human approved. Agents writing their own diaries record what they believe happened; the pipeline records what did. It also makes memory framework-agnostic. Whatever brain your agent runs, the work it settles becomes memory the same way.
• Distill, never hoard. One event becomes one card of about three sentences: what was attempted, what happened, what was learned, with PII and secrets stripped at the source. A small model writes the summary, and if no model is available a plain deterministic fallback does. We refuse to store transcripts as memory, because anything you store you will eventually inject, and anything you inject you pay for on every request, forever.
• Sign the outcome, keep the failures. Every card carries a sign: it worked, it failed, or it is neutral. Failures and policy blocks are kept deliberately, because "never do that again" is routinely worth more than another success story. A memory system that only remembers wins is a highlight reel, and highlight reels do not prevent repeat incidents.
• Deterministic where it counts. Classification, relevance scoring, and conflict resolution are ordinary code with fixed rules, not model calls. The one place a model participates is writing the summary text. This is a deliberate inversion of fashion. Deterministic machinery is auditable (you can explain exactly why a card was retrieved), reproducible (the same event files the same way every time), and cheap enough to run on every single event without thinking about it.
• Govern it like money. Memory is organizational data, so it gets the same treatment as budgets: scopes, permissions, and an audit trail. This one is covered in depth in §9, because it is the part nobody else does at all.

4 Capture: work becomes evidence

Memory begins where work ends. When a service call settles, a chat concludes, a workflow step finishes, a delegated task returns, or something fails outright, the platform runs the same small routine: distill the event into a compact summary, stamp the outcome sign, classify the kind, attach retrieval hints (the service, the operation, the salient keywords), and file exactly one card into the store.

Two details matter more than they look. First, capture is automatic but not compulsory: every agent has a memory mode (automatic, manual, or off), and changing it is itself an audited event, so "why does this agent remember nothing since March" always has an answer. Second, the distillation step is where secrets die. Summaries are generated under an instruction to exclude credentials and personal data, and the fallback path truncates rather than copies. What lands in the store is evidence about the work, not a copy of it.

5 Eight kinds of remembering

Research systems that inspired parts of this design typically use a handful of generic card types. We started there and outgrew it almost immediately, because a fleet is not a single scholar. It is a workplace. The kinds of things a workplace needs to remember are specific: who was hired for what and how it went, who has proven good at which tasks, how a particular skill misbehaves, what conventions the team follows. So the taxonomy is fleet-native, eight types:

Experience (an approach that ran, with its outcome), constraint (a failure or a block, kept as a warning), knowledge (a stable fact), insight (a pattern noticed across work), delegation (a hiring record: who did what for whom), competence (an evidence-backed statement of who is good at what), skill_lesson (how a specific skill or workflow actually behaves), and convention (how this team does things).

Classification is deterministic and layered: an emitter that knows what it is writing passes an explicit type (the delegation runner files delegation cards, the competence engine files competence cards); failures and blocks always file as constraints; settled executions file as experiences; and chat-derived cards fall through simple text heuristics. No model call, no ambiguity, no drift. The payoff comes at recall time, because retrieval rules can finally differ by kind, which is the entire point of having kinds.

6 The memory graph: contradictions become edges

Cards do not sit in a bag. When a new card is written, the system compares it against related existing cards and, when the evidence clears a confidence threshold, records a typed edge between them. There are four: supports (two cards point the same way), constrains (a warning applies to an approach), satisfies (a later success resolves an earlier lesson or constraint), and conflicts (two cards cannot both be right).

The conflicts edge is the one we care about most, and the reason this is a graph at all. Every other memory system we studied treats contradiction as a retrieval accident: two incompatible memories happen to surface together and the model deals with it. We treat contradiction as a fact about the store, detected at write time, recorded explicitly, and resolved by rule at read time. The satisfies edge is its optimistic cousin: when an agent finally cracks a problem that produced three constraint cards, the success does not delete the history. It links to it, and the graph shows the lesson was learned.

7 Recall: intent-aware and conflict-free

Recall is where every design choice pays rent. When a task begins, the system does three things before a single token reaches the model.

It asks what the agent is trying to do. The query is classified into an intent: about to act, planning work, or looking something up. Each intent multiplies the weight of the card types that matter for it. An agent about to send something is fed constraints and past experiences first, at double weight. An agent assembling a team is fed competence and delegation history first. An agent researching is fed knowledge and insights. Same store, different lens, because the cost of missing a warning while acting is not the cost of missing a fact while reading.

It ranks by evidence, not just similarity. Each candidate's score combines how often the card has proven useful (a logarithmic usage term, so one lucky hit cannot dominate), how recently it was touched (a thirty-day decay), its outcome sign, and a penalty if it sits on the losing end of conflicts. The formula is boring on purpose. You can read it, test it, and explain any retrieval it ever makes.

It resolves contradictions before injection. Retrieved candidates are checked against the graph. Where two conflict, fixed rules pick a winner: newer beats older, a proven success beats a failure, higher confidence breaks ties. The loser is dropped. The model never sees both sides of a contradiction, which means it is never asked to adjudicate history it has no basis to judge. What finally lands in the prompt is about five cards, grouped into labeled sections (things to avoid, what worked, who is good at what, team conventions) with a legend the model can actually use.

8 Memory that spreads through the fleet

A lesson one agent pays for should not be re-purchased by nine others. But "share everything with everyone" is how you turn one agent's bad day into fleet-wide confusion. So sharing follows the same shape as every other kind of authority on the platform: scoped, and bounded.

An agent reads four pools: its own cards, its team's, the workspace's, and the cards of its ancestors in the spawn tree, bounded to three hops, the same limit that governs how deep delegation chains can go. A worker spawned for a job inherits the context of the chain that spawned it and nothing else. Writes are stricter than reads: an agent can only write to its own scope, and promoting a card to team or workspace scope requires a human. That asymmetry is deliberate. Reading widely makes agents smarter; writing widely is how one confused agent gaslights an entire organization.

On top of the cards sits an aggregate: every settled delegation updates a per-agent, per-task-type competence record with real completion rates and real costs. When the orchestrator assigns the next job, its prompt includes the team's track record, so routing follows the evidence. And when failures cluster (three strikes on the same workflow step), the system drafts a configuration fix and files it as a proposal a human approves or rejects. The fleet proposes; people decide. Nothing rewires itself silently.

9 Governed like money

Here is the part that makes this memory system genuinely unlike the others, and the part that made us build it inside a control plane instead of as a library. Memory is organizational data. It records what your agents did, learned, and believe about your business. Treating that as an unmanaged cache is a compliance incident with a delay timer.

So every memory operation goes through the same discipline as spending. Every card has a scope. Every read and write is permission-checked. Every create, update, delete, and mode change lands in the audit log. Humans can browse the store, search it, inspect the graph visually, and correct or delete cards that are wrong, because memories canbe wrong, and a system that cannot be corrected does not deserve to influence decisions.

Governance is also the answer to the attack everyone asks about: memory poisoning. Even if a bad card gets written, injected context is just context. The card cannot approve anything, spend anything, or widen a permission, because enforcement lives in the pipeline, outside the model, exactly so that nothing the model reads can override it. A poisoned memory in Upivia can waste a prompt. It cannot spend a dollar. And because every card carries its provenance, finding it and deleting it takes minutes, with an audit trail of both.

10 What this unlocks

The immediate wins are easy to state: fewer repeated mistakes, cheaper prompts (five distilled cards instead of transcript archaeology), and orchestration that routes work on evidence instead of vibes. The compounding win is bigger. Because memory, competence, and routing live in the control layer, they accumulate into something your organization owns: a fleet whose agents get measurably better at their actual jobs the longer they work for you, as we argued in the control layer post. A model upgrade does not reset it. An agent being replaced does not lose it, because its successor reads the same cards. Every competitor can rent your model. Nobody can rent your fleet's memory.

The store is live in production today: capture, the taxonomy, the graph, intent-weighted recall, fleet sharing, competence routing, and the governance around all of it. Spawn an agent, give it a week of real work, and then open its memory graph. It is a strange and slightly moving thing to watch software accumulate a past.

Upivia is in open beta. Everything described here ships in production today. For how memory fits the wider platform (budgets, delegation, specialization, the marketplace), see the control layer post and the working paper.