Giving a fleet of AI agents one shared memory — when each agent runs a different model

Most agent frameworks give each agent its own context window and call it memory. That works right up until you run more than one agent, and then it quietly becomes the most expensive design decision in the system. We run a fleet where different agents are deliberately backed by different models — one family handles long-form drafting, another handles structured extraction, a couple run on a local path with no external inference at all. Routing by capability is the easy part. The hard part is that an agent which learns something has learned it alone . This is a writeup of what broke, and the design we ended up with. The failure mode The symptom shows up as repeated work. An extraction agent determines that a particular vendor's invoices put the tax line above the subtotal. Useful. Two days later a different agent — different model, different prompt, same pipeline — hits the same vendor and re-derives it from scratch. Then a third does it again. Nothing is wrong . Every agent behaves correctly. The system as a whole just has no way to accumulate anything, because knowledge lives inside whichever context window happened to be open at the time. You are paying inference costs to rediscover facts you already own. The naive fix is to pass more history. That fails for a specific reason worth naming: context windows are per-invocation and per-model. A 200k window on one model does not help an agent running a different model with a 32k window, and neither survives the session ending. You cannot solve a persistence problem with a bigger buffer. What "unified memory" has to mean Once you accept that memory has to live outside the agents, the requirements get concrete: Model-agnostic storage. If memory is stored as one model's embeddings, you have coupled your memory layer to a vendor. Swapping models later means reindexing everything. Written by one agent, readable by all. Otherwise you have per-agent memory again, with extra steps. Attributable. When memory is wrong — and it w