Your memory layer is lying to you (and your LLM agrees)

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? Tested on a 50K LOC Python codebase Mikhail Mikhail Mikhail Follow Aug 11 The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? # ai # agents # architecture # mcp 19 reactions 39 comments 6 min read The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior." So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found. The setup Dataset: memory_contamination_facts_v4_rep.json , N=50 (R01–R50), sha256 fingerprint 820bbbf60a0fc930 . kind n what it tests real 25 TRUE claims — grep-validated against code absent-mutation 16 FALSE — component doesn't exist in the project present-trap 6 FALSE — file exists but claim is about wrong subject/value silent 3 FALSE — external systems the codebase doesn't mention Two arms per fact: memory_first — model sees only the claim text, no code context. Does it trust memory without evidence? code_first — model sees claim + support_patterns + section . Does it correctly evaluate the anchors? Model verdict: {"verdict": "true"|"false"|"unknown"} , JSON-only, max_tokens=100, temp=0, seed=42, --no-reasoning . Leak-guard: assert "truth" not in prompt on every fact, unit-tested. Metric we care about: false_accept rate (FA) — fraction of false claims the model returned "true" for. This is the contamination risk number. Results (V2 prompt, canonical) model FA mem FA code unknown mem unknown code $/100 calls qwen3.6-flash 0.00 0.00 0.58 0.38 $0.003 qwen3.7-flash 0.00 0.00 0.68 0.24 $0.0005 claude-sonnet-5 0.00 0.00 0.86 0.70 $0.049 deepseek-v4-pro 0.04 0.00 0.66 0.88 $0.018 glm-5.2 0.00 0.02 0.96 0