I Almost Shipped a RAG Assistant That Lied About APIs That Don't Exist

I wrote this on X a few weeks ago: I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win. Ask my RAG assistant to verify an Interswitch webhook signature, and it didn't say "not in my knowledge base." It wrote a full authentication flow — real-looking endpoint, real-looking headers — and cited a source URL. The URL wasn't in my corpus. It wasn't anywhere. The model invented a citation for content it also invented, with zero hedging. I'm building StacksNG for the Africa Deep Tech Challenge 2026 — an offline coding assistant scoped to the African fintech stack: Paystack, Flutterwave, Monnify, Termii. Before I submitted, I ran a 20-prompt adversarial batch against my own pipeline. Category A (in-corpus baseline) and D (phrasing brittleness) came back clean. Category B — five prompts asking about payment providers I deliberately never scraped into the corpus, Kuda, PalmPay, Interswitch, Paga, OPay — did not. Three of five ignored a system prompt that already said, in plain language, "if the context doesn't contain enough information, say so." That's the failure mode that zeroes out half the score in a hackathon where accuracy is 50% of the total. My first theory was wrong, and I could prove it My instinct was: this is a retrieval-confidence problem. Set a similarity threshold, refuse to answer below it, done. I checked the actual numbers before writing that fix. Top-1 similarity What happened Correct in-corpus answer 0.718 correct Worst fabrication (Interswitch) 0.712 fully invented, fake citation Correct decline (out-of-domain topic) 0.691 "not in my knowledge base" The worst hallucination had higher retrieval similarity than the cleanest correct decline. There's no threshold that lets the good case