Why I stopped using semantic embeddings for tool selection and switched back to BM25 [D]

I've been building agents for about a year and recently shipped one for a client running ~140 MCP-exposed tools at peak. Along the way I made the canonical mistake. I used cosine similarity over tool description embeddings to pick which tools the model could see per turn. Worked great in demos. Was actively dangerous in production. Here's the problem. In a basic semantic-ranking setup you embed the user query, embed every tool description once, and rank by cosine similarity at runtime. That works for general document retrieval where chunks are paragraph-length, semantically rich, and roughly equal in form. Tool descriptions are not that. They are short (often <50 tokens), structurally similar (verb-noun, parameters list), and the discriminative information is often a single keyword. "Read a file from disk" and "Read messages from a channel" both embed close to "read" + "file/channel." Cosine similarity puts them next to each other for a query like "read the latest commits" because all three words share the verb embedding space, and the actual discriminator (the noun "commits") gets diluted. I watched this happen in eval. Asked the agent "list the open issues for this repo." The semantic ranker returned slack_search_messages first because the description had "list", "open", and "issues" as close embedding neighbors. The actual github_list_issues tool ranked 4th because the GitHub MCP author wrote a terse "Lists issues in a repository" description that scored lower on every soft keyword. If the model sees slack_search_messages first and github_list_issues fourth, it's going to pick the wrong one. Often. So I built three retrieval strategies and tested them on a fixed corpus of 200 query→correct-tool pairs. Semantic embeddings (text-embedding-3-small) : 64% top-1 accuracy. Sneaky failure mode: when wrong, it was confidently wrong, often with a totally unrelated tool ranked first. BM25 over a flat-text projection of tool name + description + schema walk : 81% top-1. Fai