API Key Management for AI Applications

An inference key is a payment instrument with an API. That is the property that makes it different from most credentials you manage: the person who steals it does not need your data to profit, because the key itself buys something they want. What makes an inference key different It is directly monetisable. A stolen database credential needs a buyer for the data. A stolen inference key is resold as capacity within hours, and automated scanners harvest public repositories continuously looking for exactly this. The loss accrues while you sleep. Usage-based billing means the damage is a function of elapsed time and rate limit, not of a single event. This is the argument for hard caps over careful monitoring. It is passed around more than most secrets. Notebooks, evaluation scripts, a colleague’s laptop, a CI job, an agent’s own environment. Every one of those is a copy you do not control. The blast radius is often the whole account. Where a provider offers one key with full access, a leak is total. Where it offers scoped keys, use them — this is the single biggest lever available. Leak paths specific to AI applications Generic advice — do not commit secrets, use a manager — is correct and widely published. These are the paths that only exist because there is a model in the system, and they are the ones that survive a conventional review: The key in the context. A key pasted into a system prompt so a tool “has access to it”. It is now one paraphrase from the transcript, and the transcript is stored. Traces and observability. LLM tracing tools capture full request bodies by default. If a header, a tool argument or an environment dump ends up in a span, your key is in a third-party dashboard with a broader access list than your secret manager. Prompt and response logs. Same problem, your own infrastructure. A logger that prints the request object on error will print the Authorization header. Evaluation datasets. Captured production traffic reused as an eval set, then share