AI Observability: Logs, Prompts, Tool Calls, And Cost

Here's a five-line function. It calls an LLM, logs the answer, returns it. async function ask ( question : string ) { const res = await openai . responses . create ({ model : " o4-mini " , input : question }); console . log ( " answer: " , res . output_text ); return res . output_text ; } This compiles. It passes tests. It ships. And it will quietly cost you four figures a month before anyone notices, because nothing in that log tells you the model burned 8,000 hidden reasoning tokens to produce a 40-token reply. That's the gap this article is about. AI calls are not regular HTTP calls. The interesting state isn't the response body - it's the messages you sent, the tools the model picked, the tokens it consumed (visible and otherwise), and the dollars that drained out of the budget. If your observability story is "we log the answer," you're flying a plane with one gauge and that gauge is the altimeter. Let's talk about what to actually capture. The four signals that matter Every AI system has the same four dimensions worth instrumenting, and most teams only track one or two of them: Logs - the request/response pair, the error, the latency. The boring stuff that traditional APM already covers. Prompts - the actual text that went in and the actual text that came out. Including system prompts, tool definitions, and history. Tool calls - which tool the model picked, with what arguments, what came back, in what order, with what retries. Cost - input tokens, output tokens, cached tokens, reasoning tokens, model, and the per-million-token price for each. Multiplied per user, per feature, per request. Lose any one of these and you're working blind on a different axis of the problem. Lose the cost signal and you wake up to a Slack message from finance. Lose the tool-call signal and you can't tell why your agent kept booking the wrong flight. Lose the prompt signal and a prod regression becomes a guessing game. Lose plain logs and you don't even know the call happened. The go