The Energy and Water Cost of Inference

Estimates of the energy in one model query differ by orders of magnitude across credible sources. Most of that spread is not disagreement about physics. It is disagreement about where the system boundary is drawn, and a figure quoted without its boundary is not a figure. Why the published figures disagree Before comparing two numbers, establish which of these each one includes. Any of them can change the answer by more than the model choice does: Which model, and how much output. A short answer from a small model and a long answer from a large reasoning model differ by several orders of magnitude on their own. A single “per query” figure averages over a distribution nobody specifies. Batch size and utilisation. The dominant engineering term. Serving many requests concurrently amortises the weight read across all of them; the same hardware at low occupancy spends nearly the same power for a fraction of the tokens. Facility overhead. Cooling, power conversion and distribution, captured as power usage effectiveness. It multiplies everything, and whether a figure includes it is frequently unstated. Training amortisation. Some analyses divide training energy across expected lifetime queries. Defensible, and it produces a different quantity from marginal serving energy. They are not comparable. Embodied energy. Manufacturing the accelerators, the building and the power infrastructure. Usually excluded, occasionally included, rarely flagged. Idle and provisioned capacity. Capacity is held for peak. Charging queries only for the seconds they compute understates the system; charging them for provisioned capacity overstates the marginal query. Both are used. Building the estimate yourself The marginal serving calculation is not complicated, and doing it once makes every published figure legible. Serving side, per accelerator: E_per_token = (P_device · n_devices · PUE) / R_tokens_per_second P_device average power draw under load, from the spec sheet (below the rated maximum in