CPU utilization lies: autoscaling a single-threaded service

The service was slow. Not down, just slow: p95 latency climbing well past where users notice, requests piling up, the kind of degradation that generates support tickets instead of alerts. And the autoscaler, the whole point of which is to add capacity when a service is under strain, sat there doing nothing. The metric it was watching said everything was fine. Average CPU utilization on the tasks was hovering around 30 percent, nowhere near the scale-out threshold. The dashboard was calm. The users were not. Both were right, and the gap between them is one of the most common autoscaling traps on a container platform. This is the first article in a series on running a multi-tenant SaaS on AWS at team scale. It is about a metric that lies, quietly, by design. Why 30 percent CPU meant 100 percent busy The service was a single-threaded application. A Node.js API, in this case, but the same is true of any process that does its real work on one thread: a classic Python or Ruby worker, most single-process runtimes. A single-threaded process can, by definition, saturate exactly one CPU core. The task it was running on had four vCPUs. So the arithmetic that matters is brutally simple: one core fully pegged / four vCPUs on the task = ~25% task-average CPU At full saturation, the busiest that process can ever make the task look is about 25 percent. Add a little async I/O overhead spread across the runtime and you land around 30 percent. That is not a service with headroom. That is a service redlining on the only core it can use, while three cores sit idle and drag the average down to a number that reads as "barely working." The autoscaling policy was tracking average CPU across the task's cores. For a workload that can only ever use one of them, that average is not a measure of load. It is a measure of load divided by four. The metric was answering a different question This is the real lesson, and it is not specific to AWS or ECS. Average CPU utilization answers "how much of th