We had a production incident where a Next.js application became painfully slow — long-running requests, ECONNRESET errors, and timeouts. We raised the CPU request from 100m to 500m, rolled out fresh pods, and the application recovered.
The confusing part: those pods never came close to their CPU limit. The limit was 500m, usage sat right around the 100m request, and by the usual dashboard logic they looked fine.
Note what we changed, though. The limit was 500m before the fix and 500m after it. We only moved the request. If the quota ceiling had been the binding constraint, that change should have done nothing — and the application recovered anyway.
That's about as close to a controlled experiment as you get during an incident, and it points at a mechanism most "CPU throttling" write-ups skip entirely.
Two Different CPU Mechanisms, Constantly Confused
When people say "CPU throttling," they almost always mean the CFS quota enforced by the limit. But requests and limits control two independent mechanisms, and only one of them was relevant here.
This was our manifest going into the incident:
resources:
requests:
cpu: 100m # entitlement — cpu.shares / cpu.weight
limits:
cpu: 500m # ceiling — CFS quota per 100ms period
The request is something else entirely. It converts into a scheduler weight — 100m becomes roughly 102 shares, 500m becomes 512. On an idle node, that weight is irrelevant and your container can happily burn whole cores. The moment the node is CPU-contended, that weight decides how much CPU time you actually win against every other pod competing for the same cores.
Why "Usage ≈ Request" Is a Red Flag, Not a Green Light
This is the part I want anyone reading to take away.
We looked at the dashboards and saw CPU usage hovering right around the 100m request, nowhere near the limit, and initially read that as "CPU is fine, look elsewhere."
That reading is backwards. Under a contended node, share-based scheduling rations a pod to approximately its entitlement. So a container that wants 400m but only requests 100m will show usage pinned near 100m — not because that's all it needed, but because that's all it was allowed to win.
Usage well below request → genuinely idle, healthy
Usage spiky, far below limit → bursty, check short-window throttling
Usage flat, pinned at request → suspicious: demand may be capped by entitlement
A flat line at exactly your request value under load is the signature of a starved container. Real demand doesn't politely plateau at a round number you chose in a YAML file.
There's a circularity here that's easy to miss. CPU usage is an outcome, not a measure of demand. It records what the kernel permitted the container to consume, not what the application was trying to do. So using low usage to rule out a CPU constraint is reasoning in a circle: the constraint is precisely what produced the low number you're treating as reassurance.
Averages hide real throttling too
A second reason usage-versus-limit is a weak signal: CFS evaluates quota every 100ms, but kubectl top and rate(...[5m]) average over minutes. A container can exhaust its quota in dozens of individual 100ms periods — freezing for 60-80ms at a time — while its five-minute average reads a comfortable 20%.
Our 500m limit is worth doing the arithmetic on. It grants 50ms of CPU per 100ms period. Node.js runs your application logic on a single thread, and one thread can consume up to 100ms per period. So any stretch of sustained synchronous work — a large JSON serialisation, a template render — exhausts the entire quota at 50% of one core and gets frozen for the remainder of the window. Add V8's parallel GC threads and libuv's four-thread pool on top and the ceiling is easier to touch than it looks.
So "average usage is far below the limit" never proves the absence of throttling. Only the throttling counters do, and we'll come back to what we could and couldn't prove.
What Actually Happened
The trigger was downstream, not in Next.js at all. The distributed trace proved the dependency chain:
Next.js app
↓
/api/submission/submit
↓
Rails app
↓
/api/webhook/receive
↓
PostgreSQL
One trace showed nextjs app waiting roughly 107 seconds for rails app. Inside rails app, that same request spent 82.47 seconds in the webhook controller — of which 81.35 seconds was a single PostgreSQL EventSubmission/find.
So there was severe real slowness in the Rails and PostgreSQL path. That alone explains slow requests. What it doesn't explain is why Next.js itself became unhealthy rather than just sitting patiently waiting on I/O.
That's where the CPU entitlement came in:
Rails/PostgreSQL becomes slow
↓
Next.js requests wait 60-100s instead of ~200ms
↓
In-flight requests accumulate — hundreds of open sockets,
timers, promise chains, buffered response objects
↓
Next.js CPU demand rises sharply: event loop scheduling,
JSON, Mongo/Redis clients, GC over a growing heap, TLS
↓
But entitlement is 100m on a contended node,
so the event loop only wins ~100m of CPU
↓
Event loop falls behind → sockets time out → ECONNRESET
↓
Retries arrive, backlog grows, demand rises further
↓
Death spiral
Waiting on a slow backend is cheap. Waiting on a slow backend while managing hundreds of accumulated connections is not — and that second-order CPU cost is what our 100m request could not cover.
Why Raising the Request Fixed It
Here's the change, in full:
# before # after
resources: resources:
requests: requests:
cpu: 100m cpu: 500m # 5x scheduler weight
limits: limits:
cpu: 500m cpu: 500m # unchanged
Raising the request from 100m to 500m multiplied our scheduler weight roughly 5× — from about 102 shares to 512. On a contended node that translates directly into winning roughly five times as much CPU time against neighbouring pods.
The quota ceiling never moved. Since the application recovered anyway, the ceiling cannot have been what was relieved. Whatever else was going on, the binding constraint was entitlement.
There's a bonus effect worth knowing about. With request now equal to limit on the only constrained resource, those pods moved from Burstable into the Guaranteed QoS class — which also makes them the last to be evicted under node pressure rather than a mid-priority target.
Two honest caveats:
The rollout did two things at once. New pods got more CPU entitlement, but they also reset every accumulated connection and all in-flight work. A restart alone can break a death spiral like this one. We can't fully separate those two effects after the fact.
Raising only the request does not remove CFS throttling. If your problem genuinely is the quota ceiling, the request is the wrong knob — you need to raise or remove the limit. The request was the right knob for us precisely because the ceiling was never the constraint. Getting this backwards is easy, and it's why naming the mechanism matters more than finding a setting that happens to work.
What I'd Check First Next Time
Start with the basics:
kubectl top pod -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o yaml
Then look at these signals together, because any one of them in isolation will mislead you:
| Signal | What it tells you |
|---|---|
| CPU usage vs request | Flat at the request under load = likely starvation |
| CPU usage vs limit | How much headroom before the ceiling |
| Throttled seconds / periods | Whether CFS quota throttling is actually occurring |
| Node CPU overcommit | Sum of all pod requests vs node allocatable |
| Scheduler / run-queue delay | Starvation that throttling metrics will never show |
| Application request latency | Whether any of it reaches users |
For quota throttling, in Prometheus:
rate(container_cpu_cfs_throttled_seconds_total[5m])
Compare against actual usage:
rate(container_cpu_usage_seconds_total[5m])
For starvation — the thing that bit us — throttling metrics stay at zero. Look instead at node contention and at CPU pressure stall information inside the container:
# PSI: "some" tells you how long tasks were runnable but waiting for CPU
kubectl exec -it <pod> -n <namespace> -- cat /proc/pressure/cpu
# cgroup v2 throttling counters, straight from the container
kubectl exec -it <pod> -n <namespace> -- cat /sys/fs/cgroup/cpu.stat
In cpu.stat, nr_throttled and throttled_usec climbing means quota throttling. Those counters flat at zero while PSI some avg10 is high means starvation, and you should be reaching for the request, not the limit.
The Distinction That Matters for an RCA
High CPU usage is not CPU throttling. And CPU starvation is not CPU throttling either.
Three different things get reported as "the pod was throttled":
- Quota throttling — you hit the limit, kernel freezes you. Visible in
container_cpu_cfs_throttled_seconds_total. - Share starvation — the node is contended and your request is too low to win timeslices. Invisible in throttling metrics entirely.
- Just plain busy — high utilisation with neither of the above.
Ours was the second. It is the one most likely to be misdiagnosed, because every dashboard and every blog post trains you to look at usage-versus-limit, and that comparison tells you nothing about it.
I should be precise about our own evidence here: we inferred throttling from application behaviour and from the fact that more CPU fixed it. We never read a throttling counter. That inference turned out to name the wrong mechanism, and the reason it went unchallenged for so long is that "the app is CPU-constrained" and "the app is being throttled" felt like the same sentence. They aren't.
Could quota throttling have been happening as well, in short bursts the averages hid? Given a single-threaded runtime under a 500m ceiling, quite plausibly. But we can't claim it, because to conclusively write "CFS throttling caused the outage" you need historical throttling counters from the affected containers during the window, and we didn't retain them. So the strongest defensible statement is this:
CPU entitlement on the Next.js workload was too low for the load it was absorbing, and the Rails/PostgreSQL downstream path experienced severe latency that created that load. Raising the CPU request resolved the entitlement problem without changing the limit. Confirming whether CFS quota throttling also occurred would require container-level throttling metrics we did not retain.
Takeaway
"It recovered after we changed X" is correlation, not root cause. The gap in our evidence wasn't analysis — it was that the metrics needed to close the question didn't exist once the pods were gone.
So the real action items from this one:
- Scrape and retain
container_cpu_cfs_throttled_seconds_total, and alert on it. - Track CPU usage against request, not just against limit. A flat line at the request value under load deserves a page.
- Watch node-level CPU overcommit, since share starvation only exists when nodes are contended.
- Remember that fixing the downstream dependency was still the actual fix. More CPU let Next.js survive a slow backend; it didn't make PostgreSQL fast.
If you've debugged similar latency issues in Kubernetes, or you've got a good approach for catching starvation before users do, I'd be glad to hear it.