Load average is 40 on an 8-core box, but CPU usage is 15%. What's going on?
The scenario
A database replica became slow. top shows high load but mostly idle CPU.
Model answer
Linux load average counts processes that are runnable and processes in uninterruptible sleep (state D), which usually means waiting on disk or network storage.
- top or vmstat 1: high wa (I/O wait) and many processes in D state.
- iostat -x 1: device utilisation, await and queue size show which disk is saturated.
- iotop or pidstat -d: which process is doing the I/O.
- Common causes: a slow EBS volume or burst credits running out, an NFS mount hanging, swap thrashing, or a backup job.
Common mistakes
- Assuming load average means CPU
- Not checking I/O wait