How to find per-process I/O statistics on Linux:

Need to monitor Linux server performance? Try these built-in command and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:

1. Finding out bottlenecks.
2. Disk (storage) bottlenecks.
3. CPU and memory bottlenecks.
4. Network bottlenecks.

Newer Linux kernels have per-process I/O accounting and you can use the iotop tool to find out what’s performing I/O, but in many cases I’m trying to find the source of an I/O problem in an older kernel. I found sort of a hack-ish way to do that today, while trying to figure out why a system was basically unresponsive.

I found a post on Stack Overflow that showed a way you can get per process I/O statistics from the kernel even in older kernels. I adapted this to my needs, and wrote a little script.

Here’s how you use it. First, get it:

wget http://aspersa.googlecode.com/svn/trunk/iodump

Then turn on kernel messages about I/O:
echo 1 > /proc/sys/vm/block_dump

This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into my script:

while true; do sleep 1; dmesg -c; done | perl iodump

Wait a little while, then cancel the script. The results should look something like the following:

[root@manoj]# while true; do sleep 1; dmesg -c; done | perl iodump

Please wait for some time to get the stats from the dmesg and after 4 to 5 minute, please press Ctrl+c to abort the recording and below stats will come to screen.

# Caught SIGINT.
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
kjournald              442       2849          0       2849          0 dm-0
nfsd                 29036       2731        217       2514          0 dm-0
nfsd                 29041       2495        111       2384          0 dm-0
nfsd                 29037       2120         74       2046          0 dm-0
nfsd                 29039       1996         94       1902          0 dm-0
nfsd                 29038       1380        110       1270          0 dm-0
nfsd                 29040        727        109        618          0 dm-0
pdflush              29423        234          0        234          0 dm-0
nfsd                 29042         89         89          0          0 dm-0
nfsd                 29035         87         87          0          0 dm-0
bash                 19067          2          2          0          0 dm-0
perl                 19066          2          2          0          0 dm-0
bash                 19068          1          1          0          0 dm-0
httpd                16476          1          1          0          0 dm-0

I deliberately generated a bunch of I/O by deleting my Firefox history and cache. If i see the above stats, it shows that NFSD (NFS Deamon) is doing lot’s of I/O and write operation is very -2 high.

Why not write a single script to do everything required, something like:

We can use below shell script to get the same stats, first it will turn on kernel messages about I/O after stats it will turn off kernel messages about I/O, as it is not good practice to keep kernel messages about I/O on.

#!/bin/sh
dmesg -c
/etc/init.d/klogd stop
echo 1 > /proc/sys/vm/block_dump

# allow 30 seconds of stats to be logged
sleep 30

dmesg -c | perl iodump

echo 0 > /proc/sys/vm/block_dump
/etc/init.d/klogd start

Thanks
Manoj

This entry was posted on Tuesday, July 13th, 2010 at 8:49 am and is filed under CentOS, Linux, Monitoring Tools. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.