Linux monitor tool

Tool          Description                      Base      Repository
vmstat      all purpose performance tool              yes      yes
mpstat      provides statistics per CPU              no      yes
sar          all purpose performance monitoring tool      no      yes
iostat      provides disk statistics              no      yes
netstat      provides network statistics              yes      yes
dstat          monitoring statistics aggregator          no      in most distributions
iptraf      traffic monitoring dashboard              no      yes
netperf      Network bandwidth tool              no      In some distributions
ethtool      reports on Ethernet interface configuration      yes      yes
iperf          Network bandwidth tool              no      yes
tcptrace      Packet analysis tool                  no      yes

CPU

内核调度器将负责调度2种资源种类:线程(单一或者多路)和中断.调度器去定义不同资源的不同优先权.以下列表从优先级高到低排列:

Interrupts - 设备通知内核,他们完成一次数据处理的过程.例子,当一块网卡设备递送网络数据包或者一块硬件提供了一次IO 请求.

Kernel(System) Processes(内核处理过程) - 所有内核处理过程就是控制优先级别.

User Processes(译注:用户进程) - 这块涉及”userland”.所有软件程序都运行在这个user space.这块在内核调度机制中处于低优先级.

上下文切换

多数现代处理器都能够运行一个进程(单一线程)或者线程.多路超线程处理器有能力运行多个线程.然而,Linux 内核还是把每个处理器核心的双核心芯片作为独立的处理器.比如,以Linux 内核的系统在一个双核心处理器上,是报告显示为两个独立的处理器.

一个标准的Linux 内核可以运行50 至 50,000 的处理线程.在只有一个CPU时,内核将调度并均衡每个进程线程.每个线程都分配一个在处理器中被开销的时间额度.一个线程要么就是获得时间额度或已抢先获得一些具有较高优先级(比如硬件中断),其中较高优先级的线程将从区域重新放置回处理器的队列中.这种线程的转换关系就是我们提到的上下文切换.

每次内核的上下文切换,资源被用于关闭在CPU寄存器中的线程和放置在队列中.系统中越多的上下文切换,在处理器的调度管理下,内核将得到更多的工作.

运行队列

每个CPU 都维护一个线程的运行队列.理论上,调度器应该不断的运行和执行线程.进程线程不是在sleep 状态中(阻塞中和等待IO中)或就是在可运行状态中.如果CPU 子系统处于高负荷下,那就意味着内核调度将无法及时响应系统请求.导致结果,可运行状态进程拥塞在运行队列里.当运行队列越来越巨大,进程线程将花费更多的时间获取被执行.

CPU 利用率

比较流行的术语就是”load”,它提供当前运行队列的详细状态.系统 load 就是指在CPU 队列中有多少数目的线程,以及其中当前有多少进程线程数目被执行的组合.如果一个双核系统执行了2个线程,还有4个在运行队列中,则 load 应该为 6. top 这个程序里显示的load averages 是指1,5,15 分钟以内的load 情况.

CPU 利用率就是定义CPU 使用的百分比.评估系统最重要的一个度量方式就是CPU 的利用率.多数性能监控工具关于CPU 利用率的分类有以下几种:

User Time(译注:用户进程时间) - 关于在user space中被执行进程在CPU 开销时间百分比.

System Time(译注:内核线程以及中断时间) - 关于在kernel space中线程和中断在CPU 开销时间百分比.

Wait IO(译注:IO 请求等待时间) - 所有进程线程被阻塞等待完成一次IO 请求所占CPU 开销idle的时间百分比.

Idle(译注:空闲) - 一个完整空闲状态的进程在CPU 处理器中开销的时间百分比.

CPU 性能监控

Run Queues -  每个处理器应该运行队列不超过1-3 个线程.例子,一个双核处理器应该运行队列不要超过6 个线程.

CPU Utiliation - 如果一个CPU 被充分使用,利用率分类之间均衡的比例应该是
65% - 70% User Time
30% - 35% System Time
0% - 5%   Idle Time

Context Switches - 上下文切换的数目直接关系到CPU 的使用率,如果CPU 利用率保持在上述均衡状态时,大量的上下文切换是正常的.

很多Linux 上的工具可以得到这些状态值,首先就是 vmstat 和 top 这2个工具.

vmstat 工具的使用

vmstat 工具提供了一种低开销的系统性能观察方式.因为 vmstat 本身就是低开销工具,在非常高负荷的服务器上,你需要查看并监控系统的健康情况,在控制窗口还是能够使用vmstat 输出结果.这个工具运行在2种模式下:average 和 sample 模式.sample 模式通过指定间隔时间测量状态值.这个模式对于理解在持续负荷下的性能表现,很有帮助.下面就是

vmstat 运行1秒间隔的示例:

# vmstat 1
procs ———–memory———- —swap– —–io—- –system– —-cpu—-
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
0  0 104300  16800  95328  72200    0    0     5    26    7    14  4  1 95  0
0  0 104300  16800  95328  72200    0    0     0    24 1021    64  1  1 98  0
0  0 104300  16800  95328  72200    0    0     0     0 1009    59  1  1 98  0

r         The amount of threads in the run queue. These are threads that are runnable, but the CPU is not available to execute them.
当前运行队列中线程的数目.代表线程处于可运行状态,但CPU 还未能执行.
b          This is the number of processes blocked and waiting on IO requests to finish.
当前进程阻塞并等待IO 请求完成的数目
in          This is the number of interrupts being processed.
当前中断被处理的数目
cs          This is the number of context switches currently happening on the system.
当前kernel system中,发生上下文切换的数目
us          This is the percentage of user CPU utilization.
CPU 利用率的百分比
sys          This is the percentage of kernel and interrupts utilization.
内核和中断利用率的百分比
wa         This is the percentage of idle processor time due to the fact that ALL runnable threads are blocked waiting on IO.
所有可运行状态线程被阻塞在等待IO 请求的百分比
id          This is the percentage of time that the CPU is completely idle.
CPU 空闲时间的百分比

sysstat packages

http://sebastien.godard.pagesperso-orange.fr/download.html

Following are the other sysstat utilities.

• sar collects and displays ALL system activities statistics.
• sadc stands for “system activity data collector”. This is the sar backend tool that does the data collection.
• sa1 stores system activities in binary data file. sa1 depends on sadc for this purpose. sa1 runs from cron.
• sa2 creates daily summary of the collected statistics. sa2 runs from cron.
• sadf can generate sar report in CSV, XML, and various other formats. Use this to integrate sar data with other tools.
• iostat generates CPU, I/O statistics
• mpstat displays CPU statistics.
• pidstat reports statistics based on the process id (PID)
• nfsiostat displays NFS I/O statistics.
• cifsiostat generates CIFS statistics.

1. CPU Usage of ALL CPUs (sar -u)

This gives the cumulative real-time CPU usage of all CPUs. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on the last field “%idle” to see the cpu load.

$ sar -u 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:27:32 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle01:27:33 PM       all      0.00      0.00      0.00      0.00      0.00    100.0001:27:34 PM       all      0.25      0.00      0.25      0.00      0.00     99.5001:27:35 PM       all      0.75      0.00      0.25      0.00      0.00     99.00Average:          all      0.33      0.00      0.17      0.00      0.00     99.50

Following are few variations:

• sar -u Displays CPU usage for the current day that was collected until that point.
• sar -u 1 3 Displays real time CPU usage every 1 second for 3 times.
• sar -u ALL Same as “sar -u” but displays additional fields.
• sar -u ALL 1 3 Same as “sar -u 1 3″ but displays additional fields.
• sar -u -f /var/log/sa/sa10 Displays CPU usage for the 10day of the month from the sa10 file.

2. CPU Usage of Individual CPU or Core (sar -P)

If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.

“-P ALL” indicates that it should displays statistics for ALL the individual Cores.

In the following example under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.

$ sar -P ALL 1 1Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:34:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle01:34:13 PM       all     11.69      0.00      4.71      0.69      0.00     82.9001:34:13 PM         0     35.00      0.00      6.00      0.00      0.00     59.0001:34:13 PM         1     22.00      0.00      5.00      0.00      0.00     73.0001:34:13 PM         2      3.00      0.00      1.00      0.00      0.00     96.0001:34:13 PM         3      0.00      0.00      0.00      0.00      0.00    100.00

“-P 1″ indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).

$ sar -P 1 1 1Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:36:25 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle01:36:26 PM         1      8.08      0.00      2.02      1.01      0.00     88.89

Following are few variations:

• sar -P ALL Displays CPU usage broken down by all cores for the current day.
• sar -P ALL 1 3 Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).
• sar -P 1 Displays CPU usage for core number 1 for the current day.
• sar -P 1 1 3 Displays real time CPU usage for core number 1, every 1 second for 3 times.
• sar -P ALL -f /var/log/sa/sa10 Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.

3. Memory Free and Used (sar -r)

This reports the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

$ sar -r 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:28:06 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact07:28:07 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     8820407:28:08 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     8820407:28:09 AM   6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204Average:      6209248   2097432     25.25    189024   1796544    141372      0.85   1921060     88204

Following are few variations:

• sar -r
• sar -r 1 3
• sar -r -f /var/log/sa/sa10

4. Swap Space Used (sar -S)

This reports the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

$ sar -S 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

07:31:06 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad07:31:07 AM   8385920         0      0.00         0      0.0007:31:08 AM   8385920         0      0.00         0      0.0007:31:09 AM   8385920         0      0.00         0      0.00Average:      8385920         0      0.00         0      0.00

Following are few variations:

• sar -S
• sar -S 1 3
• sar -S -f /var/log/sa/sa10

Notes:

• Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.
• Use “sar -H” to identify the hugepages (in KB) that are used and available.
• Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
• Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

5. Overall I/O Activities (sar -b)

This reports I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

• tps – Transactions per second (this includes both read and write)
• rtps – Read transactions per second
• wtps – Write transactions per second
• bread/s – Bytes read per second
• bwrtn/s – Bytes written per second

$ sar -b 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:56:28 PM       tps      rtps      wtps   bread/s   bwrtn/s01:56:29 PM    346.00    264.00     82.00   2208.00    768.0001:56:30 PM    100.00     36.00     64.00    304.00    816.0001:56:31 PM    282.83     32.32    250.51    258.59   2537.37Average:       242.81    111.04    131.77    925.75   1369.90

Following are few variations:

• sar -b
• sar -b 1 3
• sar -b -f /var/log/sa/sa10

Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

6. Individual Block Device I/O Activities (sar -d)

To identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition), use “sar -d”

$ sar -d 1 1Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:59:45 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util01:59:46 PM    dev8-0      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.1001:59:46 PM    dev8-1      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.1001:59:46 PM dev120-64      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.6201:59:46 PM dev120-65      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.6201:59:46 PM  dev120-0      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.3001:59:46 PM  dev120-1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.3001:59:46 PM dev120-96      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.9101:59:46 PM dev120-97      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91

In the above example “DEV” indicates the specific block device.

For example: “dev53-1″ means a block device with 53 as major number, and 1 as minor number.

The device name (DEV column) can display the actual device name (for example: sda, sda1, sdb1 etc.,), if you use the -p option (pretty print) as shown below.

$ sar -p -d 1 1Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:59:45 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util01:59:46 PM       sda      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.1001:59:46 PM      sda1      1.01      0.00      0.00      0.00      0.00      4.00      1.00      0.1001:59:46 PM      sdb1      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.6201:59:46 PM      sdc1      3.03     64.65      0.00     21.33      0.03      9.33      5.33      1.6201:59:46 PM      sde1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.3001:59:46 PM      sdf1      8.08      0.00    105.05     13.00      0.00      0.38      0.38      0.3001:59:46 PM      sda2      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.9101:59:46 PM      sdb2      1.01      8.08      0.00      8.00      0.01      9.00      9.00      0.91

Following are few variations:

• sar -d
• sar -d 1 3
• sar -d -f /var/log/sa/sa10
• sar -p -d

7. Display context switch per second (sar -w)

This reports the total number of processes created per second, and total number of context switches per second. “1 3″ reports for every 1 seconds a total of 3 times.

$ sar -w 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

08:32:24 AM    proc/s   cswch/s08:32:25 AM      3.00     53.0008:32:26 AM      4.00     61.3908:32:27 AM      2.00     57.00

Following are few variations:

• sar -w
• sar -w 1 3
• sar -w -f /var/log/sa/sa10

8. Reports run queue and load average (sar -q)

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3″ reports for every 1 seconds a total of 3 times.

$ sar -q 1 3Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

06:28:53 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked06:28:54 AM         0       230      2.00      3.00      5.00         006:28:55 AM         2       210      2.01      3.15      5.15         006:28:56 AM         2       230      2.12      3.12      5.12         0Average:            3       230      3.12      3.12      5.12         0

Note: The “blocked” column displays the number of tasks that are currently blocked and waiting for I/O operation to complete.

Following are few variations:

• sar -q
• sar -q 1 3
• sar -q -f /var/log/sa/sa10

9. Report network statistics (sar -n)

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.

sar -n KEYWORD

KEYWORD can be one of the following:

• DEV – Displays network devices vital statistics for eth0, eth1, etc.,
• EDEV – Display network device failure statistics
• NFS – Displays NFS client activities
• NFSD – Displays NFS server activities
• SOCK – Displays sockets in use for IPv4
• IP – Displays IPv4 network traffic
• EIP – Displays IPv4 network errors
• ICMP – Displays ICMPv4 network traffic
• EICMP – Displays ICMPv4 network errors
• TCP – Displays TCPv4 network traffic
• ETCP – Displays TCPv4 network errors
• UDP – Displays UDPv4 network traffic
• SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
• ALL – This displays all of the above information. The output will be very long.

$ sar -n DEV 1 1Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:11:13 PM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s01:11:14 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.0001:11:14 PM      eth0    342.57    342.57  93923.76 141773.27      0.00      0.00      0.0001:11:14 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10. Report Sar Data Using Start Time (sar -s)

When you view historic sar data from the /var/log/sa/saXX file using “sar -f” option, it displays all the sar data for that specific day starting from 12:00 a.m for that day.

Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 10:00:00″, it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.

You can combine -s option with other sar option.

For example, to report the load average on 26th of this month starting from 10 a.m in the morning, combine the -q and -s option as shown below.

$ sar -q -f /var/log/sa/sa23 -s 10:00:01Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

10:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked10:10:01 AM         0       127      2.00      3.00      5.00         010:20:01 AM         0       127      2.00      3.00      5.00         0...11:20:01 AM         0       127      5.00      3.00      3.00         012:00:01 PM         0       127      4.00      2.00      1.00         0

There is no option to limit the end-time. You just have to get creative and use head command as shown below.

For example, starting from 10 a.m, if you want to see 7 entries, you have to pipe the above output to “head -n 10″.

$ sar -q -f /var/log/sa/sa23 -s 10:00:01 | head -n 10Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

10:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked10:10:01 AM         0       127      2.00      3.00      5.00         010:20:01 AM         0       127      2.00      3.00      5.00         010:30:01 AM         0       127      3.00      5.00      2.00         010:40:01 AM         0       127      4.00      2.00      1.00         210:50:01 AM         0       127      3.00      5.00      5.00         011:00:01 AM         0       127      2.00      1.00      6.00         011:10:01 AM         0       127      1.00      3.00      7.00         2

There is lot more to cover in Linux performance monitoring and tuning. We are only getting started. More articles to come in the performance series.