AWK“煎蛋”教程

只想达到两个目的:

1)熟悉awk。

2)临近双十一值夜班打发时间。

  • 开始

  从netstat命令中提取了如下信息作为用例

[[email protected] tmp]# netstat >>netstat.txt
[[email protected] tmp]# cat netstat.txt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 appStockWorker.soa.360b:ssh 10.13.133.61:20799          ESTABLISHED
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42687       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42661       TIME_WAIT
tcp        0      0 appStockWorker.soa.360b:ssh 10.13.133.61:53685          ESTABLISHED
tcp        0     52 appStockWorker.soa.360b:ssh 10.13.184.67:60857          ESTABLISHED
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42686       TIME_WAIT
tcp        0      0 appStockWorker.soa.360b:ssh 192.168.192.104:60765       ESTABLISHED
tcp        0      0 appStockWorker.soa.360b:ssh 10.13.184.67:58558          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:58902 [UNKNOWN]:61620             ESTABLISHED
tcp        0      0 appStockWorker.soa.36:37601 [UNKNOWN]:eforward          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50465 [UNKNOWN]:36063             ESTABLISHED
tcp        0      0 appStockWorker.soa.36:20881 [UNKNOWN]:52311             ESTABLISHED
tcp        0      0 appStockWorker.soa.36:60375 [UNKNOWN]:ncube-lm          ESTABLISHED
tcp        1      0 appStockWorker.soa.36:40517 purchaseconfig.purchas:http CLOSE_WAIT
tcp        0      0 appStockWorker.soa.36:50077 [UNKNOWN]:ncube-lm          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:57027 [UNKNOWN]:ms-sql-s          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:39624 [UNKNOWN]:cgn-stat          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:50078 [UNKNOWN]:ncube-lm          ESTABLISHED
tcp        0      0 appStockWorker.soa.36:39093 [UNKNOWN]:isdnlog           ESTABLISHED
tcp        0      0 appStockWorker.soa.36:48835 purchaseconfig.purchas:8016 ESTABLISHED
tcp        0      0 appStockWorker.soa.36:36060 [UNKNOWN]:60983             ESTABLISHED
tcp        0      0 appStockWorker.soa.36:57039 [UNKNOWN]:ms-sql-s          ESTABLISHED 

下面是最简单最常用的awk示例,其输出第1列和第4例,

  其中单引号中的被大括号括着的就是awk的语句,注意,其只能被单引号包含。

  其中的$1..$n表示第几例。注:$0表示整个行。

[[email protected] tmp]# awk ‘{print $1,$4}‘ netstat.txt
Active (w/o
Proto Local
tcp appStockWorker.soa.360b:ssh
tcp appStockWorker.soa.360:http
tcp appStockWorker.soa.360:http
tcp appStockWorker.soa.360b:ssh
tcp appStockWorker.soa.360b:ssh
tcp appStockWorker.soa.360:http
tcp appStockWorker.soa.360b:ssh
tcp appStockWorker.soa.360b:ssh
tcp appStockWorker.soa.36:58902
tcp appStockWorker.soa.36:37601
tcp appStockWorker.soa.36:50057
tcp appStockWorker.soa.36:50465
tcp appStockWorker.soa.36:20881
tcp appStockWorker.soa.36:60375
tcp appStockWorker.soa.36:40517
tcp appStockWorker.soa.36:50077
tcp appStockWorker.soa.36:57027
tcp appStockWorker.soa.36:39624
tcp appStockWorker.soa.36:50078
tcp appStockWorker.soa.36:39093
tcp appStockWorker.soa.36:48835
tcp appStockWorker.soa.36:36060
tcp appStockWorker.soa.36:57039
tcp appStockWorker.soa.36:57829
tcp appStockWorker.soa.36:44759
tcp appStockWorker.soa.36:35197
tcp appStockWorker.soa.36:52525
tcp appStockWorker.soa.36:37605

我们再来看看awk的格式化输出,和C语言的printf没什么两样:

tcp appStockWorker.soa.36:33428
[[email protected] tmp]# awk ‘{printf "%-8s %-8s %-8s %-18s %-30s %-15s\n",$1,$2,$3,$4,$5,$6}‘ netstat.txt
Active   Internet connections (w/o               servers)
Proto    Recv-Q   Send-Q   Local              Address                        Foreign
tcp      0        0        appStockWorker.soa.360b:ssh 10.13.133.61:20799             ESTABLISHED
tcp      0        0        appStockWorker.soa.360:http 192.168.195.187:42687          TIME_WAIT
tcp      0        0        appStockWorker.soa.360:http 192.168.195.187:42661          TIME_WAIT
tcp      0        0        appStockWorker.soa.360b:ssh 10.13.133.61:53685             ESTABLISHED
tcp      0        52       appStockWorker.soa.360b:ssh 10.13.184.67:60857             ESTABLISHED
tcp      0        0        appStockWorker.soa.360:http 192.168.195.187:42686          TIME_WAIT
tcp      0        0        appStockWorker.soa.360b:ssh 192.168.192.104:60765          ESTABLISHED
tcp      0        0        appStockWorker.soa.360b:ssh 10.13.184.67:58558             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:58902 [UNKNOWN]:61620                ESTABLISHED
tcp      0        0        appStockWorker.soa.36:37601 [UNKNOWN]:eforward             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm             TIME_WAIT
tcp      0        0        appStockWorker.soa.36:50465 [UNKNOWN]:36063                ESTABLISHED
tcp      0        0        appStockWorker.soa.36:20881 [UNKNOWN]:52311                ESTABLISHED
tcp      0        0        appStockWorker.soa.36:60375 [UNKNOWN]:ncube-lm             ESTABLISHED
tcp      1        0        appStockWorker.soa.36:40517 purchaseconfig.purchas:http    CLOSE_WAIT
tcp      0        0        appStockWorker.soa.36:50077 [UNKNOWN]:ncube-lm             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:57027 [UNKNOWN]:ms-sql-s             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:39624 [UNKNOWN]:cgn-stat             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:50078 [UNKNOWN]:ncube-lm             ESTABLISHED
tcp      0        0        appStockWorker.soa.36:39093 [UNKNOWN]:isdnlog              ESTABLISHED 
  • 过滤记录

我们再来看看如何过滤记录(下面过滤条件为:第三列的值为0 && 第6列的值为LISTEN)

[[email protected] tmp]# awk ‘$3==0 && $6=="TIME_WAIT"‘ netstat.txt
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42687       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42661       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42686       TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43856 TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50056 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57017 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50046 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50068 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57005 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50047 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50069 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:56996 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43852 TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43855 TIME_WAIT   

其中的“==”为比较运算符。其他比较运算符:!=, >, <, >=, <=

我们来看看各种过滤记录的方式:

[[email protected] tmp]# awk ‘$3>0 {print $0}‘ netstat.txt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0     52 appStockWorker.soa.360b:ssh 10.13.184.67:60857          ESTABLISHED
tcp        0      1 appStockWorker.soa.36:59174 [UNKNOWN]:20880             SYN_SENT    

如果我们需要表头的话,我们可以引入内建变量NR:

[[email protected] tmp]# awk ‘$3==0 && $6=="TIME_WAIT" || NR==2 ‘ netstat.txt
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42687       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42661       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42686       TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43856 TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50056 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57017 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50046 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50068 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57005 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50047 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50069 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:56996 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43852 TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43855 TIME_WAIT 

再加上格式化输出:

[[email protected] tmp]# awk ‘$3==0 && $6=="TIME_WAIT" || NR==2 {printf "%-20s %-30s %s\n",$4,$5,$6}‘ netstat.txt
Local                Address                        Foreign
appStockWorker.soa.360:http 192.168.195.187:42687          TIME_WAIT
appStockWorker.soa.360:http 192.168.195.187:42661          TIME_WAIT
appStockWorker.soa.360:http 192.168.195.187:42686          TIME_WAIT
appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm             TIME_WAIT
localhost.localdomain:8010 localhost.localdomain:43856    TIME_WAIT
appStockWorker.soa.36:50056 [UNKNOWN]:ncube-lm             TIME_WAIT
appStockWorker.soa.36:57017 [UNKNOWN]:ms-sql-s             TIME_WAIT
appStockWorker.soa.36:50046 [UNKNOWN]:ncube-lm             TIME_WAIT
appStockWorker.soa.36:50068 [UNKNOWN]:ncube-lm             TIME_WAIT
appStockWorker.soa.36:57005 [UNKNOWN]:ms-sql-s             TIME_WAIT
appStockWorker.soa.36:50047 [UNKNOWN]:ncube-lm             TIME_WAIT
appStockWorker.soa.36:50069 [UNKNOWN]:ncube-lm             TIME_WAIT
appStockWorker.soa.36:56996 [UNKNOWN]:ms-sql-s             TIME_WAIT
localhost.localdomain:8010 localhost.localdomain:43852    TIME_WAIT
localhost.localdomain:8010 localhost.localdomain:43855    TIME_WAIT
  • 内建变量
$0 当前记录(这个变量中存放着整个行的内容)
$1~$n 当前记录的第n个字段,字段间由FS分隔
FS 输入字段分隔符 默认是空格或Tab
NF 当前记录中的字段个数,就是有多少列
NR 已经读出的记录数,就是行号,从1开始,如果有多个文件话,这个值也是不断累加中。
FNR 当前记录数,与NR不同的是,这个值会是各个文件自己的行号
RS 输入的记录分隔符, 默认为换行符
OFS 输出字段分隔符, 默认也是空格
ORS 输出的记录分隔符,默认为换行符
FILENAME 当前输入文件的名字

怎么使用呢,比如:我们如果要输出行号:

[[email protected] tmp]# awk ‘$3==0 && $6=="TIME_WAIT" || NR==2 {printf "%02s %s %-20s %-30s %s\n",NR,FNR,$4,$5,$6}‘ netstat.txt
02 2 Local                Address                        Foreign
04 4 appStockWorker.soa.360:http 192.168.195.187:42687          TIME_WAIT
05 5 appStockWorker.soa.360:http 192.168.195.187:42661          TIME_WAIT
08 8 appStockWorker.soa.360:http 192.168.195.187:42686          TIME_WAIT
13 13 appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm             TIME_WAIT
32 32 localhost.localdomain:8010 localhost.localdomain:43856    TIME_WAIT
42 42 appStockWorker.soa.36:50056 [UNKNOWN]:ncube-lm             TIME_WAIT
60 60 appStockWorker.soa.36:57017 [UNKNOWN]:ms-sql-s             TIME_WAIT
63 63 appStockWorker.soa.36:50046 [UNKNOWN]:ncube-lm             TIME_WAIT
66 66 appStockWorker.soa.36:50068 [UNKNOWN]:ncube-lm             TIME_WAIT
70 70 appStockWorker.soa.36:57005 [UNKNOWN]:ms-sql-s             TIME_WAIT
103 103 appStockWorker.soa.36:50047 [UNKNOWN]:ncube-lm             TIME_WAIT
108 108 appStockWorker.soa.36:50069 [UNKNOWN]:ncube-lm             TIME_WAIT
112 112 appStockWorker.soa.36:56996 [UNKNOWN]:ms-sql-s             TIME_WAIT
117 117 localhost.localdomain:8010 localhost.localdomain:43852    TIME_WAIT
127 127 localhost.localdomain:8010 localhost.localdomain:43855    TIME_WAIT
  • 指定分隔符
[[email protected] tmp]# awk ‘BEGIN{FS=":"} {print $1,$3,$6}‘ /etc/passwd
root 0 /root
bin 1 /bin
daemon 2 /sbin
adm 3 /var/adm
lp 4 /var/spool/lpd
sync 5 /sbin
shutdown 6 /sbin
halt 7 /sbin
mail 8 /var/spool/mail
uucp 10 /var/spool/uucp
operator 11 /root
games 12 /usr/games
gopher 13 /var/gopher
ftp 14 /var/ftp
nobody 99 /
vcsa 69 /dev
saslauth 499 /var/empty/saslauth
postfix 89 /var/spool/postfix
sshd 74 /var/empty/sshd
ntp 38 /etc/ntp
admin 500 /home/admin
nagios 501 /home/nagios

上面的命令也等价于:(-F的意思就是指定分隔符)

[[email protected] tmp]# awk -F: ‘{print $1,$3,$6}‘ /etc/passwd

注:如果你要指定多个分隔符,你可以这样来:

awk -F ‘[;:]‘

再来看一个以\t作为分隔符输出的例子(下面使用了/etc/passwd文件,这个文件是以:分隔的):

[[email protected] tmp]# awk -F: ‘{print $1,$3,$6}‘ OFS="\t" /etc/passwd
root    0       /root
bin     1       /bin
daemon  2       /sbin
adm     3       /var/adm
lp      4       /var/spool/lpd
sync    5       /sbin
shutdown        6       /sbin
halt    7       /sbin
mail    8       /var/spool/mail
uucp    10      /var/spool/uucp
operator        11      /root
games   12      /usr/games
gopher  13      /var/gopher
ftp     14      /var/ftp
nobody  99      /
vcsa    69      /dev
saslauth        499     /var/empty/saslauth
postfix 89      /var/spool/postfix
sshd    74      /var/empty/sshd
ntp     38      /etc/ntp
admin   500     /home/admin
nagios  501     /home/nagios
  • 字符串匹配

我们再来看几个字符串匹配的示例:

[[email protected] tmp]# awk ‘$6 ~ /TIME/ || NR==1 {print NR,$4,$5,$6}‘  OFS="\t" netstat.txt
1       (w/o    servers)
4       appStockWorker.soa.360:http     192.168.195.187:42687   TIME_WAIT
5       appStockWorker.soa.360:http     192.168.195.187:42661   TIME_WAIT
8       appStockWorker.soa.360:http     192.168.195.187:42686   TIME_WAIT
13      appStockWorker.soa.36:50057     [UNKNOWN]:ncube-lm      TIME_WAIT
32      localhost.localdomain:8010      localhost.localdomain:43856     TIME_WAIT
42      appStockWorker.soa.36:50056     [UNKNOWN]:ncube-lm      TIME_WAIT
60      appStockWorker.soa.36:57017     [UNKNOWN]:ms-sql-s      TIME_WAIT
63      appStockWorker.soa.36:50046     [UNKNOWN]:ncube-lm      TIME_WAIT
66      appStockWorker.soa.36:50068     [UNKNOWN]:ncube-lm      TIME_WAIT
70      appStockWorker.soa.36:57005     [UNKNOWN]:ms-sql-s      TIME_WAIT
103     appStockWorker.soa.36:50047     [UNKNOWN]:ncube-lm      TIME_WAIT
108     appStockWorker.soa.36:50069     [UNKNOWN]:ncube-lm      TIME_WAIT
112     appStockWorker.soa.36:56996     [UNKNOWN]:ms-sql-s      TIME_WAIT
117     localhost.localdomain:8010      localhost.localdomain:43852     TIME_WAIT
127     localhost.localdomain:8010      localhost.localdomain:43855     TIME_WAIT

上面的第一个示例匹配FIN状态,其实 ~ 表示模式开始。/ /中是模式。这就是一个正则表达式的匹配。

其实awk可以像grep一样的去匹配第一行,就像这样:

[[email protected] tmp]# awk ‘/CLOSE_WAIT/‘ netstat.txt
tcp        1      0 appStockWorker.soa.36:40517 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:57829 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:44759 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:42397 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:46115 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:36460 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:52790 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:34745 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:59139 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:37024 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:34171 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:47671 purchaseconfig.purchas:http CLOSE_WAIT

我们可以使用 “/CLOSE|TIME/” 来匹配 CLOSE 或者 TIME :

[[email protected] tmp]# awk ‘$6 ~ /CLOSE|TIME/ || NR==1 {print NR,$4,$5,$6}‘  OFS="\t" netstat.txt
1       (w/o    servers)
4       appStockWorker.soa.360:http     192.168.195.187:42687   TIME_WAIT
5       appStockWorker.soa.360:http     192.168.195.187:42661   TIME_WAIT
8       appStockWorker.soa.360:http     192.168.195.187:42686   TIME_WAIT
13      appStockWorker.soa.36:50057     [UNKNOWN]:ncube-lm      TIME_WAIT
17      appStockWorker.soa.36:40517     purchaseconfig.purchas:http     CLOSE_WAIT
26      appStockWorker.soa.36:57829     purchaseconfig.purchas:http     CLOSE_WAIT
27      appStockWorker.soa.36:44759     purchaseconfig.purchas:http     CLOSE_WAIT
32      localhost.localdomain:8010      localhost.localdomain:43856     TIME_WAIT
36      appStockWorker.soa.36:42397     purchaseconfig.purchas:http     CLOSE_WAIT
39      appStockWorker.soa.36:46115     purchaseconfig.purchas:http     CLOSE_WAIT
42      appStockWorker.soa.36:50056     [UNKNOWN]:ncube-lm      TIME_WAIT
44      appStockWorker.soa.36:36460     purchaseconfig.purchas:8016     CLOSE_WAIT
47      appStockWorker.soa.36:52790     purchaseconfig.purchas:8016     CLOSE_WAIT
48      appStockWorker.soa.36:34745     purchaseconfig.purchas:http     CLOSE_WAIT
60      appStockWorker.soa.36:57017     [UNKNOWN]:ms-sql-s      TIME_WAIT
63      appStockWorker.soa.36:50046     [UNKNOWN]:ncube-lm      TIME_WAIT
66      appStockWorker.soa.36:50068     [UNKNOWN]:ncube-lm      TIME_WAIT
70      appStockWorker.soa.36:57005     [UNKNOWN]:ms-sql-s      TIME_WAIT
75      appStockWorker.soa.36:59139     purchaseconfig.purchas:http     CLOSE_WAIT
88      appStockWorker.soa.36:37024     purchaseconfig.purchas:http     CLOSE_WAIT
100     appStockWorker.soa.36:34171     purchaseconfig.purchas:http     CLOSE_WAIT
103     appStockWorker.soa.36:50047     [UNKNOWN]:ncube-lm      TIME_WAIT
108     appStockWorker.soa.36:50069     [UNKNOWN]:ncube-lm      TIME_WAIT
112     appStockWorker.soa.36:56996     [UNKNOWN]:ms-sql-s      TIME_WAIT
117     localhost.localdomain:8010      localhost.localdomain:43852     TIME_WAIT
123     appStockWorker.soa.36:47671     purchaseconfig.purchas:http     CLOSE_WAIT
127     localhost.localdomain:8010      localhost.localdomain:43855     TIME_WAIT

再来看看模式取反的例子:

[[email protected] tmp]# awk ‘$6 !~ /WAIT/ || NR==1 {print NR,$4,$5,$6}‘ OFS="\t" netstat.txt
1       (w/o    servers)
2       Local   Address Foreign
3       appStockWorker.soa.360b:ssh     10.13.133.61:20799      ESTABLISHED
6       appStockWorker.soa.360b:ssh     10.13.133.61:53685      ESTABLISHED
7       appStockWorker.soa.360b:ssh     10.13.184.67:60857      ESTABLISHED
9       appStockWorker.soa.360b:ssh     192.168.192.104:60765   ESTABLISHED
10      appStockWorker.soa.360b:ssh     10.13.184.67:58558      ESTABLISHED
11      appStockWorker.soa.36:58902     [UNKNOWN]:61620 ESTABLISHED
12      appStockWorker.soa.36:37601     [UNKNOWN]:eforward      ESTABLISHED
14      appStockWorker.soa.36:50465     [UNKNOWN]:36063 ESTABLISHED
15      appStockWorker.soa.36:20881     [UNKNOWN]:52311 ESTABLISHED
16      appStockWorker.soa.36:60375     [UNKNOWN]:ncube-lm      ESTABLISHED
18      appStockWorker.soa.36:50077     [UNKNOWN]:ncube-lm      ESTABLISHED
19      appStockWorker.soa.36:57027     [UNKNOWN]:ms-sql-s      ESTABLISHED
20      appStockWorker.soa.36:39624     [UNKNOWN]:cgn-stat      ESTABLISHED
21      appStockWorker.soa.36:50078     [UNKNOWN]:ncube-lm      ESTABLISHED
22      appStockWorker.soa.36:39093     [UNKNOWN]:isdnlog       ESTABLISHED
23      appStockWorker.soa.36:48835     purchaseconfig.purchas:8016     ESTABLISHED

或是

awk ‘!/WAIT/‘ netstat.txt
  • 折分文件

awk拆分文件很简单,使用重定向就好了。下面这个例子,是按第6例分隔文件,相当的简单(其中的NR!=1表示不处理表头)。

[[email protected] tmp]# awk ‘NR!=1{print >$6}‘ netstat.txt
[[email protected] tmp]# ls
aaa.sh                              hudson-remoting2477836264792765148  hudson-remoting5656432457920204714  hudson-remoting9133536725345626708
CLOSE_WAIT                          hudson-remoting267624195045038773   hudson-remoting5738546470805781773  hudson-remoting9195511106968949486
ESTABLISHED                         hudson-remoting2782799830874667135  hudson-remoting5862972180048370977  jna
Foreign                             hudson-remoting3026235916867749211  hudson-remoting6048063392649864890  netstat.txt
hsperfdata_admin                    hudson-remoting3433616259928789742  hudson-remoting6507341373611789601  ssh-FhWcBH2899
hsperfdata_root                     hudson-remoting3493246008991492784  hudson-remoting6509990390611516239  ssh-WwilYy6121
hudson-remoting1020802418708175254  hudson-remoting3691137831114289115  hudson-remoting796125568617163193   SYN_SENT
hudson-remoting1288644704899054285  hudson-remoting4268187764899187207  hudson-remoting7989065960669590914  TIME_WAIT
hudson-remoting135955832302030064   hudson-remoting4541529833790610464  hudson-remoting8053604203671609757  yum.log
hudson-remoting1594902447089122032  hudson-remoting4709407012955215290  hudson-remoting8134591377726690969  zbuilder_deploy.sh
hudson-remoting1657148174411050811  hudson-remoting4718570550800607052  hudson-remoting8297456844425463707
hudson-remoting2051518431688100965  hudson-remoting5098346600070636216  hudson-remoting8859803414006288608
[[email protected] tmp]# cat CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:40517 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:57829 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:44759 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:42397 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:46115 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:36460 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:52790 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:34745 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:59139 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:37024 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:34171 purchaseconfig.purchas:http CLOSE_WAIT
tcp        1      0 appStockWorker.soa.36:47671 purchaseconfig.purchas:http CLOSE_WAIT
[[email protected] tmp]# cat TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42687       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42661       TIME_WAIT
tcp        0      0 appStockWorker.soa.360:http 192.168.195.187:42686       TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50057 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43856 TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50056 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57017 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50046 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50068 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:57005 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50047 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:50069 [UNKNOWN]:ncube-lm          TIME_WAIT
tcp        0      0 appStockWorker.soa.36:56996 [UNKNOWN]:ms-sql-s          TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43852 TIME_WAIT
tcp        0      0 localhost.localdomain:8010  localhost.localdomain:43855 TIME_WAIT 

你也可以把指定的列输出到文件:

awk ‘NR!=1{print $4,$5 >$6}‘ netstat.txt

再复杂一点:(注意其中的if-else-if语句,可见awk其实是个脚本解释器)

[[email protected] tmp]# awk ‘NR!=1{if($6 ~/TIME|ESTABLISHED/) print >"1.txt";else if($6 ~/CLOSE/) print >"2.txt";else print > "3.txt"}‘ netstat.txt 

[[email protected] tmp]# cat 2.txt
tcp 1 0 appStockWorker.soa.36:40517 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:57829 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:44759 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:42397 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:46115 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:36460 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:52790 purchaseconfig.purchas:8016 CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:34745 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:59139 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:37024 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:34171 purchaseconfig.purchas:http CLOSE_WAIT
tcp 1 0 appStockWorker.soa.36:47671 purchaseconfig.purchas:http CLOSE_WAIT
[[email protected] tmp]# cat 3.txt
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 appStockWorker.soa.36:59174 [UNKNOWN]:20880 SYN_SENT

  • 统计

下面的命令计算所有的C文件,CPP文件和H文件的文件大小总和。

[[email protected] tmp]# ls -l *.txt *.log *.sh| awk ‘{sum+=$5} END {print sum}‘
42044

我们再来看一个统计各个connection状态的用法:注意其中的数组的用法

[[email protected] tmp]# awk ‘NR!=1{a[$6]++;} END {for (i in a) print i "," a[i];}‘ netstat.txt
TIME_WAIT,15
CLOSE_WAIT,12
SYN_SENT,1
ESTABLISHED,105
Foreign,1

再来看看统计每个用户的进程的占了多少内存(注:sum的RSS那一列)

[[email protected] tmp]# ps aux | awk ‘NR!=1{a[$1]+=$6;} END { for(i in a) print i ", " a[i]"KB";}‘
nagios, 480KB
admin, 4247804KB
postfix, 4692KB
root, 157908KB
  • awk脚本

在上面我们可以看到一个END关键字。END的意思是“处理完所有的行的标识”,即然说到了END就有必要介绍一下BEGIN,这两个关键字意味着执行前和执行后的意思,语法如下:

  • BEGIN{ 这里面放的是执行前的语句 }
  • END {这里面放的是处理完所有的行后要执行的语句 }
  • {这里面放的是处理每一行时要执行的语句}

为了说清楚这个事,我们来看看下面的示例:

假设有这么一个文件(学生成绩表):

[[email protected] tmp]# cat score.txt
Marry   2143 78 84 77
Jack    2321 66 78 45
Tom     2122 48 77 71
Mike    2537 87 97 95
Bob     2415 40 57 62
[[email protected] tmp]# vi cal.awk
#!/bin/awk -f
#运行前
BEGIN{
        math = 0
        english = 0
        computer = 0
        printf "NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL\n"
        printf "---------------------------------------------\n"
}
#运行中
{
        math+=$3
        english+=$4
        computer+=$5
        printf "%-6s %-6s %4d %8d %8d %8d\n", $1, $2, $3,$4,$5, $3+$4+$5
}
#运行后
END{
        printf "---------------------------------------------\n"
        printf "  TOTAL:%10d %8d %8d \n", math, english, computer
        printf "AVERAGE:%10.2f %8.2f %8.2f\n", math/NR, english/NR, computer/NR
}

我们来看一下执行结果:(也可以这样运行 ./cal.awk score.txt)

[[email protected] tmp]# awk -f cal.awk score.txt
NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL
---------------------------------------------
Marry  2143     78       84       77      239
Jack   2321     66       78       45      189
Tom    2122     48       77       71      196
Mike   2537     87       97       95      279
Bob    2415     40       57       62      159
---------------------------------------------
  TOTAL:       319      393      350
AVERAGE:     63.80    78.60    70.00
  • 环境变量

即然说到了脚本,我们来看看怎么和环境变量交互:(使用-v参数和ENVIRON,使用ENVIRON的环境变量需要export)

[[email protected] tmp]# x=5
[[email protected] tmp]# y=10
[[email protected] tmp]# export y
[[email protected] tmp]# echo $x $y
5 10
[[email protected] tmp]# awk -v val=$x ‘{print $1,$2,$3,$4+val,$5+ENVIRON["y"]}‘ OFS="\t" score.txt
Marry   2143    78      89      87
Jack    2321    66      83      55
Tom     2122    48      82      81
Mike    2537    87      102     105
Bob     2415    40      62      72
  •  几个例子
#从file文件中找出长度大于80的行
awk ‘length>80‘ file
#按连接数查看客户端IP
[[email protected] tmp]# netstat -ntu | awk NR!=1‘{print $5}‘ | cut -d: -f1 | sort | uniq -c | sort -nr
    125
      5 192.168.195.187
      2 10.13.184.67
      2 10.13.133.61
      1 Address
      1 192.168.192.104
#打印99乘法表
[[email protected] tmp]# seq 9 | sed ‘H;g‘ | awk -v RS=‘‘ ‘{for(i=1;i<=NF;i++)printf("%dx%d=%d%s", i, NR, i*NR, i==NR?"\n":"\t")}‘
1x1=1
1x2=2   2x2=4
1x3=3   2x3=6   3x3=9
1x4=4   2x4=8   3x4=12  4x4=16
1x5=5   2x5=10  3x5=15  4x5=20  5x5=25
1x6=6   2x6=12  3x6=18  4x6=24  5x6=30  6x6=36
1x7=7   2x7=14  3x7=21  4x7=28  5x7=35  6x7=42  7x7=49
1x8=8   2x8=16  3x8=24  4x8=32  5x8=40  6x8=48  7x8=56  8x8=64
1x9=9   2x9=18  3x9=27  4x9=36  5x9=45  6x9=54  7x9=63  8x9=72  9x9=81
时间: 2024-10-13 13:30:11

AWK“煎蛋”教程的相关文章

爬虫之煎蛋网妹子图 大爬哦

今天为了测试一下urllib2模块中的headers部分,也就是模拟客户端登陆的那个东东,就对煎蛋网妹子图练了一下手,感觉还可以吧.分享一下! 代码如下 # coding:UTF-8 import urllib2,urllib,re,random def getHtml(url) : request = urllib2.Request(url,headers=headers) response = urllib2.urlopen(request) page = response.read() r

python爬虫学习(1)__抓取煎蛋图片

#coding=utf-8 #python_demo 爬取煎蛋妹子图在本地文件夹 import requests import threading import time import os from bs4 import BeautifulSoup #伪造头文件 headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chr

用python来抓取“煎蛋网”上面的美女图片,尺度很大哦!哈哈

废话不多说,先上代码: import urllib.request import re #获得当前页面的页数page_name def get_pagenum(url): req = urllib.request.Request(url) req.add_header('User-Agent','Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safar

[Java]使用HttpClient实现一个简单爬虫,抓取煎蛋妹子图

  第一篇文章,就从一个简单爬虫开始吧. 这只虫子的功能很简单,抓取到”煎蛋网xxoo”网页(http://jandan.net/ooxx/page-1537),解析出其中的妹子图,保存至本地. 先放结果: 从程序来讲,步骤分为三步: 1.发起一个http请求,获取返回的response内容: 2.解析内容,分离出有效图片的url: 3.根据这些图片的url,生成图片保存至本地. 开始详细说明: 准备工作:HttpClient的Jar包,访问http://hc.apache.org/   自行下

手把手教你用Python爬虫煎蛋妹纸海量图片

我们的目标是用爬虫来干一件略污事情 最近听说煎蛋上有好多可爱的妹子,而且爬虫从妹子图抓起练手最好,毕竟动力大嘛.而且现在网络上的妹子很黄很暴力,一下接受太多容易营养不量,但是本着有人身体就比较好的套路,特意分享下用点简单的技术去获取资源. 以后如果有机会,再给大家说说日本爱情动(大)作(雾)片的种子搜索爬取,多多关注. 请先准备 作案工具 我们只准备最简单的 python 2.7.11 Google Chrome 安装的时候记得把pip带上,这样可以方便我们安装一些好用的包,来方便我们干坏事(学

【凯子哥带你做高仿】“煎蛋”Android版的高仿及优化(三)——使用GreenDao实现本地Sqlite缓存

到目前为止,煎蛋的Android项目算是告一段落了,功能基本都已完成,那么今天,我就介绍一下在煎蛋这个项目里,是怎么完成数据缓存功能的.想看代码的请戳煎蛋项目的GITHUB地址 转载请注明出处:http://blog.csdn.net/zhaokaiqiang1992 缓存功能的解决方案 配置GreenDao 实现缓存功能 其他资料 缓存功能的解决方案 因为算是一个阅读类的应用,所以说如果在无网络情况下,用户打开App还能看到内容的话,属于比较好的用户体验.那么,这就涉及到本地缓存了. 本地缓存

Python爬虫之爬取煎蛋网妹子图

这篇文章通过简单的Python爬虫(未使用框架,仅供娱乐)获取并下载煎蛋网妹子图指定页面或全部图片,并将图片下载到磁盘. 首先导入模块:urllib.request.re.os import urllib.request import re import os urllib.request模块用于获取HTML页面数据 re模块用于通过正则表达式解析并截取HTML页面图片url os模块用于文件夹相关操作 代码不多,直接贴出来,代码解释在注释中: def crawl_jiandan(page, p

python3煎蛋网的爬虫

做的第一个爬虫就遇上了硬茬儿,可能是我http头没改好还是我点击次数过高导致无法循环爬取煎蛋网的妹子图.... 不过也算是邪恶了一把...技术本无罪~~~ 爬了几页的照片下来还是很赏心悦目的~ import urllib.request import re import time import requests k=1 def read_url(url,k): user_agent = 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36

爬虫实例——爬取煎蛋网OOXX频道(反反爬虫——伪装成浏览器)

煎蛋网在反爬虫方面做了不少工作,无法通过正常的方式爬取,比如用下面这段代码爬取无法得到我们想要的源代码. import requests url = 'http://jandan.net/ooxx' print requests.get(url).text 执行上述代码,你得到的结果应该跟我一样: 煎蛋网应该是通过检测headers来判断是否爬虫,要想获取正常的源代码,需要伪装成浏览器. # -*- coding: utf-8 -*- import re import requests from