Linux awk

  |  
阅读次数
  |  
字数 2,819
  |  
时长 ≈ 16 分钟

假用如下数据演示awk。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ cat netstat.txt
Proto Recv-Q Send-Q Local-Address Foreign-Address State
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 coolshell.cn:80 124.205.5.146:18245 TIME_WAIT
tcp 0 0 coolshell.cn:80 61.140.101.185:37538 FIN_WAIT2
tcp 0 0 coolshell.cn:80 110.194.134.189:1032 ESTABLISHED
tcp 0 0 coolshell.cn:80 123.169.124.111:49809 ESTABLISHED
tcp 0 0 coolshell.cn:80 116.234.127.77:11502 FIN_WAIT2
tcp 0 0 coolshell.cn:80 123.169.124.111:49829 ESTABLISHED
tcp 0 0 coolshell.cn:80 183.60.215.36:36970 TIME_WAIT
tcp 0 4166 coolshell.cn:80 61.148.242.38:30901 ESTABLISHED
tcp 0 1 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
tcp 0 0 coolshell.cn:80 110.194.134.189:4796 ESTABLISHED
tcp 0 0 coolshell.cn:80 183.60.212.163:51082 TIME_WAIT
tcp 0 1 coolshell.cn:80 208.115.113.92:50601 LAST_ACK
tcp 0 0 coolshell.cn:80 123.169.124.111:49840 ESTABLISHED
tcp 0 0 coolshell.cn:80 117.136.20.85:50025 FIN_WAIT2
tcp 0 0 :::22 :::* LISTEN

基本操作

过滤第一列与第四列

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ awk '{print $1,$4}' netstat.txt
Proto Local-Address
tcp 0.0.0.0:3306
tcp 0.0.0.0:80
tcp 127.0.0.1:9000
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp coolshell.cn:80
tcp :::22

按格式输出行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ awk '{printf "%-8s %-8s %-8s %-18s %-22s %-15s\n",$1,$2,$3,$4,$5,$6}' netstat.txt
Proto Recv-Q Send-Q Local-Address Foreign-Address State
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 coolshell.cn:80 124.205.5.146:18245 TIME_WAIT
tcp 0 0 coolshell.cn:80 61.140.101.185:37538 FIN_WAIT2
tcp 0 0 coolshell.cn:80 110.194.134.189:1032 ESTABLISHED
tcp 0 0 coolshell.cn:80 123.169.124.111:49809 ESTABLISHED
tcp 0 0 coolshell.cn:80 116.234.127.77:11502 FIN_WAIT2
tcp 0 0 coolshell.cn:80 123.169.124.111:49829 ESTABLISHED
tcp 0 0 coolshell.cn:80 183.60.215.36:36970 TIME_WAIT
tcp 0 4166 coolshell.cn:80 61.148.242.38:30901 ESTABLISHED
tcp 0 1 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
tcp 0 0 coolshell.cn:80 110.194.134.189:4796 ESTABLISHED
tcp 0 0 coolshell.cn:80 183.60.212.163:51082 TIME_WAIT
tcp 0 1 coolshell.cn:80 208.115.113.92:50601 LAST_ACK
tcp 0 0 coolshell.cn:80 123.169.124.111:49840 ESTABLISHED
tcp 0 0 coolshell.cn:80 117.136.20.85:50025 FIN_WAIT2
tcp 0 0 :::22 :::* LISTEN

过滤记录

过滤第三列为0与第六列为LISTEN的行

1
2
3
4
5
$ awk '$3==0 && $6=="LISTEN"' netstat.txt
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 :::22 :::* LISTEN

过滤第三列的值大于0的记录,并打印出一整行,$0d代表一整行记录

1
2
3
4
5
$ awk ' $3>0 {print $0}' netstat.txt
Proto Recv-Q Send-Q Local-Address Foreign-Address State
tcp 0 4166 coolshell.cn:80 61.148.242.38:30901 ESTABLISHED
tcp 0 1 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
tcp 0 1 coolshell.cn:80 208.115.113.92:50601 LAST_ACK

使用内置变量,过滤第三列为0与第六列为LISTEN的行和第一行

1
2
3
4
5
6
$ awk '$3==0 && $6=="LISTEN" || NR==1 ' netstat.txt
Proto Recv-Q Send-Q Local-Address Foreign-Address State
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 :::22 :::* LISTEN

使用内置变量,并格式化输出,过滤第三列为0与第六列为LISTEN的行和第一行

1
2
3
4
5
6
$ awk '$3==0 && $6=="LISTEN" || NR==1 {printf "%-20s %-20s %s\n",$4,$5,$6}' netstat.txt
Local-Address Foreign-Address State
0.0.0.0:3306 0.0.0.0:* LISTEN
0.0.0.0:80 0.0.0.0:* LISTEN
127.0.0.1:9000 0.0.0.0:* LISTEN
:::22 :::* LISTEN

内置变量

变量 描述
$0 当前记录(这个变量中存放着整个行的内容)
\$1~$n 当前记录的第n个字段,字段间由FS分隔
FS 输入字段分隔符 默认是空格或Tab
NF 当前记录中的字段个数,就是有多少列
NR 已经读出的记录数,就是行号,从1开始,如果有多个文件话,这个值也是不断累加中。
FNR 当前记录数,与NR不同的是,这个值会是各个文件自己的行号
RS 输入的记录分隔符, 默认为换行符
OFS 输出字段分隔符, 默认也是空格
ORS 输出的记录分隔符,默认为换行符
FILENAME 当前输入文件的名字

输出行号

1
2
3
4
5
6
7
$ awk '$3==0 && $6=="ESTABLISHED" || NR==1 {printf "%02s %s %-20s %-20s %s\n",NR, FNR, $4,$5,$6}' netstat.txt
01 1 Local-Address Foreign-Address State
07 7 coolshell.cn:80 110.194.134.189:1032 ESTABLISHED
08 8 coolshell.cn:80 123.169.124.111:49809 ESTABLISHED
10 10 coolshell.cn:80 123.169.124.111:49829 ESTABLISHED
14 14 coolshell.cn:80 110.194.134.189:4796 ESTABLISHED
17 17 coolshell.cn:80 123.169.124.111:49840 ESTABLISHED

指定分隔符

1
2
3
4
5
6
7
$ awk  'BEGIN{FS=":"} {print $1,$3,$6}' /etc/passwd
nobody -2 /var/empty
root 0 /var/root
daemon 1 /var/root
_uucp 4 /var/spool/uucp
_taskgated 13 /var/empty
_networkd 24 /var/networkd

上面的命令也等价于:(-F的意思就是指定分隔符)

1
$ awk -F: '{print $1,$3,$6}' /etc/passwd

以tab为输出字段分隔符

1
2
3
4
5
$ awk  -F: '{print $1,$3,$6}' OFS="\t" /etc/passwd | ccat
nobody -2 /var/empty
root 0 /var/root
daemon 1 /var/root
_uucp 4 /var/spool/uucp

字符串匹配

下面的第一个示例匹配FIN状态, 第二个示例匹配WAIT字样的状态。
其实 ~ 表示模式开始。/ /中是模式。这就是一个正则表达式的匹配。

1
2
3
4
5
6
$ awk '$6 ~ /FIN/ || NR==1 {print NR,$4,$5,$6}' OFS="\t" netstat.txt
1 Local-Address Foreign-Address State
6 coolshell.cn:80 61.140.101.185:37538 FIN_WAIT2
9 coolshell.cn:80 116.234.127.77:11502 FIN_WAIT2
13 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
18 coolshell.cn:80 117.136.20.85:50025 FIN_WAIT2

1
2
3
4
5
6
7
8
9
$ awk '$6 ~ /WAIT/ || NR==1 {print NR,$4,$5,$6}' OFS="\t" netstat.txt
1 Local-Address Foreign-Address State
5 coolshell.cn:80 124.205.5.146:18245 TIME_WAIT
6 coolshell.cn:80 61.140.101.185:37538 FIN_WAIT2
9 coolshell.cn:80 116.234.127.77:11502 FIN_WAIT2
11 coolshell.cn:80 183.60.215.36:36970 TIME_WAIT
13 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
15 coolshell.cn:80 183.60.212.163:51082 TIME_WAIT
18 coolshell.cn:80 117.136.20.85:50025 FIN_WAIT2

跟grep一样匹配行

1
2
3
4
5
$ awk '/LISTEN/' netstat.txt
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 :::22 :::* LISTEN

使用 “/FIN|TIME/” 来匹配 FIN 或者 TIME

1
2
3
4
5
6
7
8
9
$ awk '$6 ~ /FIN|TIME/ || NR==1 {print NR,$4,$5,$6}' OFS="\t" netstat.txt
1 Local-Address Foreign-Address State
5 coolshell.cn:80 124.205.5.146:18245 TIME_WAIT
6 coolshell.cn:80 61.140.101.185:37538 FIN_WAIT2
9 coolshell.cn:80 116.234.127.77:11502 FIN_WAIT2
11 coolshell.cn:80 183.60.215.36:36970 TIME_WAIT
13 coolshell.cn:80 124.152.181.209:26825 FIN_WAIT1
15 coolshell.cn:80 183.60.212.163:51082 TIME_WAIT
18 coolshell.cn:80 117.136.20.85:50025 FIN_WAIT2

取反,过滤不包含WAIT的行

1
2
3
4
5
6
7
8
9
10
11
12
13
$ awk '$6 !~ /WAIT/ || NR==1 {print NR,$4,$5,$6}' OFS="\t" netstat.txt
1 Local-Address Foreign-Address State
2 0.0.0.0:3306 0.0.0.0:* LISTEN
3 0.0.0.0:80 0.0.0.0:* LISTEN
4 127.0.0.1:9000 0.0.0.0:* LISTEN
7 coolshell.cn:80 110.194.134.189:1032 ESTABLISHED
8 coolshell.cn:80 123.169.124.111:49809 ESTABLISHED
10 coolshell.cn:80 123.169.124.111:49829 ESTABLISHED
12 coolshell.cn:80 61.148.242.38:30901 ESTABLISHED
14 coolshell.cn:80 110.194.134.189:4796 ESTABLISHED
16 coolshell.cn:80 208.115.113.92:50601 LAST_ACK
17 coolshell.cn:80 123.169.124.111:49840 ESTABLISHED
19 :::22 :::* LISTEN

或者是

1
2
3
4
5
6
7
8
9
10
11
12
13
$ awk '!/WAIT/' netstat.txt
Proto Recv-Q Send-Q Local-Address Foreign-Address State
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 coolshell.cn:80 110.194.134.189:1032 ESTABLISHED
tcp 0 0 coolshell.cn:80 123.169.124.111:49809 ESTABLISHED
tcp 0 0 coolshell.cn:80 123.169.124.111:49829 ESTABLISHED
tcp 0 4166 coolshell.cn:80 61.148.242.38:30901 ESTABLISHED
tcp 0 0 coolshell.cn:80 110.194.134.189:4796 ESTABLISHED
tcp 0 1 coolshell.cn:80 208.115.113.92:50601 LAST_ACK
tcp 0 0 coolshell.cn:80 123.169.124.111:49840 ESTABLISHED
tcp 0 0 :::22 :::* LISTEN

折分文件

awk拆分文件很简单,使用重定向就好了。下面这个例子,是按第6例分隔文件,相当的简单(其中的NR!=1表示不处理表头)。

1
2
3
4
5
6
$ awk 'NR!=1{print > $6}' netstat.txt
$ ccat LISTEN
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN
tcp 0 0 :::22 :::* LISTEN

指定输出列到输出文件

1
$ awk 'NR!=1{print $4,$5 > $6}' netstat.txt

增加逻辑if elseif else语句,根据不同条件输出到不同文件

1
2
3
$ awk 'NR!=1{if($6 ~ /TIME|ESTABLISHED/) print> "1.txt" ;
else if($6 ~ /LISTEN/) print > "2.txt";
else print > "3.txt" }' netstat.txt

统计

获取计算所有txt文件的总大小

1
2
$ ls -l *.txt | awk '{sum+=$5} END {print sum}'
8165731

统计各个链接状态的数量

1
2
3
4
5
6
7
$ awk 'NR!=1{a[$6]++;} END {for (i in a) print i ", " a[i];}' netstat.txt 
LISTEN, 4
LAST_ACK, 1
FIN_WAIT1, 1
FIN_WAIT2, 3
TIME_WAIT, 3
ESTABLISHED, 6

awk脚本

在上面我们可以看到一个END关键字。END的意思是“处理完所有的行的标识”,即然说到了END就有必要介绍一下BEGIN,这两个关键字意味着执行前和执行后的意思,语法如下:

BEGIN{ 这里面放的是执行前的语句 }
END {这里面放的是处理完所有的行后要执行的语句 }
{这里面放的是处理每一行时要执行的语句}

假设有成绩文件:

1
2
3
4
5
6
$ ccat score.txt
Marry 2143 78 84 77
Jack 2321 66 78 45
Tom 2122 48 77 71
Mike 2537 87 97 95
Bob 2415 40 57 62

编写awk脚本:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ cat cal.awk 
#!/bin/awk -f
#运行前
BEGIN {
math = 0
english = 0
computer = 0

printf "NAME NO. MATH ENGLISH COMPUTER TOTAL\n"
printf "---------------------------------------------\n"
}
#运行中
{
math+=$3
english+=$4
computer+=$5
printf "%-6s %-6s %4d %8d %8d %8d\n", $1, $2, $3,$4,$5, $3+$4+$5
}
#运行后
END {
printf "---------------------------------------------\n"
printf " TOTAL:%10d %8d %8d \n", math, english, computer
printf "AVERAGE:%10.2f %8.2f %8.2f\n", math/NR, english/NR, computer/NR
}

执行awk脚本

1
2
3
4
5
6
7
8
9
10
11
$ awk -f cal.awk score.txt
NAME NO. MATH ENGLISH COMPUTER TOTAL
---------------------------------------------
Marry 2143 78 84 77 239
Jack 2321 66 78 45 189
Tom 2122 48 77 71 196
Mike 2537 87 97 95 279
Bob 2415 40 57 62 159
---------------------------------------------
TOTAL: 319 393 350
AVERAGE: 63.80 78.60 70.00

环境变量

即然说到了脚本,我们来看看怎么和环境变量交互:(使用-v参数和ENVIRON,使用ENVIRON的环境变量需要export)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ x=5

$ y=10
$ export y

$ echo $x $y
5 10

$ awk -v val=$x '{print $1, $2, $3, $4+val, $5+ENVIRON["y"]}' OFS="\t" score.txt
Marry 2143 78 89 87
Jack 2321 66 83 55
Tom 2122 48 82 81
Mike 2537 87 102 105
Bob 2415 40 62 72

其他

打印99乘法表

1
seq 9 | sed 'H;g' | awk -v RS='' '{for(i=1;i<=NF;i++)printf("%dx%d=%d%s", i, NR, i*NR, i==NR?"\n":"\t")}'