当前位置: 首页 > Linux

01-09Linux三剑客-awk

时间:2023-04-06 23:06:06 Linux

逐行定义要读取的文件,以空格为默认分隔符,对每一行进行切片,然后对切下的部分进行后续处理。处理流程:格式:awk[parameter]'patternaction'[file]pattern:正则表达式action:对匹配到的内容执行的命令(默认是输入每一行的内容)常用参数:FILENAME:名称awk浏览的文件的序号BEGIN:处理文本前要执行的操作END:处理文本后要执行的操作FS:设置输入字段分隔符,相当于命令行-F参数NF:字段数(列)浏览记录NR:读取的记录数(行)OFS:输出字段分隔符ORS:输入记录分隔符RS:控制记录分隔符$0:整条记录$n:表示当前行的第n个域。实际应用在/etc/passwd文件中搜索所有包含root关键字的行,并打印相应的shell$cat/etc/passwd|head-2root:x:0:0:root:/root:/bin/bashbin:x:1:1:bin:/bin:/sbin/nologin$awk-F:'/root/{print$0}'/etc/passwd#$0:打印整行root:x:0:0:root:/root:/bin/bashoperator:x:11:0:operator:/root:/sbin/nologin$awk-F:'/root/{print$7}'/etc/passwd/bin/bash/sbin/nologin在/etc/passwd文件的第2行打印信息$awk-F:'NR==2{print$0}'/etc/passwdbin:x:1:1:bin:/bin:/sbin/nologin使用BEGIN添加标头$awk-F:'BEGIN{print"start"}{print$1$2}'/etc/passwd|head-5启动rootxbinxdaemonxadmx自定义行分割符号$echo"111222|333444|555666"|awk'BEGIN{RS="|"}{print$0}'111222333444555666课程实际应用找出日志中的404和500错误:$catnginx.log|头-3223.104.7.59--[05/Dec/2018:00:00:01+0000]"GET/topics/17112HTTP/2.0"2009874"https://www.googleapis.com/auth/chrome-content-suggestions""Mozilla/5.0(iPhone;CPUiPhoneOS12_1,如MacOSX)AppleWebKit/605.1.15(KHTML,如Gecko)CriOS/70.0.3538.75Mobile/15E148Safari/605.1"0.0400.040.$lessnginx.log|awk'$9~/404|500/'#统计$lessnginx.log|awk'$9~/404|500/'|awk'{print$9}'|sort|uniq-c2664041500查找访问量最大的IP$lessnginx.log|awk'{print$1}'|sort|uniq-c|sort-nr|head-3282216.244.66.241130136.243.151.90110127.0.0.1实用tips:裁剪后查看相应字段第一列$lessnginx.log|头-1|awk'{for(i=1;i<=NF;i++)print"$"i"="$i}'$1=223.104.7.59$2=-$3=-$4=[05/Dec/2018:00:00:01$5=+0000]$6="GET$7=/topics/17112$8=HTTP/2.0"$9=200$10=9874$11="https://www.googleapis.com/auth/chrome-content-suggestions"$12="Mozilla/5.0$13=(iPhone;$14=CPU$15=iPhone$16=OS$17=12_1$18=like$19=Mac$20=OS$21=X)$22=AppleWebKit/605.1.15$23=(KHTML,$24=like$25=Gecko)$26=CriOS/70.0.3538.75$27=Mobile/15E148$28=Safari/605.1"$29=0.040$30=0.040$31=.takemaxresponsetime$lessnginx.log|awk'{print$(NF-1)}'|sort-nr|head-186462.600取出top3的响应时间,打印出对应接口$lessnginx.log|awk'{print$(NF-1),$7}'|sort-nr|head-386462.600/cable77331.425/cable59394.978/cable取出所有请求的平均值$lessnginx.log|awk'{sum+=$(NF-1)}END{printsum/NR}'1082.57计算每个接口的平均响应时间,取出top3$lessnginx.log|awk'{time[$7]+=$(NF-1);count[$7]+=1}END{for(kintime)printtime[k]/count[k],k}'|sort-nr|head-33707.26/cable5.592/topics/95242.19/topics/8343?locale=en计算QPS(requestspersecond)对应顶级路由地址,打印top5less的顶级路由nginx.log|awk'{print$4,$7}'|sed's#[?!].*##'|sed-E's#([^]*)*(/[^/]*).*#\1:\2#'|排序|awk-F:'{cur=($3*60+$4);}NR==1{min=cur;max=cur;}NR>1{count[$NF]+=1;如果(curmax)max=cur;}END{for(kincount)printk,count[k]/(max-min+1)}'|排序-k2-nr|head-5每1秒统计aliyundun两个进程的cpu和内存,在分类下汇总10秒内的平均cpu和响应时间$top-b-d1-n10|grep-i阿里云盾|awk'{cpu[$NF]+=$(NF-3);mem[$NF]+=$(NF-2);count[$NF]+=1}END{for(kincpu)打印k,cpu[k]/count[k],mem[k]/count[k]}'AliYunDunUpdate00.1AliYunDun0.81.2统计当前服务器上每个监听端口对应的网络状态个数$netstat-tn|awk'{print$4,$NF}'|awk-F:'{print$NF}'|排序|uniq-c1522ESTABLISHED325TIME_WAIT151368TIME_WAIT151374TIME_WAIT151380TIME_WAIT151386TIME_WAIT157968ESTABLISHED49101TIME_WAIT19102ESTABLISHED1o本地服务器)1(w/w/