第一关练习题统计网站最大访问量sed法,隐藏知识数组下标不能重复
1.1.1 获取日志的最大top10,排序
获取两列到新的文件中第一次处理
sed截取字符串中间的内容,sed不支持贪婪匹配.找出图片在的列和图片大小到test1文件
本题需要输出三个指标：【访问次数】【访问次数*单个文件大小】【文件名（可以带URL）】
测试数据
59.33.26.105 --[08/Dec/2010:15:43:56 +0800] "GET /static/images/photos/2.jpgHTTP/1.1" 200 11299 "http://oldboy.blog.51cto.com/static/web/column/17/index.shtml?courseId=43" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
59.33.26.105 --[08/Dec/2010:15:43:56 +0800] "GET /static/images/photos/2.jpgHTTP/1.1" 200 11299"http://oldboy.blog.51cto.com/static/web/column/17/index.shtml?courseId=43" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
59.33.26.105 --[08/Dec/2010:15:44:02 +0800] "GET /static/flex/vedioLoading.swfHTTP/1.1" 200 3583"http://oldboy.blog.51cto.com/static/flex/AdobeVideoPlayer.swf?width=590&height=328&url=/[[DYNAMIC]]/2" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
124.115.4.18 --[08/Dec/2010:15:44:15 +0800] "GET /?=HTTP/1.1" 200 46232 "-" "-"
124.115.4.18 --[08/Dec/2010:15:44:25 +0800] "GET /static/js/web_js.jsHTTP/1.1" 200 4460 "-" "-"
124.115.4.18 --[08/Dec/2010:15:44:25 +0800] "GET /static/js/jquery.lazyload.jsHTTP/1.1" 200 1627 "-" "
============================答题步骤
s sed截取字符串中间的内容,sed不支持贪婪匹配.找出图片在的列和图片大小到test1文件
sed -rn 's/.*GET (.*) HTTP.*200 (.*) /\1 \t \2/gp' bb.txt >>test1.txt
/static/images/photos/2.jpg 11299
/static/images/photos/2.jpg 11299
/static/flex/vedioLoading.swf 3583
/?= 46232 "-" "-"
/static/js/web_js.js 4460 "-" "-"
/static/js/jquery.lazyload.js 1627 "-" "-"
awk -F " " '{print $2}' test1.txt |sort -nr 对数字列进行倒序排序
uniq -c 对每行进行计数
awk -F " " '{print $1"\t"$2}' test1.txt |sort -n|uniq -c>test2.txt #第二次处理
1 /?= 46232
1 /static/flex/vedioLoading.swf 3583
2 /static/images/photos/2.jpg 11299
1 /static/js/jquery.lazyload.js 1627
1 /static/js/web_js.js 4460
[root@ob data]# awk -F " " '{print (($1*$3))"\t" $2}' test2.txt|sort -nru #最后合并排序
46232 /?=
22598 /static/images/photos/2.jpg
4460 /static/js/web_js.js
3583 /static/flex/vedioLoading.swf
1627 /static/js/jquery.lazyload.js
======================================================#sed
awk -F"GET |HTTP/1.1|200 " '{print $2,$4}' /data/bb.txt
/static/images/photos/2.jpg 11299
/static/images/photos/2.jpg 11299
/static/flex/vedioLoading.swf 3583
/?= 46232 "-" "-"
/static/js/web_js.js 4460 "-" "-"
/static/js/jquery.lazyload.js 1627 "-" "-"
[root@ob1 mytmp]# awk -F"GET |HTTP/1.1|200 " '{TP[$2]++}END{for (i in TP) print i,TP[i]}' /data/bb.txt
7
/static/js/jquery.lazyload.js 1
/static/flex/vedioLoading.swf 1
/?= 1
/static/images/photos/2.jpg 2
/static/js/web_js.js 1
#逻辑错误掉坑里写法,awk的特性是按行取数据,前一个数据会将第一个顶掉,所以在END时候只有最后一个数据,
[root@ob1 mytmp]# awk -F"GET |HTTP/1.1|200 " '{TP[$2]++}END{for (i in TP) print i,TP[i],$4}' /data/bb.txt|awk -F ' ' '{print $2*$3,$1}'|sort -nrk1
3254 /static/images/photos/2.jpg
1627 /static/js/web_js.js
1627 /static/js/jquery.lazyload.js
1627 /static/flex/vedioLoading.swf
1627 /?=
0 7
===================================================
第一步查看第二列
[root@ob1 data]# awk -F"GET | HTTP" '{print $2}' bb.txt
/static/images/photos/2.jpg
/static/images/photos/2.jpg
/static/flex/vedioLoading.swf
/?=
/static/js/web_js.js
/static/js/jquery.lazyload.js
第二步 属组取出
[root@ob1 data]# awk -F"GET | HTTP|200 | \"-" '{tt[$2]++}END{for (i in tt)print i,tt[i]}' bb.txt
7
/?= 1
/static/js/web_js.js 1
/static/images/photos/2.jpg 2
/static/flex/vedioLoading.swf 1
/static/js/jquery.lazyload.js 1
第三步再次定义一个数组存放第三列,size[$2]+=$3,又买一个筐子存放第三列,可以使用相同的数组下表应为每一列相同
[root@ob1 data]# awk -F"GET | HTTP/1.1\" 200 | \"\-" '{aa[$2]++;size[$2]+=$3}END{for (i in aa)print aa[i],i,size[i]}' bb.txt
awk: warning: escape sequence `\-' treated as plain `-'
7 0
1 /?= 46232
1 /static/js/web_js.js 4460
2 /static/images/photos/2.jpg 22598
1 /static/flex/vedioLoading.swf 3583
1 /static/js/jquery.lazyload.js 1627
第三步计算排序
[root@ob1 data]# awk -F"GET | HTTP/1.1\" 200 | \"\-" '{aa[$2]++;size[$2]+=$3}END{for (i in aa)print i,aa[i]*size[i]}' bb.txt|sort -nk 2
awk: warning: escape sequence `\-' treated as plain `-'
0
/static/js/jquery.lazyload.js 1627
/static/flex/vedioLoading.swf 3583
/static/js/web_js.js 4460
/static/images/photos/2.jpg 45196
/?= 46232
================================
