使用awk进行日志信息的分组统计

起因

这是今天我线上出了一个bug，需要查看日志并统计一个我需要的信息出现的频率，可以叫做分组统计。

---

日志文件部分内容

00:09:07.655 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=1482502318|itemId=548320850657||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|
00:09:07.673 [showcase_backend][topsdk] ERROR - 2017-06-27 00:09:07.673^_^xxxxxxx^_^taobao.item.recommend.add^_^10.28.42.60^_^Linux^_^18^_^http://gwx.api.taobao.com/router/rest^_^app_key=xxxxx&partner_id=top-sdk-java-20170607&method=taobao.item.recommend.add&v=2.0&session=xxxxxxxx&format=json&sign_method=hmac&num_iid=5483473847602&timestamp=2017-06-27+00%3A09%3A07^_^{"error_response":{"code":530,"msg":"Remote service error","sub_code":"isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES","sub_msg":"橱窗推荐商品失败","request_id":"qm4l3wu2cnhu"}}
00:09:07.673 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=14825032138|itemId=548347847602||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|
00:09:07.695 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=148250232138|itemId=546199835284||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|橱

---

目的

统计每个shopId出现的错误的次数

---

方案

使用awk的分组统计

---

awk代码

cat xxx.log | grep shopId | awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}' 

---

部分统计效果如下

···

800599961375 273
800599958609 4
800599956923 674
800599960564 2
800599960811 2930
529692269 1153
800599959910 2174
800599961392 10
800599961165 1
800599959912 26
63456935 558
67376794 490
800599958627 119
800599960127 757
36903919 1263
800599960357 971
800599960604 1
800599961203 211
800599958858 400
800599960609 160
162307612 1
112880504 45
550108033 22
800599956751 1
800599958437 20
800599960164 206
800599959951 134
800599960393 151
60322340 326
72003720 239
800599959939 422
800599959491 1
800599959479 1
72743666 181
800599958012 7
800599957416 9
800599958702 18
63926103 76
800599958897 8241

···

---

分析awk代码逻辑

cat xxx.log | grep shopId | awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}' 

主要看这段

awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}' 

1.awk -F '|' '{print $2}' 分离出 shopId=1234343 这种格式的数据
2.awk -F '=' '{s[$2] += 1} 将shopId=12323443分离， 定义数组s，awk开始处理每行的数据 s[$2] += 1, 将$2的数据当成key存进arrays，value碰到一样的就加1
3.{for (i in s) {print i, s[i]}} 循环打印出数组的key和value