NetCap

项目地址：https://github.com/dreadl0ck/netcap/tree/master

Overview

概述

The Netcap (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime.
Netcap 框架可以有效地将网络数据包流转换成表示特定协议或自定义抽象类型的，平台中性的，类型安全的结构化审核记录。这些审核记录可以存储在磁盘上或通过网络进行交换，非常适合作为机器算法学习的数据源。由于解析不受信任的输入可能存在危险，并且网络数据也可能潜在恶意的风险，所以（框架）实现是通过可以提供垃圾搜集、内存运行时安全的编程语言来执行的。

It was developed for a series of experiments in my bachelor thesis: Implementation and evaluation of secure and scalable anomaly-based network intrusion detection. The thesis is included at the root of this repository (file: mied18.pdf) and can be used as an introduction to the framework, its philosophy and architecture. However, be aware that the command-line interface was refactored heavily and the thesis examples refer to very early versions. This documentation contains the latest API and usage examples. Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on researchgate.
它是为了我学士论文的一系列实验而开发的：安全的，基于异常可扩展的网络入侵检测的实现和评估。这篇论文包含在这个仓库的根目录中，它可以用来介绍这个框架，它的原理和架构。然后，需要注意的是，命令行接口已经被重构的比较厉害，同时论文示例引用的也是很早期的版本。本文包含了最新的API和使用示例。我在巴伐利亚科学与人文学院的莱布尼茨超级计算中心演示的幻灯片可以在researchgate找到。

The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest.
这个项目在布达佩斯的卡巴斯基实验室2018的SecurIT Cup赛事中赢得了第二名。

Netcap uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name Netcap was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 66 audit record types of which 55 are protocol specific and 8 are custom abstractions, such as flows or transferred files.
Netcap 使用 Google 的缓冲协议对输出进行编码，这点使得大部分编程语言可以直接进行访问。或者，输出可以作为逗号分隔值发出，这也是数据分析工具和系统常见的输入格式。这个工具支持扩展，并为新协议提供了多种添加支持的方式，同时以内存安全的方式实现了逻辑解析。它提供了关于观测到的流量更加高维的数据，同时使研究人员可以专注于试验检测网络环境恶意行为的新方法，而不是篡改数据收集机制和后续处理步骤。它具有利用多核架构的并发设计。选择 Netcap 这个名字是为了简单和直观。命令行工具的设计是考虑到了可用性和可读性，并在处理数据包时可以显示进度。目前最新版本提供了66种审核记录类型，其中55种为特定协议类型，8种为自定义抽象类型，比如流或传输文件等。

Design Goals

设计目标

• memory safety when parsing untrusted input
  解析不受信任输入时保障内存安全
• ease of extension
  容易扩展
• output format interoperable with many different programming languages
  可与许多不同编程语言相互操作的输出格式
• concurrent design
  并发设计
• output with small storage footprint on disk
  输出占用磁盘空间小
• gather everything, separate what can be understood from what can't
  收集一切，把能理解的不能理解的分开
• allow implementation of custom abstractions
  允许自定义抽象的实现
• rich platform and architecture support
  支持多个平台和架构

Framework Components

框架组件

The framework consists of 9 logically separate tools compiled into a single binary:
这个框架逻辑上由9个独立的工具组成，这些工具被编译成一个单独的二进制文件：

• capture (capture audit records live or from dumpfiles)
  捕获（实时或从转储文件捕获审核记录）
• dump (dump with audit records in various formats)
  转存（转储各种格式的审核记录）
• label (tool for creating labeled CSV datasets from netcap data)
  标签（用于从 netcap 数据创建标注的CSV数据集的工具）
• collect (collection server for distributed collection)
  搜集（用于分布式搜集的搜集器服务）
• agent (sensor agent for distributed collection)
  代理（用于分布式搜集的传感器代理）
• proxy (http reverse proxy for capturing traffic from web services)
  代理（用于捕获web服务流量的http反向代理）
• util (utility tool for validating audit records and converting timestamps)
  实用（用于验证审核记录和转换时间戳的实用工具）
• export (exporter for prometheus metrics)
  导出（用于普罗米修斯metrics的exporter）
• transform (maltego transformation plugin)
  转换（maltego转换插件）

Use Cases

用例

• monitoring honeypots
  监控蜜罐（一种安全工具）
• monitoring medical / industrial devices
  监控医疗/工业设备
• research on anomaly-based detection mechanisms
  研究基于异常的检测机制
• forensic data analysis
  取证数据分析

Demos

演示

A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways
一个从PCAP转储文件生成审核记录的简单演示，通过不同方法进行查询和展示搜集到的信息。
Working with PCAPs
使用PCAP

And live operation decoding traffic from my wireless network interface, while I am surfing the web
当我在上网时，实时操作解码来自无线网络接口的流量
Live Capture
实时捕获

Exploring HTTP audit records
探索HTTP审核记录
HTTP Audit Records
HTTP审核记录

Deep Learning

Watch a quick demo of the deep neural network for classification of malicious behavior, on a small PCAP dump file with traffic from the LOKI Bot. First, the PCAP file is parsed with netcap, in order to get audit records that will be labeled afterwards with the netlabel tool. The labeled CSV data for the TCP audit record type is then used for training (75%) and evaluation (25%) of the classification accuracy provided by the deep neural network.
Deep Learning with Tensorflow

License

Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. For more infos refer to the License page.

Source Code Stats

源码统计信息

Stats for netcap v0.5, generated with cloc version 1.80
netcap v0.5统计信息，使用 cloc 1.80版本生成

$ zeus cloc

     444 text files.
     444 unique files.                                          
     158 files ignored.

github.com/AlDanial/cloc v 1.84  T=0.26 s (1090.4 files/s, 116481.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                             277           4191           4788          21031
Markdown                         9            123              0            503
YAML                             1              5              4             14
-------------------------------------------------------------------------------
SUM:                           287           4319           4792          21548
-------------------------------------------------------------------------------

Installation

安装

Setup instructions
安装说明

Binary Distributions

二进制包

Compiled versions for macOS, Linux and Windows are available on GitHub:
GitHub上提供了macOS, Linux 和 Windows 的编译版本：
NETCAP GitHub Releases Page
NETCAP GitHub 发布页面

Go Get

Installation via go get:
通过 go get 方式安装：

$ go get -u github.com/dreadl0ck/netcap/...

Manual Build

手动构建

$ go build -ldflags "-s -w" -o /usr/local/bin/net -i github.com/dreadl0ck/netcap/cmd

Reproducible Builds via Go Modules

通过go模块实现可复制构建

In order to provide stable and reproducible builds, Go modules are used to pin the versions of source code dependencies to specific versions.
为了提供稳定和可复制的构建，go模块用于将源码依赖版本固定到特定版本上。

Go has included support for versioned modules as proposed here since 1.11. The initial prototype vgo was announced in February 2018. In July 2018, versioned modules landed in the main Go repository. They are used by default by the go toolchain starting from version 1.13 .
go从版本1.11开始，就正如这里提到的包含了对版本化模块的支持。vgo的最初原型于2018年2月发布，在2018年7月，版本化模块加入到go的主代码仓库中。go toolchain 从1.13版本开始会作为默认使用。

You can read about Go modules here:
你可以在此阅读 Go modules 相关的信息：
https://github.com/golang/go/wiki/Modules
https://blog.golang.org/using-go-modules

Development Build

开发构建

To install the command-line tool:
安装命令行工具：

$ go build -o /usr/local/bin/net -i github.com/dreadl0ck/netcap/cmd

Cross Compilation

To cross compile for other architectures, set the GOARCH and GOOS environment variables. For example to cross compile a binary for linux amd64:
要交叉编译其他（CPU）架构，需要设置 GOARCH 和 GOOS 环境变量。例如为linux arm64交叉编译一个二进制文件：

$ GOARCH=amd64 GOOS=linux go build -o bin/net -i github.com/dreadl0ck/netcap/cmd

Homebrew

On macOS, you can install the netcap command-line tool with Homebrew:
在macOS，你可以使用Homebrew来安装netcap命令行工具：

$ brew tap dreadl0ck/formulas
$ brew install netcap

Buildsystem

Netcap uses the zeus build system, it can be found on GitHub along with installation instructions:
ZEUS Build System GitHub
Netcap 使用 zeus 来构建系统，可以在 GitHub 上找到它以及安装说明：ZEUS Build System GitHub

To install the Netcap and Netlabel command-line tool and the library with zeus, run:
要使用 zeus 安装 Netcap 和 Netlabel 命令行工具和库，请运行：

$ zeus install

Quickstart

For those who can't wait to get their hands dirty.
对那些迫不及待想弄脏手的人。

Capture traffic to create audit records

捕获流量以创建审核记录

Read traffic live from interface, stop with Ctrl-C (SIGINT):
从接口读取实时流量，使用 Ctrl-C (SIGINT) 停止：

$ net capture -iface eth0

Read traffic from a dump file (supports PCAP or PCAPNG):
从 dump文件读取流量（支持PCAP 或 PCAPNG）

$ net capture -read traffic.pcap

Read audit records

读取审核记录

Read a netcap dumpfile and print to stdout as CSV:
读取 netcap dump文件并打印标准输出为CSV：

$ net dump -read TCP.ncap.gz

Show the available fields for a specific Netcap dump file:
显示 特定Netcap dump文件 可用字段：

$ net dump -fields -read TCP.ncap.gz

Print only selected fields and output as CSV:
仅打印所选字段并输出为CSV：

$ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort

Save CSV output to file:
保存CSV输出为文件：

$ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort > tcp.csv

Print output separated with tabs:
打印输出并用tabs制表符分隔：

$ net dump -read TPC.ncap.gz -tsv

Run with 24 workers and disable gzip compression and buffering:
启用24个workers运行并禁用gzip压缩和缓冲：

$ net capture -workers 24 -buf false -comp false -read traffic.pcapng

Parse pcap and write all data to output directory (will be created if it does not exist):
解析pcap并将所有数据写入到输出目录（如果目录不存在则会被创建）：

$ net capture -read traffic.pcap -out traffic_ncap

Convert timestamps to UTC:
将时间戳转换为UTC：

$ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,Dstport -utc

Show Audit Record File Header

显示审核记录文件头

To display the header of the supplied audit record file, the -header flag can be used:
要展示提供的审核记录文件头，可以使用 -header 标志：

$ net capture -read TCP.ncap.gz -header

+----------+---------------------------------------+
|  Field   |                Value                  |
+----------+---------------------------------------+
| Created  | 2018-11-15 04:42:22.411785 +0000 UTC  |
| Source   | Wednesday-WorkingHours.pcap           |
| Version  | v0.3.3                                |
| Type     | NC_TCP                                |
+----------+---------------------------------------+

Print Structured Audit Records

打印结构化的审核记录

Audit records can be printed structured, this makes use of the proto.MarshalTextString() function. This is sometimes useful for debugging, but very verbose.
审核日志可以被进行结构化打印，这点是利用了 proto.MarshalTextString() 函数。这有时用来调试很有用，但比较冗长。

$ net dump -read TCP.ncap.gz -struc
...
NC_TCP
Timestamp: "1499255023.848884"
SrcPort: 80
DstPort: 49472
SeqNum: 1959843981
AckNum: 3666268230
DataOffset: 5
ACK: true
Window: 1025
Checksum: 2348
PayloadEntropy: 7.836586993143013
PayloadSize: 1460
...

Print as CSV

以CSV形式打印

This is the default behavior. First line contains all field names.
这是默认行为，第一行会包括所有字段名。

$ net dump -read TCP.ncap.gz
Timestamp,SrcPort,DstPort,SeqNum,AckNum,DataOffset,FIN,SYN,RST,PSH,ACK,URG,...
1499254962.234259,443,49461,1185870107,2940396492,5,false,false,false,true,true,false,...
1499254962.282063,49461,443,2940396492,1185870976,5,false,false,false,false,true,false,...
...

Print as Tab Separated Values

以制表符分隔的值形式打印

To use a tab as separator, the -tsv flag can be supplied:
想使用制表符tab进行分隔，可以使用 -tsv 标识：

$ net dump -read TCP.ncap.gz -tsv
Timestamp               SrcPort DstPort Length  Checksum PayloadEntropy  PayloadSize
1499254962.084372       49792   1900    145     34831    5.19616448      137
1499254962.084377       49792   1900    145     34831    5.19616448      137
1499254962.084378       49792   1900    145     34831    5.19616448      137
1499254962.084379       49792   1900    145     34831    5.19616448      137
...

Print as Table

以表格形式打印

The -table flag can be used to print output as a table. Every 100 entries the table is printed to stdout.
打印输出为表格形式时可以使用 -table 标识。每100个条目，该表格会打印到标准输出。

$ net dump -read UDP.ncap.gz -table -select Timestamp,SrcPort,DstPort,Length,Checksum
+--------------------+----------+----------+---------+-----------+
|     Timestamp      | SrcPort  | DstPort  | Length  | Checksum  |
+--------------------+----------+----------+---------+-----------+
| 1499255691.722212  | 62109    | 53       | 43      | 38025     |
| 1499255691.722216  | 62109    | 53       | 43      | 38025     |
| 1499255691.722363  | 53       | 62109    | 59      | 37492     |
| 1499255691.722366  | 53       | 62109    | 59      | 37492     |
| 1499255691.723146  | 56977    | 53       | 43      | 7337      |
| 1499255691.723149  | 56977    | 53       | 43      | 7337      |
| 1499255691.723283  | 53       | 56977    | 59      | 6804      |
| 1499255691.723286  | 53       | 56977    | 59      | 6804      |
| 1499255691.723531  | 63427    | 53       | 43      | 17441     |
| 1499255691.723534  | 63427    | 53       | 43      | 17441     |
| 1499255691.723682  | 53       | 63427    | 87      | 14671     |
...

Print with Custom Separator

使用自定义分隔符打印

Output can also be generated with a custom separator:
输出也可以使用自定义分隔符生成：

$ net dump -read TCP.ncap.gz -sep ";"
Timestamp;SrcPort;DstPort;Length;Checksum;PayloadEntropy;PayloadSize
1499254962.084372;49792;1900;145;34831;5.19616448;137
1499254962.084377;49792;1900;145;34831;5.19616448;137
1499254962.084378;49792;1900;145;34831;5.19616448;137
...

Validate generated CSV output

验证生成的CSV输出

To ensure values in the generated CSV would not contain the separator string, the -check flag can be used.
为了确认生成的CSV值不包含分隔符字符串，可以使用 -check 标识。

This will determine the expected number of separators for the audit record type, and print all lines to stdout that do not have the expected number of separator symbols. The separator symbol will be colored red with ansi escape sequences and each line is followed by the number of separators in red color.
这个能用于确认审核记录类型的期望分隔符数量，并把所有不包含期望分隔符符号的行打印到标准输出。分隔符号会使用ansi转义序列标红，并且每行后面会有红色的分隔符数量。

The -sep flag can be used to specify a custom separator.
-seq 标识可以用来指定自定义分隔符。

$ net util -read TCP.ncap.gz -check
$ net util -read TCP.ncap.gz -check -sep=";"