Metron pcap backend代码解析
Metron pcap backend代码解析
metron pcap backend 主要用于创建一个storm topology 处理流，将来自于Kafka的原始数据解析到HDFS中。
其数据流如下：
Pcap topology
这个topology 主要是读取数据，并将数据以顺序的方式写入到HDFS中。
主要流程如下：
抓取数据包，并发送到kafka中
数据发送到Kafka以后，pcap_backend提交到storm中的流处理任务将消费该数据，并将数据存储到HDFS中。
最后在运行另外一段解析的程序 pcap_inspector.sh。
可以看到数据包的一些基础信息。那么其具体实现流程如何，我们开始跟踪实现的具体步骤。
所谓的启动，其实主要是看 pcap_backend中的start_pcap_topology脚本。
metron-pcap-backend->src->main->scripts->start_pcap_topology
export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export TOPOLOGIES_JAR=${project.artifactId}-$METRON_VERSION.jar
storm jar $METRON_HOME/lib/$TOPOLOGIES_JAR org.apache.storm.flux.Flux --remote $METRON_HOME/flux/pcap/remote.yaml --filter $METRON_HOME/config/pcap.properties
由上可以看实际是通过flux的方式加载对应的yaml配置进行的项目提交。
pcap.properites:
spout.kafka.topic.pcap=pcap
topology.worker.childopts=
# kerberos token ticke 值配置
topology.auto-credentials=[]
topology.workers=1
kafka.zk=node1:2181
hdfs.sync.every=1
hdfs.replication.factor=-1
kafka.security.protocol=PLAINTEXT
# One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
kafka.pcap.start=UNCOMMITTED_EARLIEST
#一个文件里面保留的最大包数量
kafka.pcap.numPackets=1000
#配置这个保存包的时间
kafka.pcap.maxTimeMS=300000
kafka.pcap.ts_scheme=FROM_KEY
#HDFS中存储这些采集包的路径
kafka.pcap.out=/apps/metron/pcap
#时间单位
kafka.pcap.ts_granularity=MICROSECONDS
kafka.spout.parallelism=1
看了配置，在看topology的流配置。metron 中的storm统一采用Flunk进行配置。
name: "pcap"
config:
topology.workers: ${topology.workers}
topology.worker.childopts: ${topology.worker.childopts}
topology.auto-credentials: ${topology.auto-credentials}
components:
# Any kafka props for the producer go here.
- id: "kafkaProps"
className: "java.util.HashMap"
configMethods:
- name: "put"
args:
- "value.deserializer"
- "org.apache.kafka.common.serialization.ByteArrayDeserializer"
- name: "put"
args:
- "key.deserializer"
- "org.apache.kafka.common.serialization.ByteArrayDeserializer"
- name: "put"
args:
- "group.id"
- "pcap"
- name: "put"
args:
- "security.protocol"
- "${kafka.security.protocol}"
- id: "kafkaConfig"
className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder"
constructorArgs:
- ref: "kafkaProps"
# topic name
- "${spout.kafka.topic.pcap}"
- "${kafka.zk}"
configMethods:
- name: "setFirstPollOffsetStrategy"
args:
# One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
- ${kafka.pcap.start}
- id: "writerConfig"
className: "org.apache.metron.spout.pcap.HDFSWriterConfig"
configMethods:
- name: "withOutputPath"
args:
- "${kafka.pcap.out}"
- name: "withNumPackets"
args:
- ${kafka.pcap.numPackets}
- name: "withMaxTimeMS"
args:
- ${kafka.pcap.maxTimeMS}
- name: "withZookeeperQuorum"
args:
- "${kafka.zk}"
- name: "withSyncEvery"
args:
- ${hdfs.sync.every}
- name: "withReplicationFactor"
args:
- ${hdfs.replication.factor}
- name: "withDeserializer"
args:
- "${kafka.pcap.ts_scheme}"
- "${kafka.pcap.ts_granularity}"
spouts:
- id: "kafkaSpout"
className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout"
parallelism: ${kafka.spout.parallelism}
constructorArgs:
- ref: "kafkaConfig"
- ref: "writerConfig"
看了上面的配置，我们再核实一下：
在分析写入HDFSF代码前。我们先查看下，直接消费kafka里面数据的结果：
通过上图可以看出，Kafka中和我们原来想象的不太一样，不是pcap的格式。其具体格式内容的问题：可以参考笔者关于pcapy的博客，这里不再讲述。
根据配置我们可以看到起入口主函数为pcap下的KafkaToHDFSSpout（其关联了2个配置构造函数）
我们先看spout 源
public KafkaToHDFSSpout( SimpleStormKafkaBuilder<byte[], byte[]> spoutConfig
, HDFSWriterConfig config
)
{
super(spoutConfig
, HDFSWriterCallback.class
);
this.config = config;
}
该函数继承的是kafka
核心其实就是调用了HDFSWriteCallback这个回调函数。在继续跟踪到HDFSWriteCallback下面
import org.apache.storm.kafka.Callback;
import org.apache.storm.kafka.EmitContext;
import org.apache.storm.kafka.spout.KafkaSpoutConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* A callback which gets executed as part of the spout to write pcap data to HDFS.
*/
public class HDFSWriterCallback implements Callback {
该函数继承的是org.apache.storm.kafka.Callback下面的函数
public interface Callback extends AutoCloseable, Serializable {
List<Object> apply(List<Object> tuple, EmitContext context);
void initialize(EmitContext context);
}
可以看到先提交，再初始化，而在HDFSWrite对这个数据进行了重写。
@Override
public List<Object> apply(List<Object> tuple, EmitContext context) {
#读取流数据
byte[] key = (byte[]) tuple.get(0);
byte[] value = (byte[]) tuple.get(1);
long tsDeserializeStart = System.nanoTime();
#解析kafka中的Key-value值
KeyValueDeserializer.Result result = config.getDeserializer().deserializeKeyValue(key, value);
long tsDeserializeEnd = System.nanoTime();
if (LOG.isDebugEnabled() && !result.foundTimestamp) {
List<String> debugStatements = new ArrayList<>();
if (key != null) {
debugStatements.add("Key length: " + key.length);
debugStatements.add("Key: " + DatatypeConverter.printHexBinary(key));
} else {
debugStatements.add("Key is null!");
}
if (value != null) {
debugStatements.add("Value length: " + value.length);
debugStatements.add("Value: " + DatatypeConverter.printHexBinary(value));
} else {
debugStatements.add("Value is null!");
}
LOG.debug("Dropping malformed packet: {}", Joiner.on(" / ").join(debugStatements));
}
long tsWriteStart = System.nanoTime();
try {
getWriter(new Partition( topic
, context.get(EmitContext.Type.PARTITION))
).handle(result.key, result.value);
} catch (IOException e) {
LOG.error(e.getMessage(), e);
//drop? not sure..
}
long tsWriteEnd = System.nanoTime();
if(LOG.isDebugEnabled() && (Math.random() < 0.001 || !inited)) {
LOG.debug("Deserialize time (ns): {}", (tsDeserializeEnd - tsDeserializeStart));
LOG.debug("Write time (ns): {}", (tsWriteEnd - tsWriteStart));
}
inited = true;
return tuple;
}
上面有一段解析Kafka的方法函数deserializeKeyValue，获取其返回值result,（里面就是解析出来的发送的key和value,但需要注意的是这里的Key-value已经和发送的略微有了点变化），具体如下：
metron->spout->pcap->deserializer->FromKeyDeserializer
@Override
public Result deserializeKeyValue(byte[] key, byte[] value) {
if (key == null) {
throw new IllegalArgumentException("Expected a key but none provided");
}
#从前文可以知道Kafa发送的消息中的key也就是时间戳，这部分比较简单
long ts = converter.toNanoseconds(fromBytes(key));
#返回的第2部分是一个数据包头（addHeaders
return new Result(ts, PcapHelper.addHeaders(ts, value, endianness), true);
}
在上面代码中我们看到解析的值里面有一个addHeaders函数，注意这部分代码已经位于metron-pcap模块中了
#根据大小端生成不同的pcap 的gloabla header
public static byte[] getPcapGlobalHeader(Endianness endianness) {
if(swapBytes(endianness)) {
//swap
return new byte[] {
(byte) 0xd4, (byte) 0xc3, (byte) 0xb2, (byte) 0xa1 //swapped magic number 0xa1b2c3d4
, 0x02, 0x00 //swapped major version 2
, 0x04, 0x00 //swapped minor version 4
, 0x00, 0x00, 0x00, 0x00 //GMT to local tz offset (= 0)
, 0x00, 0x00, 0x00, 0x00 //sigfigs (= 0)
, (byte) 0xff, (byte) 0xff, 0x00, 0x00 //snaplen (=65535)
, 0x01, 0x00, 0x00, 0x00 // swapped link layer header type (1 = ethernet)
};
}
else {
//no need to swap
return new byte[] {
(byte) 0xa1, (byte) 0xb2, (byte) 0xc3, (byte) 0xd4 //magic number 0xa1b2c3d4
, 0x00, 0x02 //major version 2
, 0x00, 0x04 //minor version 4
, 0x00, 0x00, 0x00, 0x00 //GMT to local tz offset (= 0)
, 0x00, 0x00, 0x00, 0x00 //sigfigs (= 0)
, 0x00, 0x00, (byte) 0xff, (byte) 0xff //snaplen (=65535)
, 0x00, 0x00, 0x00, 0x01 // link layer header type (1 = ethernet)
};
}
}
......
public static byte[] addHeaders(long tsNano, byte[] packet, Endianness endianness) {
#endianness 是代表大小端属性
#这里的packet就是原始的数据包了
byte[] ret = new byte[GLOBAL_HEADER_SIZE + PACKET_HEADER_SIZE + packet.length];
#根据大小端生成pcap 格式中的globalHeader投
byte[] globalHeader = getPcapGlobalHeader(endianness);
int offset = 0;
System.arraycopy(globalHeader, 0, ret, offset, GLOBAL_HEADER_SIZE);
offset += globalHeader.length;
#packet 时间等信息，获取并移位填写
{
boolean swapBytes = swapBytes(endianness);
long micros = Long.divideUnsigned(tsNano, 1000);
int secs = (int)(micros / 1000000);
int usec = (int)(micros % 1000000);
int capLen = packet.length;
{
byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(secs):secs);
System.arraycopy(b, 0, ret, offset, Integer.BYTES);
offset += Integer.BYTES;
}
{
byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(usec):usec);
System.arraycopy(b, 0, ret, offset, Integer.BYTES);
offset += Integer.BYTES;
}
{
byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
System.arraycopy(b, 0, ret, offset, Integer.BYTES);
offset += Integer.BYTES;
}
{
byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
System.arraycopy(b, 0, ret, offset, Integer.BYTES);
offset += Integer.BYTES;
}
}
System.arraycopy(packet, 0, ret, offset, packet.length);
return ret;
}
根据代码所示，在写入到HDFS的文件中应该有一部分是pcap Header的头（24byte）,packer_hader(16byte),后面就是原始包才对。我们打开一个存入的HDFS文件进行验证：
从上图可以看到完全和存入到HDFS的文件能对上，但是在gloabal header前面还有115byte的数据，不知道是怎么来的？
在前面List函数中最后可以看到一句：
try {
getWriter(new Partition( topic
, context.get(EmitContext.Type.PARTITION))
).handle(result.key, result.value);
} catch (IOException e) {
LOG.error(e.getMessage(), e);
//drop? not sure..
}
....
继续跟到hadle代码中
public void handle(long ts, byte[] value) throws IOException {
turnoverIfNecessary(ts);
BytesWritable bw = new BytesWritable(value);
try {
writer.append(new LongWritable(ts), bw);
}
catch(ArrayIndexOutOfBoundsException aioobe) {
LOG.warn("This appears to be HDFS-7765 (https://issues.apache.org/jira/browse/HDFS-7765), " +
"which is an issue with syncing and not problematic: {}", aioobe.getMessage(), aioobe);
}
numWritten++;
if(numWritten % config.getSyncEvery() == 0) {
syncHandler.sync(outputStream);
}
}
似乎什么也没做，但是有一个很不起眼的函数turnoverIfNecessary
private void turnoverIfNecessary(long ts, boolean force) throws IOException {
long duration = ts - batchStartTime;
boolean initial = outputStream == null;
boolean overDuration = config.getMaxTimeNS() <= 0 ? false : Long.compareUnsigned(duration, config.getMaxTimeNS()) >= 0;
boolean tooManyPackets = numWritten >= config.getNumPackets();
if(force || initial || overDuration || tooManyPackets ) {
//turnover
Path path = getPath(ts);
close();
if(fs instanceof LocalFileSystem) {
outputStream = new FSDataOutputStream(new FileOutputStream(new File(path.toString())));
syncHandler = SyncHandlers.LOCAL.getHandler();
}
else {
outputStream = fs.create(path, true);
if (outputStream instanceof HdfsDataOutputStream) {
if (initial) {
LOG.info("Using the HDFS sync handler.");
}
syncHandler = SyncHandlers.HDFS.getHandler();
} else {
if (initial) {
LOG.info("Using the default sync handler, which cannot guarantee atomic appends at the record level, be forewarned!");
}
syncHandler = SyncHandlers.DEFAULT.getHandler();
}
}
writer = SequenceFile.createWriter(this.fsConfig
, SequenceFile.Writer.keyClass(LongWritable.class)
, SequenceFile.Writer.valueClass(BytesWritable.class)
, SequenceFile.Writer.stream(outputStream)
, SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE)
);
//reset state
LOG.info("Turning over and writing to {}: [duration={} NS, force={}, initial={}, overDuration={}, tooManyPackets={}]", path, duration, force, initial, overDuration, tooManyPackets);
batchStartTime = ts;
numWritten = 0;
}
}
}
看到没，writer 写入了一堆序列化的信息到里面去，但注意根据代码这段只有在创建时才有效，所以一个保存的文件，也只有一个序列化信息，这点和现状也刚好吻合。
这是否就是HDFS文件中的内容了，可以看到开始写入的是LongWriteable的类名（org.apache.hadoop.io.LongWriteable）和BytesWritable的类名（org.apache.hadoop.io.ByteWritable）
