Flume性能测试报告(翻译Flume官方wiki报告)
因使用flume的时候总是会对其性能有所调研，网上找的要么就是自测的
这里找到一份官方wiki的测试报告供大家参考
https://cwiki.apache.org/confluence/display/FLUME/Performance+Measurements+-+round+2
测试环境:
以下测试基于单个agent
hadoop集群配置：20-node Hadoop cluster (1 name node and 19 data nodes).
服务器配置: 24 cores – Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM, 7200 rpm Hard Drive.
1. File channel with HDFS Sink (Sequence File):
基于1.4版本的flume测试，source为4个exec，channel为file，sink为hdfs
Flume version: 1.4
Source: 4 x Exec Source, 100k batchSize
HDFS Sink Batch size: 500,000
Event Size: 500 byte events.
Channel: File
| Events/Sec | ||||||
| Sinks | 1 data dirs | 2 data dirs | 4 data dirs | 6 data dirs | 8 data dirs | 10 data dirs |
| 1 | 14.3k(7Mb/s) | |||||
| 2 | 21.9k | |||||
| 4 | 35.8k | |||||
| 8 | 72.5k | 77k | 78.6(37Mb/s) | 76.6k | ||
| 10 | 58k | |||||
| 12 | 49.3k | 49k | ||||
Measurements were taken to get an idea around the configuration that yields best performance. So took measurements only for all data points in the grid that made sense. For example it was not necessary to take measurements for multiple dataDirs at single sink, as it was evident multiple HDFS sink would better than single sink config.
混合的多sinks要比单sink的效果好
2. HDFS Sink:
相比1使用了内存channel ，memory channel
Flume version: 1.4
Channel: Memory
Event Size: 500 byte events.
| #hdfs sinks |
snappy batch sz:1.2mill |
snappy batch sz:1.4mill |
Sequence File batch sz:1.2mill |
| 1 | 34.3k(17Mb/s) | 33k | 33k |
| 2 |
71k |
75k | 69k |
| 4 | 141k | 145k | 141k |
| 8 | 271k | 273k | 251k |
| 12 | 382k | 380k | 370k |
| 16 | 478k | 538k(240M/s) | 486k(232M/s) |
Some simple observations:
- increasing number of dataDirs helps FC perf even on single disk systems
- Increasing number of sinks helps
提高sink的数量是有显著效果的
3. Hive Sink:
hive sink ，channel为内存，flume版本为1.5或者1.6
Flume version: 1.5 & 1.6
Channel: Memory
BatchSz:1million
Event Size: 500 byte events.
| Flume 1.5 | Flume 1.6 | |||
| Events/s | Mps | Events/s | Mps | |
| 1 Sink | ||||
| DELIMITED Text | 36,885 | 18 | 138,461 | 66 |
| Json | 12,735 | 6 | ||
| 16 sinks(agent maxed out) | ||||
| DELIMITED Text | 209,600 | 100 | 348,214 | 166 |
| Json | 25,751 | 12 | 31,135 | 14 |
Observation: Feeding JSON data to Hive sink is much slower, potentially due to higher parsing overhead of JSON in part.
发送json数据格式会慢一些，主要是慢在json的解析上
4. HBase Sink:
Flume version: 1.5
Channel: Memory
Serializer: RegexHbaseEventSerializer
Total Sinks: 1
| Event Size(bytes) | Batch Sz:1 | Batch Sz:100 | Batch Sz:1000 | Batch Sz:10000 |
| 500 | 11mb/s | 11mb/s | ||
| 1000 | 0.5bB/s | 14/mb/s | 22mb/s | 27mb/s |
5. ASync HBase Sink:
Flume version: 1.5
Channel: Memory
Serializer: SimpleAsyncHbaseEventSerializer
Total Sinks: 1
| Event Size(bytes) | Batch Sz:1 | Batch Sz:100 | Batch Sz:1000 |
| 500 | 0.4mb/s | 0.5mb/s | |
| 1000 | 0.8mb/s | 0.8mb/s | 0.9mb/s |
6. Kafka Source:
Flume version: 1.6
Channel: Memory
Sink: Null Sink
Event Size: 1000 bytes
Total Sinks: 1
|
Batch Size (bytes) |
Mb/s |
| 1,000 | 62 |
| 10,000 | 112 |
| 20,000 | 125 |
| 40,000 | 147 |
| 80,000 | 153 |
作 者：小闪电
出处：http://www.cnblogs.com/yueyanyu/
本文版权归作者和博客园共有，欢迎转载、交流，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文链接。如果觉得本文对您有益，欢迎点赞、欢迎探讨。本博客来源于互联网的资源，若侵犯到您的权利，请联系博主予以删除。
