jstorm单机部署和实时gps点统计实践
1 安装zookeeper
1.1 下载
wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
1.2 配置
配置zookeeper参数zoo.cfg，步骤如下：
a cp opt/zookeeper-3.4.6/conf/zoo_sample.cfg opt/zookeeper-3.4.6/conf/zoo.cfg
b 确认参数clientPort=2181（ 由于jstorm依赖zk服务,其storm.yaml默认zk的ip和port是：localhost:2181）
1.3 启动和检验
运行zookeeper-3.4.6/bin/zkServer.sh，如看到如下类似日志，表示单点zk服务启动成功：
|
2015-02-05 19:39:38,849 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg 2015-02-05 19:39:38,855 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2015-02-05 19:39:38,855 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2015-02-05 19:39:38,855 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2015-02-05 19:39:38,856 [myid:] - WARN [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode 2015-02-05 19:39:38,876 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg 2015-02-05 19:39:38,876 [myid:] - INFO [main:ZooKeeperServerMain@95] - Starting server |
2 安装jstorm
2.1 下载
wget http://42.121.19.155/jstorm/jstorm-0.9.6.2.zip
2.2 运行
opt/jstorm-0.9.6.2/bin/start.sh
2.3 查看
在/opt/jstorm-0.9.6.2/logs/可以分别看到nimbus.log和supervisor.log，其内容如下:
|
2015-02-06 00:23:19 JStormServerUtils:83 main] Successfully touch pid /opt/storm-test/jstorm-0.9.6.2/data/supervisor/pids/10520 |
3 安装jstorm-ui
3.1 下载tomcat
wget http://mirrors.cnnic.cn/apache/tomcat/tomcat-8/v8.0.15/bin/apache-tomcat-8.0.15.tar.gz
3.2 拷贝jstorm-ui
a cp jstorm-ui-0.9.6.2.war opt/apache-tomcat-8.0.17/webapps
b ln -s jstorm-ui-0.9.6.2 ROOT
3.3 启动
运行opt/apache-tomcat-8.0.17/bin/startup.sh
3.4 查看
访问 http://10.17.133.52:8080/cluster.jsf?clusterName=，可以看到jstorm如下监控页面:
 |
4 jstorm实践
4.1. 背景
a. 功能
实时接收gps数据采集点，按指定的时间维度(如1分钟)统计打印各个源头的gps个数
b. 编码
实现一个Spout:fpSubscriber和两个Bolt:fpCombiner和fpWriter，代码如下：
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("fpSubscriber", new FpSubscriber(), 1);
builder.setBolt("fpCombiner", new FpStat(), 1).shuffleGrouping("fpSubscriber");
builder.setBolt("fpWriter", new FpWriter(), 1).shuffleGrouping("fpCombiner");
Config conf = new Config();
conf.put(Config.NIMBUS_HOST, "localhost");
conf.put(Config.NIMBUS_THRIFT_PORT, 6627);
conf.setDebug(true);
conf.setNumWorkers(3);
if(args!=null && args.length > 0) {
StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
} else {
StormSubmitter.submitTopology("fp-stat", conf, builder.createTopology());
}
说明：
fpSubscriber：实时接收线上fp发布数据，推给下游fpCombiner的节点;
fpCombiner:进行gps按dscode的分组和统计，并定时分发下游fpWriter节点;
fpWriter：打印或持久化每个dscode的定期gpscount个数的统计结果.
4.2 部署
a. 打包并上传应用程序fp-stat.jar
b. 启动 ./bin/jstorm jar fp-stat.jar storm.test.FpStatTopology
4.3 展示验证
a. 界面展示storm-ui如下：
 |
可以看到一个fp-stat的topo任务,其状态为active；点击进入fp-stat，发现有三个worker,分别是一个Spout:fpSubscriber，两个Bolt:fpCombiner和fpWriter。
b. 结果显示
在fpWriter输出日志目录opt/ jstorm-0.9.6.2/logs/ fp-stat-worker-6801.log.out可以发现如下统计结果：
 |
