在CentOS上搭建Storm集群

 

Here's a summary of the steps for setting up a Storm cluster:

Set up a Zookeeper cluster
Install dependencies on Nimbus and worker machines
Download and extract a Storm release to Nimbus and worker machines
Fill in mandatory configurations into storm.yaml
Launch daemons under supervision using "storm" script and a supervisor of your choice

 

1 基本环境

操作系统：CentOS-6.5-x86_64-bin-DVD1

　　JDK：java version "1.7.0_67"

2 安装ZOOKEEPER-3.4.6

 见http://www.cnblogs.com/fesh/p/3900253.html

3 安装Storm依赖库（ZeroMQ、JZMQ、Python）

3.1 安装ZEROMQ-4.0.5

tar -zxvf zeromq-4.0.5.tar.gz
cd zeromq-4.0.5
./configure
make
make install

3.2 安装JZMQ

git clone https://github.com/nathanmarz/jzmq.git
./autogen.sh
./configure
make
make install

3.3 安装PYTHON-3.4.2

./configure
make
make test
sudo make install

测试

python -V

4 配置STORM-0.9.3

配置$STORM_HOME/conf/storm.yaml
1) storm.zookeeper.servers:This is a list of the hosts in the Zookeeper cluster for your Storm cluster. It should look something like:

storm.zookeeper.servers:
  - "111.222.333.444"
  - "555.666.777.888"

如果Zookeeper集群使用的不是默认端口，那么还需要设置storm.zookeeper.port选项。

2) storm.local.dir: The Nimbus and Supervisor daemons require a directory on the local disk to store small amounts of state (like jars, confs, and things like that).

storm.local.dir: "/mnt/storm"

3) nimbus.host:The worker nodes need to know which machine is the master in order to download topology jars and confs.

nimbus.host: "111.222.333.44"

4) supervisor.slots.ports: For each worker machine, you configure how many workers run on that machine with this config. Each worker uses a single port for receiving messages, and this setting defines which ports are open for use. If you define five ports here, then Storm will allocate up to five workers to run on this machine. If you define three ports, Storm will only run up to three. By default, this setting is configured to run 4 workers on the ports 6700, 6701, 6702, and 6703.

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

设置完成后，scp -r storm cluster2:/opt/ 将文件分发到其他集群。

5 启动STORM集群

Nimbus: Run the command "bin/storm nimbus" under supervision on the master machine.

Supervisor: Run the command "bin/storm supervisor" under supervision on each worker machine. The supervisor daemon is responsible for starting and stopping worker processes on that machine.

UI: Run the Storm UI (a site you can access from the browser that gives diagnostics on the cluster and topologies) by running the command "bin/storm ui" under supervision. The UI can be accessed by navigating your web browser to http://{nimbus host}:8080.

 启动Nimbus后台程序:

storm nimbus&

启动Supervisor后台程序:

storm supervisor& 

 6 向集群提交任务

1）启动Storm Topology

storm jar allmycode.jar org.me.MyTopology arg1 arg2 arg3

2）停止Storm Topology

storm kill {toponame}

其中，{toponame}为Topology提交到Storm集群时指定的Topology任务名称。

 

参考：

1、http://www.cnblogs.com/panfeng412/archive/2012/11/30/how-to-install-and-deploy-storm-cluster.html

2、http://storm.apache.org/documentation/Setting-up-a-Storm-cluster.html