Storm 配置图文解析
Storm 配置图文解析
參考阅读：http://www.xiaofateng.com/?
==============================
| sample-topology |
| ------------------------ | Task 1 Task 2 Task 3
| | Worker Process 1 | | T1 T2 T3
| | +--------+ | | Spout => Bolt => Bolt
| | +------+ | +----+ | | | parallelism parallelism parallelism
| | | T3 | | | T2 | | | | hint=2 hint=2 hint=6
| | +------+ | +----+ | | |
| | | +----+ | | | combined parallelism = 2 + 2 + 6 = 10
| | +------+ | | T2 | | | |
| | | T3 | | +----+ | | | Each of the 2 worker processes will spawn 10/2=5 threads
| | +------+ +--------+ | |
| | | | T1: parallelism hint = initial executors
| | +------+ +--------+ | |
| | | T3 | | T1 | | | T2: the T2 bolt was configured to use 2 executors and four tasks.
| | +------+ +--------+ | | For this reason each executor runs two tasks for this bolt.
| ------------------------ |
| |
| ------------------------ | Config conf = new Config();
| | Worker Process 2 | |
| | +--------+ | | // run as 2 workers on 2 supervisors
| | +------+ | +----+ | | | conf.setNumWorkers(2);
| | | T3 | | | T2 | | | |
| | +------+ | +----+ | | | // T1: 2 executors for spout
| | | +----+ | | | topologyBuilder.setSpout("T1-spout", new T1Spout(), 2);
| | +------+ | | T2 | | | |
| | | T3 | | +----+ | | | // T2: 2 executors for bolt with total 4 tasks
| | +------+ +--------+ | | topologyBuilder.setBolt("T2-bolt", new T2Bolt(), 2)
| | | | .setNumTasks(4).shuffleGrouping("T1-spout");
| | +------+ +--------+ | | // T3: 6 executors for bolt (default 1 task for 1 executor)
| | | T3 | | T1 | | | topologyBuilder.setBolt("T3-bolt", new T3Bolt(), 6).shuffleGrouping("T2-bolt");
| | +------+ +--------+ | |
| ------------------------ | StormSubmitter.submitTopology("sample-topology", conf, topologyBuilder.createTopology());
==============================
说明：
一个worker进程(process)会产生N个线程(executor)，那么并行度(parallelism)就是全部线程的数目。setNumWorkers
任务(task)是线程运行的工作队列。线程的任务数说明线程的吞吐能力。一个线程的各个任务之间并非并发的。
setNumTasks
线程(executor)是运行任务的上下文环境。
參照上图理解各个配置的含义。
