Storm Trident示例partitionAggregate
partitionAggregate是针对于每个partition,而不是每个batch,对每个partition当中的tuple做聚合
省略部分代码,省略部分可参考:https://blog.csdn.net/nickta/article/details/79666918
FixedBatchSpout spout = new FixedBatchSpout(new Fields("user", "score"), 3, new Values("nickt1", 4), new Values("nickt2", 7), new Values("nickt1", 8), new Values("nickt4", 9), new Values("nickt4", 7), new Values("nickt1", 11), new Values("nickt7", 5) ); spout.setCycle(false); TridentTopology topology = new TridentTopology(); topology.newStream("spout1", spout) .partitionBy(new Fields("user")) .partitionAggregate(new Fields("score", "user"), new Sum(), new Fields("sum")) .parallelismHint(5) .each(new Fields("sum"),new Debug("print:"));
以上代码
batch1的内容为:
nickt1 4
nickt2 7
nickt1 8
batch2的内容为:
nickt4 9
nickt4 7
nickt1 11
batch3的内容为:
nickt7 5
因为是按partitionBy(new Fields("user"))进行repartition, 以batch1内容为例:
值为nickt1的tuples会被分配到目标partition当中去,
值为nickt2的tuple会被分配到另外一个目标partition当中去(也有可能分配到同一个,因为是按Target Partition = hash(fields) % (number of target partition)算法)
我们于是可以推测:输出结果当中,就包括有nickt1的score相加为12, 另一个为7。
batch2,batch3的结果,可自行分析。
实际输出结果如下,与推测相同:
<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [7]
<Fri Mar 23 18:09:53 CST 2018[partition0-Thread-60-b-0-executor[33 33]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition3-Thread-114-b-0-executor[36 36]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition1-Thread-132-b-0-executor[34 34]]> DEBUG(print:): [12]
<Fri Mar 23 18:09:53 CST 2018[partition4-Thread-136-b-0-executor[37 37]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition3-Thread-114-b-0-executor[36 36]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition1-Thread-132-b-0-executor[34 34]]> DEBUG(print:): [11]
<Fri Mar 23 18:09:53 CST 2018[partition4-Thread-136-b-0-executor[37 37]]> DEBUG(print:): [16]
<Fri Mar 23 18:09:53 CST 2018[partition0-Thread-60-b-0-executor[33 33]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [5]