Flink SideOutput 和 Filter 分流对比

Flink 分流有Filter、Split（已经废弃移除）、Side Output进行分流，到底时有什么区别，哪个种更好呢？

对比

代码对比

直接上代码对比：

import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
public class FlinkSideOutput {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment();
        executionEnvironment.setParallelism(1);
        DataStreamSource<String> streamSource = executionEnvironment.addSource(new SocketTextStreamFunction("localhost", 9999, "\n", 1));
        OutputTag<String> zero = new OutputTag<String>("zero") {
        };
        SingleOutputStreamOperator<String> process = streamSource.process(new ProcessFunction<String, String>() {
            @Override
            public void processElement(String s, ProcessFunction<String, String>.Context context, Collector<String> collector) throws Exception {
                System.out.println("SideOuput:" + s);
                if (Integer.parseInt(s) % 2 == 0) {
                    context.output(zero, s);
                } else {
                    collector.collect(s);
                }
            }
        });
        process.print();
        process.getSideOutput(zero).print();
        System.out.println(executionEnvironment.getExecutionPlan());
        executionEnvironment.execute();
    }
}

import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction;
public class FlinkStreamFilter {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment();
        executionEnvironment.setParallelism(1);
        DataStreamSource<String> streamSource = executionEnvironment.addSource(new SocketTextStreamFunction("localhost", 9999, "\n", 1));
        streamSource.filter(s -> {
            System.out.println("filter % 2 == 0 ,value:" + s);
            return Integer.parseInt(s) % 2 == 0;
        }).print();
        streamSource.filter(s -> {
            System.out.println("filter % 2 != 0 ,value:" + s);
            return Integer.parseInt(s) % 2 != 0;
        }).print();
        System.out.println(executionEnvironment.getExecutionPlan());
        executionEnvironment.execute();
    }
}

从代码上看，SideOutput的写法更为麻烦，且对于代码理解上Filter 更容易去理解，如果了解Spark的可能会想到Filter 这边可能会对 streamSource 进行两次计算。

执行图对比

当全局 parallelism = 1 时

从SideOutput 这个执行图Name看到

Custom Source -> Process , Process 分流到两个Sink:Print

从Filter 这个执行图的Name 可以看到

Custom Source 分成两个流，Custom Source -> Filter -> Sink:Print , Custom Source -> Filter -> Sink:Print

当把 Sink的 Parallelism = 8 ，刚刚说的就更明显了。

比Operator 图对比

通过打印计划

executionEnvironment.getExecutionPlan()

SideOuput：

{
  "nodes" : [ {
    "id" : 1,
    "type" : "Source: Custom Source",
    "pact" : "Data Source",
    "contents" : "Source: Custom Source",
    "parallelism" : 1
  }, {
    "id" : 2,
    "type" : "Process",
    "pact" : "Operator",
    "contents" : "Process",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 1,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  }, {
    "id" : 3,
    "type" : "Sink: Print to Std. Out",
    "pact" : "Data Sink",
    "contents" : "Sink: Print to Std. Out",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 2,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  }, {
    "id" : 5,
    "type" : "Sink: Print to Std. Out",
    "pact" : "Data Sink",
    "contents" : "Sink: Print to Std. Out",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 2,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  } ]
}

对应的图就是：

Filter：

{
  "nodes" : [ {
    "id" : 1,
    "type" : "Source: Custom Source",
    "pact" : "Data Source",
    "contents" : "Source: Custom Source",
    "parallelism" : 1
  }, {
    "id" : 2,
    "type" : "Filter",
    "pact" : "Operator",
    "contents" : "Filter",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 1,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  }, {
    "id" : 4,
    "type" : "Filter",
    "pact" : "Operator",
    "contents" : "Filter",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 1,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  }, {
    "id" : 3,
    "type" : "Sink: Print to Std. Out",
    "pact" : "Data Sink",
    "contents" : "Sink: Print to Std. Out",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 2,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  }, {
    "id" : 5,
    "type" : "Sink: Print to Std. Out",
    "pact" : "Data Sink",
    "contents" : "Sink: Print to Std. Out",
    "parallelism" : 1,
    "predecessors" : [ {
      "id" : 4,
      "ship_strategy" : "FORWARD",
      "side" : "second"
    } ]
  } ]
}

这样就很明显了，Filter分流，Source会把数据发两次，一次给ID=2 的FilterOperator ，一次给ID=4 的FilterOperator，然后进行过滤分别转发给下一个Sink，而 SideOutput，Source直接把数据分发给Process，Process然后分别在内部根据逻辑转发给不同的Sink。

这下就很清楚了，SideOutput 是优于Filter ，Filter的方式会对数据进行两个过滤，SideOutput是只有一次。

验证

nc -lk 9999
1
2
3
4
5
6
7
8
9
10

执行代码

SideOutput输出：

SideOuput:1
1
SideOuput:2
2
SideOuput:3
3
SideOuput:4
4
SideOuput:5
5
SideOuput:6
6
SideOuput:7
7
SideOuput:8
8
SideOuput:9
9
SideOuput:0
0

Filter输出：

filter % 2 == 0 ,value:1
filter % 2 != 0 ,value:1
1
filter % 2 == 0 ,value:2
2
filter % 2 != 0 ,value:2
filter % 2 == 0 ,value:3
filter % 2 != 0 ,value:3
3
filter % 2 == 0 ,value:4
4
filter % 2 != 0 ,value:4
filter % 2 == 0 ,value:5
filter % 2 != 0 ,value:5
5
filter % 2 == 0 ,value:6
6
filter % 2 != 0 ,value:6
filter % 2 == 0 ,value:7
filter % 2 != 0 ,value:7
7
filter % 2 == 0 ,value:8
8
filter % 2 != 0 ,value:8
filter % 2 == 0 ,value:9
filter % 2 != 0 ,value:9
9
filter % 2 == 0 ,value:0
0
filter % 2 != 0 ,value:0