闲来无事，编个Flink 1.9 玩玩
1、下载flink、flink-shaded 源码、解压
[venn@venn release]$ ll total 19880 drwxrwxr-x. 44 venn venn 4096 Sep 12 14:00 flink-release-1.9.0 drwxrwxr-x. 13 venn venn 4096 Sep 11 11:05 flink-shaded-release-7.0 -rw-rw-r--. 1 venn venn 20318507 Sep 12 09:24 release-1.9.0.tar.gz -rw-rw-r--. 1 venn venn 28196 Sep 11 11:05 release-7.0.tar.gz
最开始直接拿flink 的 releast-1.9分支，发现都带了SNAPSHOT，遂放弃
flink-shaded 包含flink 的很多依赖，比如 flink-shaded-hadoop-2，中央仓库只提供了几个hadoop 版本的包，可能没有与自己hadoop 对应的 flink-shaded-hadoop-2 的包。flink1.9 对应的flink-shaded 版本是 7.0
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<version>${hadoop.version}-${flink.shaded.version}</version>
<optional>true</optional>
</dependency>
我用的hadoop 版本是2.7.7，然而并没有（2.7.5应该可以用）。
2、编译flink-shaded
指定hadoop版本2.7.7， 把编译好的包打到本地maven仓库，后面编译flink 用
[venn@venn flink-shaded-release-7.0]$ mvn clean install -DskipTests -Dhadoop.version=2.7.7 [INFO] Reactor Summary: [INFO] [INFO] flink-shaded 7.0 ................................... SUCCESS [ 3.483 s] [INFO] flink-shaded-force-shading 7.0 ..................... SUCCESS [ 0.597 s] [INFO] flink-shaded-asm-6 6.2.1-7.0 ....................... SUCCESS [ 0.627 s] [INFO] flink-shaded-guava-18 18.0-7.0 ..................... SUCCESS [ 0.994 s] [INFO] flink-shaded-netty-4 4.1.32.Final-7.0 .............. SUCCESS [ 3.469 s] [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-7.0 SUCCESS [ 0.572 s] [INFO] flink-shaded-jackson-parent 2.9.8-7.0 .............. SUCCESS [ 0.035 s] [INFO] flink-shaded-jackson-2 2.9.8-7.0 ................... SUCCESS [ 1.479 s] [INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-7.0 . SUCCESS [ 0.857 s] [INFO] flink-shaded-hadoop-2 2.7.7-7.0 .................... SUCCESS [ 9.132 s] [INFO] flink-shaded-hadoop-2-uber 2.7.7-7.0 ............... SUCCESS [ 17.173 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 38.612 s [INFO] Finished at: 2019-09-12T15:14:42+08:00 [INFO] ------------------------------------------------------------------------
3、编译 flink
指定hadoop 版本 2.7.7
[venn@venn flink-release-1.9.0]$ mvn clean package -DskipTests -Dhadoop.version=2.7.7 [INFO] Reactor Summary for flink 1.9.0: [INFO] [INFO] force-shading ...................................... SUCCESS [ 1.281 s] [INFO] flink .............................................. SUCCESS [ 1.829 s] [INFO] flink-annotations .................................. SUCCESS [ 1.280 s] [INFO] flink-shaded-curator ............................... SUCCESS [ 1.501 s] [INFO] flink-metrics ...................................... SUCCESS [ 0.188 s] [INFO] flink-metrics-core ................................. SUCCESS [ 0.736 s] [INFO] flink-test-utils-parent ............................ SUCCESS [ 0.157 s] [INFO] flink-test-utils-junit ............................. SUCCESS [ 0.928 s] [INFO] flink-core ......................................... SUCCESS [ 33.771 s] [INFO] flink-java ......................................... SUCCESS [ 4.827 s] [INFO] flink-queryable-state .............................. SUCCESS [ 0.100 s] [INFO] flink-queryable-state-client-java .................. SUCCESS [ 0.643 s] [INFO] flink-filesystems .................................. SUCCESS [ 0.082 s] [INFO] flink-hadoop-fs .................................... SUCCESS [ 1.168 s] [INFO] flink-runtime ...................................... SUCCESS [01:58 min] [INFO] flink-scala ........................................ SUCCESS [ 29.960 s] [INFO] flink-mapr-fs ...................................... SUCCESS [ 0.394 s] [INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [ 2.916 s] [INFO] flink-s3-fs-base ................................... SUCCESS [ 5.740 s] [INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 8.235 s] [INFO] flink-s3-fs-presto ................................. SUCCESS [ 12.510 s] [INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 13.618 s] [INFO] flink-oss-fs-hadoop ................................ SUCCESS [ 4.483 s] [INFO] flink-azure-fs-hadoop .............................. SUCCESS [ 5.729 s] [INFO] flink-optimizer .................................... SUCCESS [ 11.589 s] [INFO] flink-clients ...................................... SUCCESS [ 1.179 s] [INFO] flink-streaming-java ............................... SUCCESS [ 8.882 s] [INFO] flink-test-utils ................................... SUCCESS [ 1.453 s] [INFO] flink-runtime-web .................................. SUCCESS [02:06 min] [INFO] flink-examples ..................................... SUCCESS [ 0.112 s] [INFO] flink-examples-batch ............................... SUCCESS [ 10.577 s] [INFO] flink-connectors ................................... SUCCESS [ 0.096 s] [INFO] flink-hadoop-compatibility ......................... SUCCESS [ 4.225 s] [INFO] flink-state-backends ............................... SUCCESS [ 0.070 s] [INFO] flink-statebackend-rocksdb ......................... SUCCESS [ 1.102 s] [INFO] flink-tests ........................................ SUCCESS [ 28.820 s] [INFO] flink-streaming-scala .............................. SUCCESS [ 26.143 s] [INFO] flink-table ........................................ SUCCESS [ 0.054 s] [INFO] flink-table-common ................................. SUCCESS [ 1.771 s] [INFO] flink-table-api-java ............................... SUCCESS [ 1.416 s] [INFO] flink-table-api-java-bridge ........................ SUCCESS [ 0.570 s] [INFO] flink-table-api-scala .............................. SUCCESS [ 5.054 s] [INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 7.626 s] [INFO] flink-sql-parser ................................... SUCCESS [ 3.073 s] [INFO] flink-libraries .................................... SUCCESS [ 0.056 s] [INFO] flink-cep .......................................... SUCCESS [ 2.437 s] [INFO] flink-table-planner ................................ SUCCESS [01:37 min] [INFO] flink-orc .......................................... SUCCESS [ 0.579 s] [INFO] flink-jdbc ......................................... SUCCESS [ 0.546 s] [INFO] flink-table-runtime-blink .......................... SUCCESS [ 3.868 s] [INFO] flink-table-planner-blink .......................... SUCCESS [02:02 min] [INFO] flink-hbase ........................................ SUCCESS [ 1.725 s] [INFO] flink-hcatalog ..................................... SUCCESS [ 4.158 s] [INFO] flink-metrics-jmx .................................. SUCCESS [ 0.272 s] [INFO] flink-connector-kafka-base ......................... SUCCESS [ 1.671 s] [INFO] flink-connector-kafka-0.9 .......................... SUCCESS [ 0.768 s] [INFO] flink-connector-kafka-0.10 ......................... SUCCESS [ 0.547 s] [INFO] flink-connector-kafka-0.11 ......................... SUCCESS [ 0.692 s] [INFO] flink-formats ...................................... SUCCESS [ 0.054 s] [INFO] flink-json ......................................... SUCCESS [ 0.332 s] [INFO] flink-connector-elasticsearch-base ................. SUCCESS [ 0.810 s] [INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 8.239 s] [INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 9.715 s] [INFO] flink-connector-elasticsearch6 ..................... SUCCESS [ 0.747 s] [INFO] flink-csv .......................................... SUCCESS [ 0.251 s] [INFO] flink-connector-hive ............................... SUCCESS [ 1.942 s] [INFO] flink-connector-rabbitmq ........................... SUCCESS [ 0.332 s] [INFO] flink-connector-twitter ............................ SUCCESS [ 1.304 s] [INFO] flink-connector-nifi ............................... SUCCESS [ 0.349 s] [INFO] flink-connector-cassandra .......................... SUCCESS [ 2.205 s] [INFO] flink-avro ......................................... SUCCESS [ 1.565 s] [INFO] flink-connector-filesystem ......................... SUCCESS [ 0.861 s] [INFO] flink-connector-kafka .............................. SUCCESS [ 0.973 s] [INFO] flink-connector-gcp-pubsub ......................... SUCCESS [ 0.582 s] [INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [ 4.583 s] [INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [ 0.256 s] [INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [ 0.312 s] [INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [ 0.413 s] [INFO] flink-sql-connector-kafka .......................... SUCCESS [ 0.641 s] [INFO] flink-connector-kafka-0.8 .......................... SUCCESS [ 0.700 s] [INFO] flink-avro-confluent-registry ...................... SUCCESS [ 0.843 s] [INFO] flink-parquet ...................................... SUCCESS [ 0.726 s] [INFO] flink-sequence-file ................................ SUCCESS [ 0.247 s] [INFO] flink-examples-streaming ........................... SUCCESS [ 10.970 s] [INFO] flink-examples-table ............................... SUCCESS [ 5.554 s] [INFO] flink-examples-build-helper ........................ SUCCESS [ 0.069 s] [INFO] flink-examples-streaming-twitter ................... SUCCESS [ 0.464 s] [INFO] flink-examples-streaming-state-machine ............. SUCCESS [ 0.298 s] [INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [ 3.640 s] [INFO] flink-container .................................... SUCCESS [ 0.304 s] [INFO] flink-queryable-state-runtime ...................... SUCCESS [ 0.555 s] [INFO] flink-end-to-end-tests ............................. SUCCESS [ 0.056 s] [INFO] flink-cli-test ..................................... SUCCESS [ 0.151 s] [INFO] flink-parent-child-classloading-test-program ....... SUCCESS [ 0.154 s] [INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [ 0.098 s] [INFO] flink-dataset-allround-test ........................ SUCCESS [ 0.153 s] [INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [ 0.141 s] [INFO] flink-datastream-allround-test ..................... SUCCESS [ 1.119 s] [INFO] flink-batch-sql-test ............................... SUCCESS [ 0.136 s] [INFO] flink-stream-sql-test .............................. SUCCESS [ 0.183 s] [INFO] flink-bucketing-sink-test .......................... SUCCESS [ 0.233 s] [INFO] flink-distributed-cache-via-blob ................... SUCCESS [ 0.138 s] [INFO] flink-high-parallelism-iterations-test ............. SUCCESS [ 5.724 s] [INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [ 0.636 s] [INFO] flink-queryable-state-test ......................... SUCCESS [ 1.688 s] [INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [ 0.221 s] [INFO] flink-elasticsearch2-test .......................... SUCCESS [ 3.299 s] [INFO] flink-elasticsearch5-test .......................... SUCCESS [ 3.969 s] [INFO] flink-elasticsearch6-test .......................... SUCCESS [ 2.275 s] [INFO] flink-quickstart ................................... SUCCESS [ 0.781 s] [INFO] flink-quickstart-java .............................. SUCCESS [ 0.350 s] [INFO] flink-quickstart-scala ............................. SUCCESS [ 0.145 s] [INFO] flink-quickstart-test .............................. SUCCESS [ 0.323 s] [INFO] flink-confluent-schema-registry .................... SUCCESS [ 1.202 s] [INFO] flink-stream-state-ttl-test ........................ SUCCESS [ 3.172 s] [INFO] flink-sql-client-test .............................. SUCCESS [ 0.671 s] [INFO] flink-streaming-file-sink-test ..................... SUCCESS [ 0.221 s] [INFO] flink-state-evolution-test ......................... SUCCESS [ 0.822 s] [INFO] flink-e2e-test-utils ............................... SUCCESS [ 6.170 s] [INFO] flink-mesos ........................................ SUCCESS [ 17.591 s] [INFO] flink-yarn ......................................... SUCCESS [ 0.915 s] [INFO] flink-gelly ........................................ SUCCESS [ 2.601 s] [INFO] flink-gelly-scala .................................. SUCCESS [ 13.500 s] [INFO] flink-gelly-examples ............................... SUCCESS [ 7.711 s] [INFO] flink-metrics-dropwizard ........................... SUCCESS [ 0.193 s] [INFO] flink-metrics-graphite ............................. SUCCESS [ 0.123 s] [INFO] flink-metrics-influxdb ............................. SUCCESS [ 0.594 s] [INFO] flink-metrics-prometheus ........................... SUCCESS [ 0.371 s] [INFO] flink-metrics-statsd ............................... SUCCESS [ 0.179 s] [INFO] flink-metrics-datadog .............................. SUCCESS [ 0.237 s] [INFO] flink-metrics-slf4j ................................ SUCCESS [ 0.157 s] [INFO] flink-cep-scala .................................... SUCCESS [ 9.399 s] [INFO] flink-table-uber ................................... SUCCESS [ 1.794 s] [INFO] flink-table-uber-blink ............................. SUCCESS [ 2.050 s] [INFO] flink-sql-client ................................... SUCCESS [ 1.090 s] [INFO] flink-state-processor-api .......................... SUCCESS [ 0.598 s] [INFO] flink-python ....................................... SUCCESS [ 0.621 s] [INFO] flink-scala-shell .................................. SUCCESS [ 8.821 s] [INFO] flink-dist ......................................... SUCCESS [ 10.765 s] [INFO] flink-end-to-end-tests-common ...................... SUCCESS [ 0.285 s] [INFO] flink-metrics-availability-test .................... SUCCESS [ 0.153 s] [INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [ 0.132 s] [INFO] flink-heavy-deployment-stress-test ................. SUCCESS [ 6.045 s] [INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [ 0.516 s] [INFO] flink-streaming-kafka-test-base .................... SUCCESS [ 0.192 s] [INFO] flink-streaming-kafka-test ......................... SUCCESS [ 5.923 s] [INFO] flink-streaming-kafka011-test ...................... SUCCESS [ 5.786 s] [INFO] flink-streaming-kafka010-test ...................... SUCCESS [ 5.000 s] [INFO] flink-plugins-test ................................. SUCCESS [ 0.095 s] [INFO] flink-tpch-test .................................... SUCCESS [ 1.210 s] [INFO] flink-contrib ...................................... SUCCESS [ 0.052 s] [INFO] flink-connector-wikiedits .......................... SUCCESS [ 0.258 s] [INFO] flink-yarn-tests ................................... SUCCESS [ 2.919 s] [INFO] flink-fs-tests ..................................... SUCCESS [ 0.330 s] [INFO] flink-docs ......................................... SUCCESS [ 0.531 s] [INFO] flink-ml-parent .................................... SUCCESS [ 0.052 s] [INFO] flink-ml-api ....................................... SUCCESS [ 0.284 s] [INFO] flink-ml-lib ....................................... SUCCESS [ 0.164 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15:33 min [INFO] Finished at: 2019-09-12T17:41:19+08:00 [INFO] ------------------------------------------------------------------------
正常情况耗时15分钟，异常是可能会卡在 flink-runtime和 flink-runtime-web 上面
4、遇到的问题
1)、 flink 1.9 中移除了 map-r 的包，所有不用在这上面耗时间，但是我在编译的时候，还是遇到很多包无法下载（阿里云的镜像都有），只好在本地下载，上传上去（centos 的虚拟机里编译的，虚拟机下不下的包，本地都可以正常下载，就把本地的仓库覆盖上去了）
2)、 flink-runtime/flink-runtime-web 有时候会编译很久
3)、编译flink-table-api-java 的时候编译不过
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project flink-table-api-java: Compilation failure [ERROR] /home/venn/git/release/flink-release-1.9.0/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ut
ils/factories/AggregateOperationFactory.java:[550,53] unreported exception X; must be caught or declared to be thrown [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :flink-table-api-java
是低版本jdk的bug，下面这种用法有点问题
List<String> aliases = children.subList(1, children.size()) .stream() .map(alias -> ExpressionUtils.extractValue(alias, String.class) .orElseThrow(() -> new ValidationException("Unexpected alias: " + alias))) .collect(toList());
解决方法： 升级jdk 1.8 的小版本， 从jdk1.8.0_91 升到 jdk1.8.0_111 解决了
4)、可以预装个 gcc、gcc-c++，中间会有报错，不过应该不影响
