资源
手把手教你做数据分析:http://www.360doc.com/content/15/0426/12/15688396_466098219.shtml
用Kibana和logstash快速搭建实时日志查询、收集与分析系统:http://storysky.blog.51cto.com/628458/1158707/
架构：http://toutiao.com/i6223959691949507074/
工作流流程在线设计器：http://flowdesign.leipi.org/
java 数学表达是计算工具：http://blog.csdn.net/kuyuyingzi/article/details/42100945
spark系统学习：http://www.cnblogs.com/shishanyuan/p/4723604.html
spark资源：http://www.aboutyun.com/thread-11575-1-1.html
北邮资源：http://mirror.bit.edu.cn/web/
hadoop+hive+hbase : http://skambc.iteye.com/blog/2069943
hive高可用：http://www.aboutyun.com/thread-10938-1-1.html
presto:https://prestodb.io/
Transactional SQL on HBase:http://blog.csdn.net/china_world/article/details/40426183
中小企业的大数据技术路线选择: http://blog.csdn.net/china_world/article/details/40779509
好博客：http://blog.csdn.net/zhangzhaokun/article/details/51030612
hive博客学习：http://blog.csdn.net/lnho2015/article/details/51355511
报表：http://www.gcpowertools.com.cn/products/activereports/excel.htm#MultiArea
http://blog.csdn.net/ttyttytty12/article/details/11726569
awk+sed+sort常用命令收集
http://blog.csdn.net/ttyttytty12/article/details/11635169
http://blog.csdn.net/ttyttytty12/article/details/9959867
工作流设计器：http://www.cnblogs.com/2018/archive/2011/11/22/2240259.html
session共享：http://git.oschina.net/dengdai68/hsession
lingshugate: round-trip min/avg/max/stddev = 4.725/5.011/5.498/0.261 ms localhost:lingshugate YANGQING1$ localhost:lingshugate YANGQING1$ localhost:lingshugate YANGQING1$ localhost:lingshugate YANGQING1$ pwd /Users/YANGQING1/letv/project/project2/lingshugate/lingshugate localhost:lingshugate YANGQING1$ ls com.lingshu.gate.iml pom.xml src target localhost:lingshugate YANGQING1$ git pull origin remote: Counting objects: 144, done. remote: Compressing objects: 100% (93/93), done. remote: Total 144 (delta 77), reused 84 (delta 49) Receiving objects: 100% (144/144), 19.78 MiB | 428.00 KiB/s, done. Resolving deltas: 100% (77/77), completed with 20 local objects. From git.letv.cn:yubangxu/lingshugate 4191f11..2dfb217 dev -> origin/dev Updating 4191f11..2dfb217 Fast-forward src/main/java/Handle/ScheduleRouteHandle.java | 8 +++++++ src/main/java/Handle/SqlSchedule.java | 2 +- src/main/java/common/SqlConstant.java | 30 ++++-------------------- src/main/java/controller/AlluxioController.java | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/main/java/controller/FileController.java | 7 ++++++ src/main/java/controller/FileV2Controller.java | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/main/java/controller/SqlController.java | 22 ++++++++++------- src/main/java/db/AbstractConnection.java | 2 +- src/main/java/db/ConnectionPool.java | 7 +++--- src/main/java/gate/App.java | 4 +++- src/main/java/hadoop/HadoopWriterUtil.java | 147 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/main/java/model/ConnPoolParamEnum.java | 9 ++++--- src/main/java/model/ParaModel.java | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/main/java/utils/CommonUtil.java | 11 +++++++++ src/main/java/utils/HttpUtils.java | 8 +++++++ src/main/java/utils/MysqlUtil.java | 39 +++++++++++++++++++++++++++---- target/lib/alluxio-core-client-1.2.0.jar | Bin 0 -> 210704 bytes target/lib/alluxio-core-common-1.2.0.jar | Bin 0 -> 2156011 bytes target/lib/alluxio-underfs-gcs-1.2.0.jar | Bin 0 -> 506863 bytes target/lib/alluxio-underfs-hdfs-1.2.0.jar | Bin 0 -> 12581 bytes target/lib/alluxio-underfs-local-1.2.0.jar | Bin 0 -> 8776 bytes target/lib/alluxio-underfs-s3-1.2.0.jar | Bin 0 -> 507758 bytes target/lib/alluxio-underfs-s3a-1.2.0.jar | Bin 0 -> 16447 bytes target/lib/alluxio-underfs-swift-1.2.0.jar | Bin 0 -> 15660 bytes target/lib/aws-java-sdk-core-1.11.7.jar | Bin 0 -> 609843 bytes target/lib/aws-java-sdk-kms-1.11.7.jar | Bin 0 -> 382470 bytes target/lib/aws-java-sdk-s3-1.11.7.jar | Bin 0 -> 634652 bytes target/lib/commons-lang3-3.4.jar | Bin 0 -> 434678 bytes target/lib/gson-2.2.4.jar | Bin 0 -> 190432 bytes target/lib/{hadoop-common-2.5.1.jar => hadoop-common-2.6.0.jar} | Bin 2962685 -> 3302825 bytes target/lib/{hadoop-hdfs-2.5.1.jar => hadoop-hdfs-2.6.0.jar} | Bin 7095230 -> 7822588 bytes target/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar | Bin 165684 -> 165748 bytes target/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar | Bin 1134377 -> 1135023 bytes target/lib/hadoop-yarn-server-web-proxy-2.6.0.jar | Bin 28501 -> 28513 bytes target/lib/htrace-core-3.0.4.jar | Bin 0 -> 31212 bytes target/lib/jackson-annotations-2.6.0.jar | Bin 0 -> 46968 bytes target/lib/jackson-core-2.6.6.jar | Bin 0 -> 258917 bytes target/lib/jackson-databind-2.6.6.jar | Bin 0 -> 1170626 bytes target/lib/jackson-dataformat-cbor-2.6.6.jar | Bin 0 -> 48468 bytes target/lib/jcl-over-slf4j-1.7.2.jar | Bin 0 -> 16457 bytes target/lib/joss-0.9.10.jar | Bin 0 -> 286981 bytes target/lib/json-serde-1.3.1.jar | Bin 69882 -> 0 bytes target/lib/parquet-column-1.8.2-SNAPSHOT.jar | Bin 0 -> 966339 bytes target/lib/parquet-common-1.8.2-SNAPSHOT.jar | Bin 0 -> 41389 bytes target/lib/parquet-encoding-1.8.2-SNAPSHOT.jar | Bin 0 -> 868950 bytes target/lib/parquet-hadoop-1.8.2-SNAPSHOT.jar | Bin 0 -> 264678 bytes target/lib/servlet-api-2.5.jar | Bin 105112 -> 105334 bytes target/lib/xercesImpl-2.9.1.jar | Bin 0 -> 1229125 bytes target/lib/xml-apis-1.3.04.jar | Bin 0 -> 194354 bytes target/lingshuGate-1.0.0.1.jar | Bin 115301 -> 136289 bytes 50 files changed, 563 insertions(+), 47 deletions(-) create mode 100644 src/main/java/controller/AlluxioController.java create mode 100644 src/main/java/controller/FileV2Controller.java create mode 100644 src/main/java/hadoop/HadoopWriterUtil.java create mode 100644 src/main/java/model/ParaModel.java create mode 100644 target/lib/alluxio-core-client-1.2.0.jar create mode 100644 target/lib/alluxio-core-common-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-gcs-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-hdfs-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-local-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-s3-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-s3a-1.2.0.jar create mode 100644 target/lib/alluxio-underfs-swift-1.2.0.jar create mode 100644 target/lib/aws-java-sdk-core-1.11.7.jar create mode 100644 target/lib/aws-java-sdk-kms-1.11.7.jar create mode 100644 target/lib/aws-java-sdk-s3-1.11.7.jar create mode 100644 target/lib/commons-lang3-3.4.jar create mode 100644 target/lib/gson-2.2.4.jar rename target/lib/{hadoop-common-2.5.1.jar => hadoop-common-2.6.0.jar} (63%) rename target/lib/{hadoop-hdfs-2.5.1.jar => hadoop-hdfs-2.6.0.jar} (56%) create mode 100644 target/lib/htrace-core-3.0.4.jar create mode 100644 target/lib/jackson-annotations-2.6.0.jar create mode 100644 target/lib/jackson-core-2.6.6.jar create mode 100644 target/lib/jackson-databind-2.6.6.jar create mode 100644 target/lib/jackson-dataformat-cbor-2.6.6.jar create mode 100644 target/lib/jcl-over-slf4j-1.7.2.jar create mode 100644 target/lib/joss-0.9.10.jar delete mode 100644 target/lib/json-serde-1.3.1.jar create mode 100644 target/lib/parquet-column-1.8.2-SNAPSHOT.jar create mode 100644 target/lib/parquet-common-1.8.2-SNAPSHOT.jar create mode 100644 target/lib/parquet-encoding-1.8.2-SNAPSHOT.jar create mode 100644 target/lib/parquet-hadoop-1.8.2-SNAPSHOT.jar create mode 100644 target/lib/xercesImpl-2.9.1.jar create mode 100644 target/lib/xml-apis-1.3.04.jar localhost:lingshugate YANGQING1$ localhost:lingshugate YANGQING1$
./spark-sql --master yarn-client --total-executor-cores 20 --executor-memory 6g --executor-cores 12
命令： create table yqtest (key int, value string) row format delimited fields terminated by '\t'
