01 2018 档案

scala mysql jdbc oper
摘要：package egsql import java.util.Properties import com.sun.org.apache.xalan.internal.xsltc.compiler.util.IntType import org.apache.spark.sql.types._ import org.apache.spark.sql.{Row, SparkSession} im... 阅读全文
posted @ 2018-01-29 20:29 Alamps 阅读(423) 评论(0) 推荐(0) 编辑
RDD、DataFrame、Dataset三者三者之间转换
摘要：package dataframe import org.apache.spark.sql.{DataFrame, Dataset, SparkSession} //// Explore interoperability between DataFrame and Dataset. Note tha 阅读全文
posted @ 2018-01-23 09:19 Alamps 阅读(4129) 评论(0) 推荐(0) 编辑
storm 001
摘要：Hadoop、Storm系统和组件接口对比表： package storm; import org.apache.storm.Config;import org.apache.storm.StormSubmitter;import org.apache.storm.generated.StormTo 阅读全文
posted @ 2018-01-18 18:52 Alamps 阅读(532) 评论(0) 推荐(0) 编辑
Spark Streaming 002 统计单词的例子
摘要：[root@alamps TestDir]# hadoop fs -chown hadoop:hadoop hdfs://alamps:9000/library/SparkStreaming/data/[root@alamps TestDir]# hadoop fs -chown hadoop:ha 阅读全文
posted @ 2018-01-13 15:57 Alamps 阅读(554) 评论(0) 推荐(0) 编辑
Exception in Spark
摘要：1: Exception in thread "main" org.apache.spark.SparkException: org.apache.spark.streaming.dstream.MappedDStream@5a69b104 has not been initialized 原因： StreamingContext.checkpoint(...)指定的checkpoint目录下... 阅读全文
posted @ 2018-01-13 15:53 Alamps 阅读(214) 评论(0) 推荐(0) 编辑
spark stream001
摘要：package stream.scala import java.io.PrintWriter import java.net.ServerSocket class LoggerSimulation { } object LoggerSimulation{ /** * 生成一个字母 * @param 字母的下标 * @return 生成的字母 ... 阅读全文
posted @ 2018-01-12 11:01 Alamps 阅读(316) 评论(0) 推荐(0) 编辑
spark2.2.1 sql001
摘要：package sql import org.apache.spark.sql.SparkSession import org.apache.spark.SparkContext object Parquet extends App { val spark = SparkSession .builder() .appName("Spark SQL basic exampl... 阅读全文
posted @ 2018-01-11 13:14 Alamps 阅读(600) 评论(0) 推荐(0) 编辑
CountdownLatch
摘要：import java.util.concurrent.CountDownLatch; import java.util.concurrent.CyclicBarrier; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; public class CountdownLatch... 阅读全文
posted @ 2018-01-04 22:50 Alamps 阅读(189) 评论(3) 推荐(0) 编辑