build jar(sbt)

project 中遇到

  1. log file & project file
  2. build.sbt
  3. startup.sh
  4. then command line:  > sh startup.sh

example:

.sbt

name := "Simple Project" //project name
version := "1.0"  //project version
scalaVersion := "2.11.8" //scala version
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.1.0"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.2.11"
//all
Dependencies should be aware of the version compatibility

.sh

 /usr/local/spark/bin/spark-submit --driver-class-path /usr/local/spark/jars/*:/usr/local/spark/jars/kafka/* --class "org.apache.spark.examples.streaming.KafkaWordCount" /usr/local/spark/mycode/kafka/target/scala-2.11/simple-project_2.11-1.0.jar 127.0.0.1:2181 1 sex 1
spark-submit目录 
参数:
--driver-class-path  spark中的所有jars以及kafka中所有jars

--class  工程文件main所在class + 生成的jar的位置
127.0.0.1:2181 : zookeeper-server kafka-server ip+port

1 sex 1: consumer group;topics; number of consumers

REFERENCE:
http://dblab.xmu.edu.cn

 

 

 

posted on 2017-09-26 23:14  satyrs  阅读(128)  评论(0编辑  收藏  举报

导航