spark streaming 笔记
spark streaming项目 学习笔记
为什么要flume+kafka?
生成数据有高峰与低峰，如果直接高峰数据过来flume+spark/storm，实时处理容易处理不过来，扛不住压力。而选用flume+kafka添加了消息缓冲队列，spark可以去kafka里面取得数据，那么就可以起到缓冲的作用。
Flume架构：
参考学习：http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
启动一个agent:
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
添加example.conf:
|
# example.conf: A single-node Flume configuration
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1
# Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444
# Describe the sink a1.sinks.k1.type = logger
# Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 |
开一个终端测试：
|
$ telnet localhost 44444 T Trying 127.0.0.1... C Connected to localhost.localdomain (127.0.0.1). E Escape character is '^]'. H Hello world! <ENTER> O OK |
Flume将会输出：
|
12/06/19 15:32:19 INFO source.NetcatSource: Source starting 12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. } |
<二> kafka架构
producer:生产者
consumer:消费者
broker:缓冲代理
topic:主题
安装：
下载->解压->修改配置
添加环境变量：
|
$ vim ~/.bash_profile ……
export ZK_HOME=/home/centos/develop/zookeeper export PATH=$ZK_HOME/bin/:$PATH
export KAFKA_HOME=/home/centos/develop/kafka export PATH=$KAFKA_HOME/bin:$PATH |
启动zk:
zkServer.sh start
查看zk状态：
zkServer.sh status
|
$ vim config/server.properties: #需要修改配置内容 broker.id=1 listeners=PLAINTEXT://:9092 log.dirs=/home/centos/app/kafka-logs |
后台启动kafka:
nohup kafka-server-start.sh $KAFKA_HOME/config/server.properties &
创建topic:
kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 1 --partitions 1 --topic halo
-- 注：这里2181是zk端口
查看topic列表：
kafka-topics.sh --list --zookeeper node1:2181
-- 注：这里2181是zk端口
生产一个主题halo：
kafka-console-producer.sh --broker-list node1:9092 --topic halo
-- 注：这里9092是kafka端口
消费主题halo数据：
kafka-console-consumer.sh --zookeeper node1:2181 --topic halo --from-beginning
Setting up a multi-broker cluster
复制server.properties :
|
> cp config/server.properties config/server-1.properties > cp config/server.properties config/server-2.properties |
编辑内容：
|
config/server-1.properties: broker.id=1 listeners=PLAINTEXT://:9093 log.dirs=/home/centos/app/kafka-logs-1
config/server-2.properties: broker.id=2 listeners=PLAINTEXT://:9094 log.dirs=/home/centos/app//kafka-logs-2 |
现在后台启动broker:
|
>nohup kafka-server-start.sh $KAFKA_HOME/config/server-1.properties & ... >nohup kafka-server-start.sh $KAFKA_HOME/config/server-2.properties & ... |
现在我们创建一个具有三个副本的主题：
|
> bin/kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic replicated-halo |
好了，我们查看下topic主题下详细信息
|
> bin/kafka-topics.sh --describe --zookeeper node1:2181 --topic replicated-halo Topic:replicated-halo PartitionCount:1 ReplicationFactor:3 Configs: Topic: replicated-halo Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0 |
- "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
- "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
- "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
【附：jps -m显示具体的进程信息】
一个kafka生产栗子：
package com.lin.spark.kafka; import kafka.javaapi.producer.Producer; import kafka.producer.KeyedMessage; import kafka.producer.ProducerConfig; import java.util.Properties; /** * Created by Administrator on 2019/6/1. */ public class KafkaProducer extends Thread { private String topic; private Producer<Integer, String> producer; public KafkaProducer(String topic) { this.topic = topic; Properties properties = new Properties(); properties.put("metadata.broker.list", KafkaProperities.BROKER_LIST); properties.put("serializer.class", "kafka.serializer.StringEncoder"); properties.put("request.required.acks", "1"); producer = new Producer<Integer, String>(new ProducerConfig(properties)); } @Override public void run() { int messageNo = 1; while (true) { String message = "message_" + messageNo; producer.send(new KeyedMessage<Integer, String>(topic,message)); System.out.println("Send:"+message); messageNo++; try{ Thread.sleep(2000);//2秒钟打印一次 }catch (Exception e){ e.printStackTrace(); } } } //测试 public static void main(String[] args){ KafkaProducer producer = new KafkaProducer("halo"); producer.run(); } }
测试消费的数据：
> kafka-console-consumer.sh --zookeeper node1:2181 --topic halo --from-beginning
对应的消费者代码：
package com.lin.spark.kafka; import kafka.consumer.Consumer; import kafka.consumer.ConsumerConfig; import kafka.consumer.ConsumerIterator; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; /** * Created by Administrator on 2019/6/2. */ public class KafkaConsumer extends Thread { private String topic; public KafkaConsumer(String topic) { this.topic = topic; } private ConsumerConnector createConnector(){ Properties properties = new Properties(); properties.put("zookeeper.connect", KafkaProperities.ZK); properties.put("group.id",KafkaProperities.GROUP_ID); return Consumer.createJavaConsumerConnector(new ConsumerConfig(properties)); } @Override public void run() { ConsumerConnector consumer = createConnector(); Map<String,Integer> topicCountMap = new HashMap<String, Integer>(); topicCountMap.put(topic,1); Map<String, List<KafkaStream<byte[], byte[]>>> streams = consumer.createMessageStreams(topicCountMap); KafkaStream<byte[], byte[]> kafkaStream = streams.get(topic).get(0); ConsumerIterator<byte[], byte[]> iterator = kafkaStream.iterator(); while (iterator.hasNext()){ String result = new String(iterator.next().message()); System.out.println("result:"+result); } } public static void main(String[] args){ KafkaConsumer kafkaConsumer = new KafkaConsumer("halo"); kafkaConsumer.run(); } }
一个简单kafka与spark streaming整合例子：
启动kafka，并生产数据
> kafka-console-producer.sh --broker-list 172.16.182.97:9092 --topic halo
参数固定：
package com.lin.spark import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka.KafkaUtils import org.apache.spark.streaming.{Seconds, StreamingContext} object KafkaStreaming { def main(args: Array[String]): Unit = { val conf = new SparkConf().setAppName("SparkStreamingKakfaWordCount").setMaster("local[4]") val ssc = new StreamingContext(conf,Seconds(5)) val topicMap = "halo".split(":").map((_, 1)).toMap val zkQuorum = "hadoop:2181"; val group = "consumer-group" val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2) lines.print() ssc.start() ssc.awaitTermination() } }
参数输入：
package com.lin.spark import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka.KafkaUtils import org.apache.spark.streaming.{Seconds, StreamingContext} object KafkaStreaming { def main(args: Array[String]): Unit = { if (args.length != 4) { System.err.println("参数不对") } //args: hadoop:2181 consumer-group halo,hello_topic 2 val Array(zkQuorum, group, topics, numThreads) = args val conf = new SparkConf().setAppName("SparkStreamingKakfaWordCount").setMaster("local[4]") val ssc = new StreamingContext(conf, Seconds(5)) val topicMap = topics.split(",").map((_,numThreads.toInt)).toMap val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2) lines.print() ssc.start() ssc.awaitTermination() } }
