kafka+pyspark

安装kafka

kafka 三部分 server producer consumer

pyspark 监控

一、环境部署

1.导入对应版本的spark-streaming-kafka-*-*.jar

2.相应jar追加到SPARK_DIST_CLASSPATH

二、kafka+spark测试

1.启动kafka的server和producer

2.代码

from pyspark.streaming.kafka import KafkaUtils

if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: kafka_wordcount.py <zk> <topic>", file=sys.stderr)
exit(-1)

sc = SparkContext(appName="PythonStreamingKafkaWordCount")
ssc = StreamingContext(sc, 1)

zkQuorum, topic = sys.argv[1:]
kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
lines = kvs.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
counts.pprint()

ssc.start()
ssc.awaitTermination()

3.启动 开始监控生产者 即时计算词频数

4.注意各个版本匹配问题