kafka：
分布式消息系统
p2p + ps = 消费者组
JMS：
java message service
p2p：
peer to peer
point to point
ps:
publish && subscribe
kafka： scala + java
===============
用于实时流处理
特点：1、持久化数据
2、高吞吐量
3、分布式
4、多客户端支持
5、实时性
kafka：broker：代理
生产者
消费者
kafka_2.11-1.1.0.tgz：
// 2.11 ===> scala版本
// 1.1.0 ==> kafka版本
kafka安装：
========================
1、解压
2、符号链接
3、环境变量
# kafka环境变量
export KAFKA_HOME=/soft/kafka
export PATH=$PATH:$KAFKA_HOME/bin
4、生效环境变量
5、
kafka本地模式：
=======================
1、修改配置文件:/soft/kafka/config/server.properties
修改zookeeper.connect=s102:2181,s103:2181,s104:2181
修改log.dirs=/home/centos/kafka/logs
修改listeners=PLAINTEXT://s101:9092
2、启动kafka
kafka-server-start.sh [-daemon] /soft/kafka/config/server.properties
3、jps查看kafka进程
4、关闭kafka
kafka-server-stop.sh
kafka完全分布式：s102-s104
1、同步kafka安装目录和符号链接
xsync.sh /soft/kafka
xsync.sh /soft/kafka_2.11-1.1.0
2、root同步环境变量
su root
xsync.sh /etc/profile
exit
3、分别使s102-s104环境变量生效
s102> source /etc/profile
s103> source /etc/profile
s104> source /etc/profile
4、分别修改s102-s104的kafka配置文件/soft/kafka/config/server.properties
s102:
修改broker.id=102
修改listeners=PLAINTEXT://s102:9092
s103:
修改broker.id=103
修改listeners=PLAINTEXT://s103:9092
s104:
修改broker.id=104
修改listeners=PLAINTEXT://s104:9092
5、分别启动s102-s104的kafka
s102> kafka-server-start.sh -daemon /soft/kafka/config/server.properties
s103> kafka-server-start.sh -daemon /soft/kafka/config/server.properties
s104> kafka-server-start.sh -daemon /soft/kafka/config/server.properties
6、编写批量启动kafka
#!/bin/bash
if [ $# -ge 1 ] ; then echo param must be 0 ; exit ; fi
for (( i=102 ; i<=104 ; i++ )) ; do
tput setaf 2
echo ================ s$i starting kafka ================
tput setaf 9
ssh s$i "source /etc/profile ; kafka-server-start.sh -daemon /soft/kafka/config/server.properties"
done
7、编写批量关闭kafka
#!/bin/bash
if [ $# -ge 1 ] ; then echo param must be 0 ; exit ; fi
for (( i=102 ; i<=104 ; i++ )) ; do
tput setaf 2
echo ================ s$i stoping kafka ================
tput setaf 9
ssh s$i "source /etc/profile ; kafka-server-stop.sh"
done
8、结合：xkafka.sh
#!/bin/bash
cmd=$1
if [ $# -gt 1 ] ; then echo param must be 1 ; exit ; fi
for (( i=102 ; i<=104 ; i++ )) ; do
tput setaf 2
echo ================ s$i $cmd kafka ================
tput setaf 9
case $cmd in
start ) ssh s$i "source /etc/profile ; kafka-server-start.sh -daemon /soft/kafka/config/server.properties" ;;
stop ) ssh s$i "source /etc/profile ; kafka-server-stop.sh" ;;
* ) echo illegal argument ; exit ;;
esac
done
kafka基本命令：s102-s104操作
==============================
开启kafka服务：kafka-server-start.sh -daemon /soft/kafka/config/server.properties
关闭kafka ：kafka-server-stop.sh
topic: kafka生产消费的基本单位
record：kafka生产消费的基本数据单位
以K-V形式存在
topic： kafka向指定topic生产数据，向指定topic消费数据
创建topic： kafka-topics.sh --create --topic t1 --zookeeper s102:2181 --partitions 2 --replication-factor 2
列出topic： kafka-topics.sh --list --zookeeper s102:2181
创建生产者：kafka-console-producer.sh --broker-list s102:9092 --topic t1
创建消费者：kafka-console-consumer.sh --zookeeper s102:2181 --topic t1 //接着上一次位置消费
kafka-console-consumer.sh --zookeeper s102:2181 --topic t1 --from-beginning //从头开始消费
kafka使用API生产者和消费者：
==============================
1、pom文件
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>1.1.0</version>
</dependency>
2、编写生产者
public class MyProducer {
public static void main(String[] args) throws Exception {
//初始化java配置文件
Properties props = new Properties();
props.put("metadata.broker.list", "s102:9092, s103:9093, s104:9094 ");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
//将java配置文件封装成kafka配置文件
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
for (int i = 0; i < 100; i++) {
String msg = "tom" + i;
System.out.println(msg);
// 创建kafka消息实例
KeyedMessage<String, String> data = new KeyedMessage<String, String>("t1", msg);
// 发送数据
producer.send(data);
System.out.println("Message Count - " + i);
Thread.sleep(1000);
}
// 关闭生产者
producer.close();
}
}
3、编写消费者
public class MyConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("zookeeper.connect", "s102:2181,s103:2181,s104:2181");
props.put("group.id", "g1");
props.put("zookeeper.session.timeout.ms", "500");
props.put("zookeeper.sync.time.ms", "250");
props.put("auto.commit.interval.ms", "1000");
//初始化consumer配置
ConsumerConfig config = new ConsumerConfig(props);
//初始化consumer
ConsumerConnector consumer = kafka.consumer.Consumer
.createJavaConsumerConnector(config);
Map<String, Integer> topicMap = new HashMap<String, Integer>();
//指定消费线程数
topicMap.put("t1", 1);
//创建消息对象
//Map<topic, List<k, message>>
Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreamsMap = consumer.createMessageStreams(topicMap);
//获取t1主题的数据(k-v)
List<KafkaStream<byte[], byte[]>> streamList = consumerStreamsMap.get("t1");
for (final KafkaStream<byte[], byte[]> stream : streamList) {
ConsumerIterator<byte[], byte[]> consumerIte = stream.iterator();
while (consumerIte.hasNext()) {
//迭代取出消息数据
System.out.println("value: " + new String(consumerIte.next().message()));
}
}
if (consumer != null) {
consumer.shutdown();
}
}
}
