2021.06.03刘一辰的软件工程随笔

今日目标：

JAVAcheng

 

今日总结：

class WindowWordCount {

    public static void main(String[] args) throws Exception{
        
        SparkConf conf = new SparkConf().setAppName("WindowWordCount");
        
        // create context
        JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(10));
    
        // kafka properties map
        Map<String, String> KafkaParams = new HashMap<String, String>();
        KafkaParams.put("bootstrap.servers", "hserver-1:9092,hserver-2:9092,hserver-3:9092");
        KafkaParams.put("group.id", "WindowWordCount");
        KafkaParams.put("auto.offset.reset", "smallest");
        
        // kafka topic set
        Set<String> topics = new HashSet<String>();
        topics.add(args[0]);
        
        // access data DStream
        JavaPairInputDStream<String, String> SearchLogDStream = 
                KafkaUtils.createDirectStream(
                        jssc, 
                        String.class, 
                        String.class, 
                        StringDecoder.class, 
                        StringDecoder.class, 
                        KafkaParams, 
                        topics
                        );
        
        // flatMap
        JavaDStream<String> WordDSteram = 
                SearchLogDStream.flatMap(
                        
                        new FlatMapFunction<Tuple2<String,String>, String>() {

                            private static final long serialVersionUID = 4034522628037914742L;

                            @Override
                            public Iterator<String> call(Tuple2<String, String> tuple) throws Exception {

                                return Arrays.asList(tuple._2.split(" ")).iterator();
                            }
                        });

明日目标：

mapToPair