求uv
uv定义
在同一天内,UV指记录第一次进入网站的具有独立IP的访问者,在同一天内再次访问该网站则不计数.
代码实现:方法一
SparkConf conf = new SparkConf();
conf.setMaster("local").setAppName("uv");
JavaSparkContext jsc = new JavaSparkContext(conf);
JavaRDD<String> lineRDD = jsc.textFile("data/pvuvdata.txt");
lineRDD.flatMap(new FlatMapFunction<String, Tuple2<String, String>>() {
@Override
public Iterator<Tuple2<String, String>> call(String s) throws Exception {
String[] str = s.split(" ");
Tuple2<String, String> tuple2= new Tuple2<String, String>(str[5], str[0]);
List<Tuple2<String, String>> list = new ArrayList<>();
list.add(tuple2);
return list.iterator();
}
}).distinct()
.mapToPair(new PairFunction<Tuple2<String, String>, String, Integer>() {
@Override
public Tuple2<String, Integer> call(Tuple2<String, String> tuple2) throws Exception {
return new Tuple2<String, Integer>(tuple2._1, 1);
}
})
.reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer integer, Integer integer2) throws Exception {
return integer + integer2;
}
}).foreach(new VoidFunction<Tuple2<String, Integer>>() {
@Override
public void call(Tuple2<String, Integer> stringIntegerTuple2) throws Exception {
System.out.println(stringIntegerTuple2);
}
});
jsc.stop();
代码实现:方法二
使用countByKey
本文来自博客园,作者:jsqup,转载请注明原文链接:https://www.cnblogs.com/jsqup/p/16388774.html