java操作hdfs到数据库或者缓存
使用hadoop工具将数据分析出来以后,须要做入库处理或者存到缓存中。不然就没了意义
一下是使用javaAPI操作hdfs存入缓存的代码:
<span style="font-family:Microsoft YaHei;font-size:14px;">public class InterestToRedisJob { FileSystem hdfs = null; public InterestToRedisJob(){ init(); } private void init(){ Configuration conf = new Configuration(); conf.set("fs.default.name", "hdfs地址"); try { hdfs = FileSystem.get(conf); } catch (IOException e) { e.printStackTrace(); } } public void ReadFileToReids(String path) throws IOException{ Path paths = new Path(path); FileStatus[] files = hdfs.listStatus(paths);</span>
<span style="font-family:Microsoft YaHei;font-size:14px;"><span style="white-space:pre"> </span>//这儿是自己实现的一个路径顾虑器,也可不适用,在【1】处直接推断part-r-等标示</span>
<span style="font-family:Microsoft YaHei;font-size:14px;"> PathFilter filter = new ResultNameFilter("part-r-"); Text line = new Text(); RedisClient redis = new RedisClient(); for(FileStatus file:files){ if(file.isDir() || !filter.accept(file.getPath())){//【1】 continue; }else{ FSDataInputStream input = null; try{ input = hdfs.open(file.getPath()); LineReader reader = new LineReader(input); while(reader.readLine(line) > 0){ System.out.println(line); String[] arr = line.toString().split("\t");</span>
<span style="font-family:Microsoft YaHei;font-size:14px;"><span style="white-space:pre"> </span>//做存入redis处理 redis.saveHsetValue(arr[0], "interest", arr[1]); } }catch(Exception e){ e.printStackTrace(); }finally{ if(input != null){ input.close(); } } } } } public static void main(String[] args) { InterestToRedisJob job = new InterestToRedisJob(); try { job.ReadFileToReids("你的path"); } catch (IOException e) { e.printStackTrace(); } } }</span>上面代码须要改动后使用,请勿直接粘贴。
posted on 2017-04-12 14:31 gavanwanggw 阅读(718) 评论(0) 编辑 收藏 举报