Hadoop集群-HDFS集群中大数据运维常用的命令总结
Hadoop集群-HDFS集群中大数据运维常用的命令总结
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
本篇博客会简单涉及到滚动编辑，融合镜像文件，目录的空间配额等运维操作简介。话不多少，直接上命令便于以后查看。
一.查看hadf的帮助信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs Usage: hdfs [--config confdir] COMMAND where COMMAND is one of: dfs run a filesystem command on the file systems supported in Hadoop. namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode journalnode run the DFS journalnode zkfc run the ZK Failover Controller daemon datanode run a DFS datanode dfsadmin run a DFS admin client diskbalancer Distributes data evenly among disks on a given node haadmin run a DFS HA admin client fsck run a DFS filesystem checking utility balancer run a cluster balancing utility jmxget get JMX exported values from NameNode or DataNode. mover run a utility to move block replicas across storage types oiv apply the offline fsimage viewer to an fsimage oiv_legacy apply the offline fsimage viewer to an legacy fsimage oev apply the offline edits viewer to an edits file fetchdt fetch a delegation token from the NameNode getconf get config values from configuration groups get the groups which users belong to snapshotDiff diff two snapshots of a directory or diff the current directory contents with a snapshot lsSnapshottableDir list all snapshottable dirs owned by the current user Use -help to see options portmap run a portmap service nfs3 run an NFS version 3 gateway cacheadmin configure the HDFS cache crypto configure HDFS encryption zones storagepolicies list/get/set block storage policies version print the version Most commands print help when invoked w/o parameters. [hdfs@node101.yinzhengjie.org.cn ~]$
综上所述，hdfs有多个子选项，作为一枚新手建议从dfs入手，dfs子选项意思是在hdfs文件系统上运行当前系统的命令，而这些命令跟咱们学习的Linux命令长得几乎一样，接下来我们一起来看看如果使用它们吧。
二.hdfs与dfs结合使用的案例
其实hdfs 和dfs 结合使用的话实际上调用的是hadoop fs这个命令。不信你自己看帮助信息如下：
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] [-v] [-x] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] [-x] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] [hdfs@node101.yinzhengjie.org.cn ~]$
1>.查看hdfs子命令的帮助信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -help ls -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...] : List the contents that match the specified file pattern. If path is not specified, the contents of /user/<currentUser> will be listed. For a directory a list of its direct children is returned (unless -d option is specified). Directory entries are of the form: permissions - userId groupId sizeOfDirectory(in bytes) modificationDate(yyyy-MM-dd HH:mm) directoryName and file entries are of the form: permissions numberOfReplicas userId groupId sizeOfFile(in bytes) modificationDate(yyyy-MM-dd HH:mm) fileName -C Display the paths of files and directories only. -d Directories are listed as plain files. -h Formats the sizes of files in a human-readable fashion rather than a number of bytes. -q Print ? instead of non-printable characters. -R Recursively list the contents of directories. -t Sort files by modification time (most recent first). -S Sort files by size. -r Reverse the order of the sort. -u Use time of last access instead of modification for display and sorting. [hdfs@node101.yinzhengjie.org.cn ~]$
2>.查看hdfs文件系统中已经存在的文件
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -ls / Found 5 items drwxr-xr-x - hbase hbase 0 2019-05-25 00:03 /hbase drwxr-xr-x - hdfs supergroup 0 2019-05-22 19:17 /jobtracker drwxr-xr-x - hdfs supergroup 0 2019-05-28 12:11 /system drwxrwxrwt - hdfs supergroup 0 2019-05-23 13:37 /tmp drwxrwxrwx - hdfs supergroup 0 2019-05-23 13:47 /user [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$
3>.在hdfs文件系统中创建文件
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -ls /user/yinzhengjie/data/ Found 1 items drwxr-xr-x - root supergroup 0 2019-05-22 19:46 /user/yinzhengjie/data/day001 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -touchz /user/yinzhengjie/data/1.txt [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -ls /user/yinzhengjie/data/ Found 2 items -rw-r--r-- 3 hdfs supergroup 0 2019-05-28 15:16 /user/yinzhengjie/data/1.txt drwxr-xr-x - root supergroup 0 2019-05-22 19:46 /user/yinzhengjie/data/day001 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$
4>.上传文件至根目录(在上传的过程中会产生一个以"*.Copying"字样的临时文件)
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 0 2018-05-25 21:56 /1.txt -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -put hadoop-2.7.3.tar.gz / [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 0 2018-05-25 21:56 /1.txt -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
5>.在hdfs文件系统中下载文件
[yinzhengjie@s101 ~]$ ll total 0 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 0 2018-05-25 21:56 /1.txt -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs dfs -get /1.txt [yinzhengjie@s101 ~]$ ll total 0 -rw-r--r--. 1 yinzhengjie yinzhengjie 0 May 25 22:06 1.txt drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell [yinzhengjie@s101 ~]$
6>.在hdfs文件系统中删除文件
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 0 2018-05-25 21:56 /1.txt -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -rm /1.txt 18/05/25 22:08:07 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /1.txt [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
7>.在hdfs文件系统中查看文件内容
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -cat /xrsync.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -lt 1 ];then echo "请输入参数"; exit fi #获取文件路径 file=$@ #获取子路径 filename=`basename $file` #获取父路径 dirpath=`dirname $file` #获取完整路径 cd $dirpath fullpath=`pwd -P` #同步文件到DataNode for (( i=102;i<=104;i++ )) do #使终端变绿色 tput setaf 2 echo =========== s$i %file =========== #使终端变回原来的颜色，即白灰色 tput setaf 7 #远程执行命令 rsync -lr $filename `whoami`@s$i:$fullpath #判断命令是否执行成功 if [ $? == 0 ];then echo "命令执行成功" fi done [yinzhengjie@s101 ~]$
8>.在hdfs文件系统中创建目录
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs dfs -mkdir /shell [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:12 /shell -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
9>.在hdfs文件系统中修改文件名称（当然你可以可以用来移动文件到目录哟）
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:12 /shell -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /xcall.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs dfs -mv /xcall.sh /call.sh [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /call.sh -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:12 /shell -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 4 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /call.sh -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:12 /shell -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -mv /call.sh /shell [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:19 /shell -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -ls /shell Found 1 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /shell/call.sh [yinzhengjie@s101 ~]$
10>.在hdfs问系统中拷贝文件到目录
[yinzhengjie@s101 ~]$ hdfs dfs -ls /shell Found 1 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /shell/call.sh [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:19 /shell -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -cp /xrsync.sh /shell [yinzhengjie@s101 ~]$ hdfs dfs -ls /shell Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 517 2018-05-25 21:17 /shell/call.sh -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 22:21 /shell/xrsync.sh [yinzhengjie@s101 ~]$
11>.递归删除目录
[yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 3 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz drwxr-xr-x - yinzhengjie supergroup 0 2018-05-25 22:21 /shell -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -rmr /shell rmr: DEPRECATED: Please use 'rm -r' instead. 18/05/25 22:22:31 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /shell [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 /xrsync.sh [yinzhengjie@s101 ~]$
12>.列出本地文件的内容（默认是hdfs文件系统哟）
[yinzhengjie@s101 ~]$ hdfs dfs -ls file:///home/yinzhengjie/ Found 9 items -rw------- 1 yinzhengjie yinzhengjie 940 2018-05-25 19:17 file:///home/yinzhengjie/.bash_history -rw-r--r-- 1 yinzhengjie yinzhengjie 18 2015-11-19 21:02 file:///home/yinzhengjie/.bash_logout -rw-r--r-- 1 yinzhengjie yinzhengjie 193 2015-11-19 21:02 file:///home/yinzhengjie/.bash_profile -rw-r--r-- 1 yinzhengjie yinzhengjie 231 2015-11-19 21:02 file:///home/yinzhengjie/.bashrc drwxrwxr-x - yinzhengjie yinzhengjie 39 2018-05-25 09:14 file:///home/yinzhengjie/.oracle_jre_usage drwx------ - yinzhengjie yinzhengjie 76 2018-05-25 19:20 file:///home/yinzhengjie/.ssh -rw-r--r-- 1 yinzhengjie yinzhengjie 0 2018-05-25 22:06 file:///home/yinzhengjie/1.txt drwxrwxr-x - yinzhengjie yinzhengjie 35 2018-05-25 19:08 file:///home/yinzhengjie/hadoop drwxrwxr-x - yinzhengjie yinzhengjie 96 2018-05-25 22:05 file:///home/yinzhengjie/shell [yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ hdfs dfs -ls hdfs:/ Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-25 21:59 hdfs:///hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 700 2018-05-25 21:17 hdfs:///xrsync.sh [yinzhengjie@s101 ~]$
13>.追加文件内容到hdfs文件系统中的文件
[yinzhengjie@s101 ~]$ ll total 390280 drwxrwxr-x. 3 yinzhengjie yinzhengjie 16 May 27 00:01 hadoop drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Aug 17 2016 hadoop-2.7.3 -rw-rw-r--. 1 yinzhengjie yinzhengjie 214092195 Aug 26 2016 hadoop-2.7.3.tar.gz -rw-rw-r--. 1 yinzhengjie yinzhengjie 185540433 May 17 2017 jdk-8u131-linux-x64.tar.gz -rwxrwxr-x. 1 yinzhengjie yinzhengjie 615 May 26 23:24 xcall.sh -rwxrwxr-x. 1 yinzhengjie yinzhengjie 742 May 26 23:29 xrsync.sh [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-27 00:16 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 615 2018-05-27 00:15 /xcall.sh [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs dfs -appendToFile xrsync.sh /xcall.sh [yinzhengjie@s101 ~]$ hdfs dfs -ls / Found 2 items -rw-r--r-- 3 yinzhengjie supergroup 214092195 2018-05-27 00:16 /hadoop-2.7.3.tar.gz -rw-r--r-- 3 yinzhengjie supergroup 1357 2018-05-27 01:28 /xcall.sh [yinzhengjie@s101 ~]$
14>.格式化名称节点
[root@yinzhengjie ~]# hdfs namenode 18/05/27 17:23:56 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = yinzhengjie/211.98.71.195 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /soft/hadoop-2.7.3/etc/hadoop:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_131 ************************************************************/ 18/05/27 17:23:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/05/27 17:23:56 INFO namenode.NameNode: createNameNode [] 18/05/27 17:23:56 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 18/05/27 17:23:57 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 18/05/27 17:23:57 INFO impl.MetricsSystemImpl: NameNode metrics system started 18/05/27 17:23:57 INFO namenode.NameNode: fs.defaultFS is hdfs://localhost/ 18/05/27 17:23:57 INFO hdfs.DFSUtil: Starting Web-server for hdfs at: http://0.0.0.0:50070 18/05/27 17:23:57 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 18/05/27 17:23:57 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 18/05/27 17:23:57 INFO http.HttpRequestLog: Http request log for http.requests.namenode is not defined 18/05/27 17:23:57 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 18/05/27 17:23:57 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 18/05/27 17:23:57 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 18/05/27 17:23:57 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 18/05/27 17:23:57 INFO http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 18/05/27 17:23:57 INFO http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 18/05/27 17:23:57 INFO http.HttpServer2: Jetty bound to port 50070 18/05/27 17:23:57 INFO mortbay.log: jetty-6.1.26 18/05/27 17:23:58 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070 18/05/27 17:23:58 WARN namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories! 18/05/27 17:23:58 WARN namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of data loss due to lack of redundant storage directories! 18/05/27 17:23:58 INFO namenode.FSNamesystem: No KeyProvider found. 18/05/27 17:23:58 INFO namenode.FSNamesystem: fsLock is fair:true 18/05/27 17:23:58 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 18/05/27 17:23:58 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/05/27 17:23:58 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 18/05/27 17:23:58 INFO blockmanagement.BlockManager: The block deletion will start around 2018 May 27 17:23:58 18/05/27 17:23:58 INFO util.GSet: Computing capacity for map BlocksMap 18/05/27 17:23:58 INFO util.GSet: VM type = 64-bit 18/05/27 17:23:58 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 18/05/27 17:23:58 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/05/27 17:23:58 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 18/05/27 17:23:58 INFO blockmanagement.BlockManager: defaultReplication = 1 18/05/27 17:23:58 INFO blockmanagement.BlockManager: maxReplication = 512 18/05/27 17:23:58 INFO blockmanagement.BlockManager: minReplication = 1 18/05/27 17:23:58 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/05/27 17:23:58 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/05/27 17:23:58 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/05/27 17:23:58 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/05/27 17:23:58 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) 18/05/27 17:23:58 INFO namenode.FSNamesystem: supergroup = supergroup 18/05/27 17:23:58 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/05/27 17:23:58 INFO namenode.FSNamesystem: HA Enabled: false 18/05/27 17:23:58 INFO namenode.FSNamesystem: Append Enabled: true 18/05/27 17:23:58 INFO util.GSet: Computing capacity for map INodeMap 18/05/27 17:23:58 INFO util.GSet: VM type = 64-bit 18/05/27 17:23:58 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 18/05/27 17:23:58 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/05/27 17:23:58 INFO namenode.FSDirectory: ACLs enabled? false 18/05/27 17:23:58 INFO namenode.FSDirectory: XAttrs enabled? true 18/05/27 17:23:58 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 18/05/27 17:23:58 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/05/27 17:23:58 INFO util.GSet: Computing capacity for map cachedBlocks 18/05/27 17:23:58 INFO util.GSet: VM type = 64-bit 18/05/27 17:23:58 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 18/05/27 17:23:58 INFO util.GSet: capacity = 2^18 = 262144 entries 18/05/27 17:23:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 18/05/27 17:23:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 18/05/27 17:23:58 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 18/05/27 17:23:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/05/27 17:23:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/05/27 17:23:58 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/05/27 17:23:58 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/05/27 17:23:58 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/05/27 17:23:58 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/05/27 17:23:58 INFO util.GSet: VM type = 64-bit 18/05/27 17:23:58 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 18/05/27 17:23:58 INFO util.GSet: capacity = 2^15 = 32768 entries 18/05/27 17:23:58 WARN common.Storage: Storage directory /tmp/hadoop-root/dfs/name does not exist 18/05/27 17:23:58 WARN namenode.FSNamesystem: Encountered exception loading fsimage org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-root/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:327) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:215) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559) 18/05/27 17:23:58 INFO mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070 18/05/27 17:23:58 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system... 18/05/27 17:23:58 INFO impl.MetricsSystemImpl: NameNode metrics system stopped. 18/05/27 17:23:58 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 18/05/27 17:23:58 ERROR namenode.NameNode: Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-root/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:327) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:215) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559) 18/05/27 17:23:58 INFO util.ExitUtil: Exiting with status 1 18/05/27 17:23:58 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at yinzhengjie/211.98.71.195 ************************************************************/ [root@yinzhengjie ~]#
15>.创建快照（关于快照更详细的用法请参考：https://www.cnblogs.com/yinzhengjie/p/9099529.html）
[root@yinzhengjie ~]# hdfs dfs -ls -R / drwxr-xr-x - root supergroup 0 2018-05-27 20:37 /data drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html -rw-r--r-- 1 root supergroup 12 2018-05-27 20:28 /data/name.txt -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/yinzhengjie.sql [root@yinzhengjie ~]# [root@yinzhengjie ~]# echo "hello" > 1.txt [root@yinzhengjie ~]# [root@yinzhengjie ~]# echo "world" > 2.txt [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -put 1.txt /data [root@yinzhengjie ~]# hdfs dfs -put 2.txt /data/etc [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls -R / drwxr-xr-x - root supergroup 0 2018-05-27 20:58 /data -rw-r--r-- 1 root supergroup 6 2018-05-27 20:58 /data/1.txt drwxr-xr-x - root supergroup 0 2018-05-27 20:58 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 20:58 /data/etc/2.txt -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html -rw-r--r-- 1 root supergroup 12 2018-05-27 20:28 /data/name.txt -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/yinzhengjie.sql [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfsadmin -allowSnapshot /data #启用快照功能 Allowing snaphot on /data succeeded [root@yinzhengjie ~]# hdfs dfs -createSnapshot /data firstSnapshot #创建快照并起名为“firstSnapshot”。 Created snapshot /data/.snapshot/firstSnapshot [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls -R /data/.snapshot/firstSnapshot -rw-r--r-- 1 root supergroup 6 2018-05-27 20:58 /data/.snapshot/firstSnapshot/1.txt drwxr-xr-x - root supergroup 0 2018-05-27 20:58 /data/.snapshot/firstSnapshot/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 20:58 /data/.snapshot/firstSnapshot/etc/2.txt -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/.snapshot/firstSnapshot/index.html -rw-r--r-- 1 root supergroup 12 2018-05-27 20:28 /data/.snapshot/firstSnapshot/name.txt -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/.snapshot/firstSnapshot/yinzhengjie.sql [root@yinzhengjie ~]#
16>.重命名快照
[root@yinzhengjie ~]# hdfs dfs -ls /data/.snapshot/ Found 1 items drwxr-xr-x - root supergroup 0 2018-05-27 21:02 /data/.snapshot/firstSnapshot [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -renameSnapshot /data firstSnapshot newSnapshot #将/data目录的firstSnapshot快照名称改名为newSnapshot [root@yinzhengjie ~]# hdfs dfs -ls /data/.snapshot/ Found 1 items drwxr-xr-x - root supergroup 0 2018-05-27 21:02 /data/.snapshot/newSnapshot [root@yinzhengjie ~]#
17>.删除快照
[root@yinzhengjie ~]# hdfs dfs -ls /data/.snapshot/ Found 1 items drwxr-xr-x - root supergroup 0 2018-05-27 21:02 /data/.snapshot/newSnapshot [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -deleteSnapshot /data newSnapshot [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls /data/.snapshot/ [root@yinzhengjie ~]# [root@yinzhengjie ~]#
18>.查看hadoop的Sequencefile文件内容
[yinzhengjie@s101 data]$ hdfs dfs -text file:///home/yinzhengjie/data/seq 18/06/01 06:32:32 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 18/06/01 06:32:32 INFO compress.CodecPool: Got brand-new decompressor [.deflate] yinzhengjie 18 [yinzhengjie@s101 data]$
19>.使用df命令查看可用空间
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -df Filesystem Size Used Available Use% hdfs://yinzhengjie-hdfs-ha 1804514672640 4035805184 1800478867456 0% [hdfs@node101.yinzhengjie.org.cn ~]$
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -df -h Filesystem Size Used Available Use% hdfs://yinzhengjie-hdfs-ha 1.6 T 3.8 G 1.6 T 0% [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$
20>.降低复制因子
[hdfs@node105.yinzhengjie.org.cn ~]$ hdfs dfs -ls /user/yinzhengjie/data/ Found 2 items -rw-r--r-- 3 hdfs supergroup 0 2019-05-28 15:16 /user/yinzhengjie/data/1.txt drwxr-xr-x - root supergroup 0 2019-05-22 19:46 /user/yinzhengjie/data/day001 [hdfs@node105.yinzhengjie.org.cn ~]$ [hdfs@node105.yinzhengjie.org.cn ~]$ hdfs dfs -setrep -w 2 /user/yinzhengjie/data/1.txt Replication 2 set: /user/yinzhengjie/data/1.txt Waiting for /user/yinzhengjie/data/1.txt ... done [hdfs@node105.yinzhengjie.org.cn ~]$ [hdfs@node105.yinzhengjie.org.cn ~]$ hdfs dfs -ls /user/yinzhengjie/data/ Found 2 items -rw-r--r-- 2 hdfs supergroup 0 2019-05-28 15:16 /user/yinzhengjie/data/1.txt drwxr-xr-x - root supergroup 0 2019-05-22 19:46 /user/yinzhengjie/data/day001 [hdfs@node105.yinzhengjie.org.cn ~]$
21>.使用du命令查看已用空间
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -du /user/yinzhengjie/data/day001 1000000000 3000000000 /user/yinzhengjie/data/day001/test_input 1000000165 1000001650 /user/yinzhengjie/data/day001/test_output 24 72 /user/yinzhengjie/data/day001/ts_validate [hdfs@node101.yinzhengjie.org.cn ~]$
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -du -h /user/yinzhengjie/data/day001 953.7 M 2.8 G /user/yinzhengjie/data/day001/test_input 953.7 M 953.7 M /user/yinzhengjie/data/day001/test_output 24 72 /user/yinzhengjie/data/day001/ts_validate [hdfs@node101.yinzhengjie.org.cn ~]$
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -du -s -h /user/yinzhengjie/data/day001 1.9 G 3.7 G /user/yinzhengjie/data/day001 [hdfs@node101.yinzhengjie.org.cn ~]$
三.hdfs与getconf结合使用的案例
1>.获取NameNode的节点名称(可能包含多个)
[yinzhengjie@s101 ~]$ hdfs getconf hdfs getconf is utility for getting configuration information from the config file. hadoop getconf [-namenodes] gets list of namenodes in the cluster. [-secondaryNameNodes] gets list of secondary namenodes in the cluster. [-backupNodes] gets list of backup nodes in the cluster. [-includeFile] gets the include file path that defines the datanodes that can join the cluster. [-excludeFile] gets the exclude file path that defines the datanodes that need to decommissioned. [-nnRpcAddresses] gets the namenode rpc addresses [-confKey [key]] gets a specific key from the configuration [yinzhengjie@s101 ~]$ hdfs getconf -namenodes s101 [yinzhengjie@s101 ~]$
2>.获取hdfs最小块信息（默认大小为1M,即1048576字节，如果想要修改的话必须为512的倍数，因为HDFS底层传输数据是每512字节进行校验）
[yinzhengjie@s101 ~]$ hdfs getconf hdfs getconf is utility for getting configuration information from the config file. hadoop getconf [-namenodes] gets list of namenodes in the cluster. [-secondaryNameNodes] gets list of secondary namenodes in the cluster. [-backupNodes] gets list of backup nodes in the cluster. [-includeFile] gets the include file path that defines the datanodes that can join the cluster. [-excludeFile] gets the exclude file path that defines the datanodes that need to decommissioned. [-nnRpcAddresses] gets the namenode rpc addresses [-confKey [key]] gets a specific key from the configuration [yinzhengjie@s101 ~]$ hdfs getconf -confKey dfs.namenode.fs-limits.min-block-size 1048576 [yinzhengjie@s101 ~]$
3>.查找hdfs的NameNode的RPC地址
[root@node101.yinzhengjie.org.cn ~]# hdfs getconf -nnRpcAddresses calculation101.aggrx:8022 calculation111.aggrx:8022 [root@node101.yinzhengjie.org.cn ~]#
四.hdfs与dfsadmin结合使用的案例
1>.查看hdfs dfsadmin的帮助信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin Usage: hdfs dfsadmin Note: Administrative commands can only be run as the HDFS superuser. [-report [-live] [-dead] [-decommissioning]] [-safemode <enter | leave | get | wait>] [-saveNamespace] [-rollEdits] [-restoreFailedStorage true|false|check] [-refreshNodes] [-setQuota <quota> <dirname>...<dirname>] [-clrQuota <dirname>...<dirname>] [-setSpaceQuota <quota> <dirname>...<dirname>] [-clrSpaceQuota <dirname>...<dirname>] [-finalizeUpgrade] [-rollingUpgrade [<query|prepare|finalize>]] [-refreshServiceAcl] [-refreshUserToGroupsMappings] [-refreshSuperUserGroupsConfiguration] [-refreshCallQueue] [-refresh <host:ipc_port> <key> [arg1..argn] [-reconfig <datanode|...> <host:ipc_port> <start|status|properties>] [-printTopology] [-refreshNamenodes datanode_host:ipc_port] [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]] [-setBalancerBandwidth <bandwidth in bytes per second>] [-fetchImage <local directory>] [-allowSnapshot <snapshotDir>] [-disallowSnapshot <snapshotDir>] [-shutdownDatanode <datanode_host:ipc_port> [upgrade]] [-getDatanodeInfo <datanode_host:ipc_port>] [-metasave filename] [-triggerBlockReport [-incremental] <datanode_host:ipc_port>] [-listOpenFiles [-blockingDecommission] [-path <path>]] [-help [cmd]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] [hdfs@node101.yinzhengjie.org.cn ~]$
2>.查看指定命令的帮助信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -help rollEdits -rollEdits: Rolls the edit log. [hdfs@node101.yinzhengjie.org.cn ~]$
3>.手动滚动日志（关于日志滚动更详细的用法请参考：https://www.cnblogs.com/yinzhengjie/p/9098092.html）
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -rollEdits Successfully rolled edit logs. New segment starts at txid 82206 [hdfs@node101.yinzhengjie.org.cn ~]$
4>.查看当前的模式
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$
5>.进入安全模式
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode enter Safe mode is ON in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is ON in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is ON in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is ON in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$
6>.离开安全模式
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is ON in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is ON in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode leave Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$
7>.安全模式的wait状态
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode Usage: hdfs dfsadmin [-safemode enter | leave | get | wait] [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode wait Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -safemode get Safe mode is OFF in node105.yinzhengjie.org.cn/10.1.2.105:8020 Safe mode is OFF in node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$
8>.检查HDFS集群的状态
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -help report -report [-live] [-dead] [-decommissioning]: Reports basic filesystem information and statistics. The dfs usage can be different from "du" usage, because it measures raw space used by replication, checksums, snapshots and etc. on all the DNs. Optional flags may be used to filter the list of displayed DNs. [hdfs@node101.yinzhengjie.org.cn ~]$
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -report Configured Capacity: 1804514672640 (1.64 TB) #此集群中HDFS已配置的容量 Present Capacity: 1804514672640 (1.64 TB) #现有的HDFS容量。 DFS Remaining: 1800478859331 (1.64 TB) #剩余的HDFS容量。 DFS Used: 4035813309 (3.76 GB) #HDFS使用存储的统计信息，按照文件大小统计。 DFS Used%: 0.22% #同上，这里安装的是百分比统计。 Under replicated blocks: 1 #显示是否有任何未充分复制的块。 Blocks with corrupt replicas: 0 #显示是否有损坏的块。 Missing blocks: 0 #显示是否有丢失的块。 Missing blocks (with replication factor 1): 0 #同上 ------------------------------------------------- Live datanodes (4): #显示集群中有多少个DataNode是活动的并可用。 Name: 10.1.2.102:50010 (node102.yinzhengjie.org.cn) #主机名称或者机架名称 Hostname: node102.yinzhengjie.org.cn #主机名 Rack: /default #默认机架 Decommission Status : Normal #当前节点的DataNode的状态（已停用或者未停用，Normal表示正常的） Configured Capacity: 451128668160 (420.15 GB) #DataNode的配置和使用的容量 DFS Used: 1089761280 (1.01 GB) Non DFS Used: 0 (0 B) DFS Remaining: 450038906880 (419.13 GB) DFS Used%: 0.24% DFS Remaining%: 99.76% Configured Cache Capacity: 1782579200 (1.66 GB) Cache Used: 0 (0 B) #缓存使用情况统计信息（如果已配置） Cache Remaining: 1782579200 (1.66 GB) Cache Used%: 0.00% Cache Remaining%: 100.00% Xceivers: 2 Last contact: Tue May 28 14:55:25 CST 2019 Name: 10.1.2.103:50010 (node103.yinzhengjie.org.cn) Hostname: node103.yinzhengjie.org.cn Rack: /default Decommission Status : Normal Configured Capacity: 451128668160 (420.15 GB) DFS Used: 1009278976 (962.52 MB) Non DFS Used: 0 (0 B) DFS Remaining: 450119389184 (419.21 GB) DFS Used%: 0.22% DFS Remaining%: 99.78% Configured Cache Capacity: 1782579200 (1.66 GB) Cache Used: 0 (0 B) Cache Remaining: 1782579200 (1.66 GB) Cache Used%: 0.00% Cache Remaining%: 100.00% Xceivers: 2 Last contact: Tue May 28 14:55:26 CST 2019 Name: 10.1.2.104:50010 (node104.yinzhengjie.org.cn) Hostname: node104.yinzhengjie.org.cn Rack: /default Decommission Status : Normal Configured Capacity: 451128668160 (420.15 GB) DFS Used: 981598141 (936.12 MB) Non DFS Used: 0 (0 B) DFS Remaining: 450147070019 (419.23 GB) DFS Used%: 0.22% DFS Remaining%: 99.78% Configured Cache Capacity: 1782579200 (1.66 GB) Cache Used: 0 (0 B) Cache Remaining: 1782579200 (1.66 GB) Cache Used%: 0.00% Cache Remaining%: 100.00% Xceivers: 2 Last contact: Tue May 28 14:55:27 CST 2019 Name: 10.1.2.105:50010 (node105.yinzhengjie.org.cn) Hostname: node105.yinzhengjie.org.cn Rack: /default Decommission Status : Normal Configured Capacity: 451128668160 (420.15 GB) DFS Used: 955174912 (910.93 MB) Non DFS Used: 0 (0 B) DFS Remaining: 450173493248 (419.26 GB) DFS Used%: 0.21% DFS Remaining%: 99.79% Configured Cache Capacity: 942669824 (899 MB) Cache Used: 0 (0 B) Cache Remaining: 942669824 (899 MB) Cache Used%: 0.00% Cache Remaining%: 100.00% Xceivers: 2 Last contact: Tue May 28 14:55:24 CST 2019 [hdfs@node101.yinzhengjie.org.cn ~]$
9>.目录配额（计算目录下的所有文件的总个数，如果为1，表示目录下不能放文件，即空目录！）
[root@yinzhengjie ~]# ll total 16 -rw-r--r--. 1 root root 6 May 27 17:41 index.html -rw-r--r--. 1 root root 12 May 27 17:42 nginx.conf -rw-r--r--. 1 root root 11 May 27 17:42 yinzhengjie.sql -rw-r--r--. 1 root root 7 May 27 18:20 zabbix.conf [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls / [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -mkdir -p /data/etc [root@yinzhengjie ~]# hdfs dfs -ls / Found 1 items drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfsadmin -setQuota 3 /data [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -put index.html /data [root@yinzhengjie ~]# hdfs dfs -put yinzhengjie.sql /data put: The NameSpace quota (directories and files) of directory /data is exceeded: quota=3 file count=4 [root@yinzhengjie ~]# hdfs dfs -ls /data Found 2 items drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html [root@yinzhengjie ~]# hdfs dfsadmin -setQuota 5 /data [root@yinzhengjie ~]# hdfs dfs -put yinzhengjie.sql /data [root@yinzhengjie ~]# hdfs dfs -ls /data Found 3 items drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/yinzhengjie.sql [root@yinzhengjie ~]#
10>.空间配额（计算目录下所有文件的总大小，包括副本数，因此空间配最小的值可以得到一个等式："空间配最小的值 >= 需要上传文件的实际大小 * 副本数"）
[root@yinzhengjie ~]# ll total 181196 -rw-r--r--. 1 root root 185540433 May 27 19:34 jdk-8u131-linux-x64.tar.gz -rw-r--r--. 1 root root 12 May 27 19:38 name.txt [root@yinzhengjie ~]# [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls -R / drwxr-xr-x - root supergroup 0 2018-05-27 20:27 /data drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/yinzhengjie.sql [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfsadmin -setSpaceQuota 134217745 /data #这里设置/data 目录配额大小为128M，我测试机器是伪分布式，指定副本数为1，因此设置目录配个大小为128 [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -put name.txt /data #上传文件的大小/data目录中去，发现可以正常上传 [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -ls -R / #检查上传已经成功 drwxr-xr-x - root supergroup 0 2018-05-27 20:28 /data drwxr-xr-x - root supergroup 0 2018-05-27 18:24 /data/etc -rw-r--r-- 1 root supergroup 6 2018-05-27 18:25 /data/index.html -rw-r--r-- 1 root supergroup 12 2018-05-27 20:28 /data/name.txt -rw-r--r-- 1 root supergroup 11 2018-05-27 18:27 /data/yinzhengjie.sql [root@yinzhengjie ~]# [root@yinzhengjie ~]# hdfs dfs -put jdk-8u131-linux-x64.tar.gz /data #当我们上传第二个文件时，就会报以下的错误！ 18/05/27 20:29:40 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /data is exceeded: quota = 134217745 B = 128.00 MB but diskspace consumed = 134217757 B = 128.00 MB at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211) at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:878) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:707) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:666) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:491) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3573) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.storeAllocatedBlock(FSNamesystem.java:3157) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3038) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1458) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.DSQuotaExceededException): The DiskSpace quota of /data is exceeded: quota = 134217745 B = 128.00 MB but diskspace consumed = 134217757 B = 128.00 MB at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211) at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:878) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:707) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:666) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:491) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3573) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.storeAllocatedBlock(FSNamesystem.java:3157) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3038) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy11.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455) ... 2 more put: The DiskSpace quota of /data is exceeded: quota = 134217745 B = 128.00 MB but diskspace consumed = 134217757 B = 128.00 MB [root@yinzhengjie ~]#
11>.清空配额管理
[root@yinzhengjie ~]# hdfs dfsadmin -clrSpaceQuota /data [root@yinzhengjie ~]# echo $? 0 [root@yinzhengjie ~]#
12>.对某个目录启用快照功能（快照功能默认为禁用状态）
[root@yinzhengjie ~]# hdfs dfsadmin -allowSnapShot /data Allowing snaphot on /data succeeded [root@yinzhengjie ~]#
13>.对某个目录禁用快照功能
[root@yinzhengjie ~]# hdfs dfsadmin -disallowSnapShot /data Disallowing snaphot on /data succeeded [root@yinzhengjie ~]#
14>.获取某个namenode的节点状态
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs haadmin -getServiceState namenode23 active [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs haadmin -getServiceState namenode31 standby [hdfs@node101.yinzhengjie.org.cn ~]$
15>.使用dfsadmin -metasave命令提供的信息比dfsadmin -report命令提供的更多。使用此命令可以获取各种的块相关的信息（比如：块总数，正在等待复制的块，当前正在复制的块）
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfs -ls / Found 5 items drwxr-xr-x - hbase hbase 0 2019-05-25 00:03 /hbase drwxr-xr-x - hdfs supergroup 0 2019-05-22 19:17 /jobtracker drwxr-xr-x - hdfs supergroup 0 2019-05-28 12:11 /system drwxrwxrwt - hdfs supergroup 0 2019-05-23 13:37 /tmp drwxrwxrwx - hdfs supergroup 0 2019-05-23 13:47 /user [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ hdfs dfsadmin -metasave /hbase #我们获取某个目录的详细信息，允许成功后会有以下输出，并在“/var/log/hadoop-hdfs/”目录中创建一个文件，该文件名称和咱们这里输入的path名称一致，即“hbase” Created metasave file /hbase in the log directory of namenode node105.yinzhengjie.org.cn/10.1.2.105:8020 Created metasave file /hbase in the log directory of namenode node101.yinzhengjie.org.cn/10.1.2.101:8020 [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$ cat /var/log/hadoop-hdfs/hbase #查看输入文件包含有关块的以下信息 2027 files and directories, 60 blocks = 2087 total Live Datanodes: 4 Dead Datanodes: 0 Metasave: Blocks waiting for replication: 0 Mis-replicated blocks that have been postponed: Metasave: Blocks being replicated: 0 Metasave: Blocks 0 waiting deletion from 0 datanodes. Metasave: Number of datanodes: 4 10.1.2.102:50010 /default IN 451128668160(420.15 GB) 1089761280(1.01 GB) 0.24% 450038906880(419.13 GB) 1782579200(1.66 GB) 0(0 B) 0.00% 1782579200(1.66 GB) Tue May 28 15:26:58 CST 2019 10.1.2.105:50010 /default IN 451128668160(420.15 GB) 955174912(910.93 MB) 0.21% 450173493248(419.26 GB) 942669824(899 MB) 0(0 B) 0.00% 942669824(899 MB) Tue May 28 15:26:59 CST 2019 10.1.2.103:50010 /default IN 451128668160(420.15 GB) 1009278976(962.52 MB) 0.22% 450119389184(419.21 GB) 1782579200(1.66 GB) 0(0 B) 0.00% 1782579200(1.66 GB) Tue May 28 15:26:58 CST 2019 10.1.2.104:50010 /default IN 451128668160(420.15 GB) 981590016(936.12 MB) 0.22% 450147078144(419.23 GB) 1782579200(1.66 GB) 0(0 B) 0.00% 1782579200(1.66 GB) Tue May 28 15:26:57 CST 2019 [hdfs@node101.yinzhengjie.org.cn ~]$
五.hdfs与fsck结合使用的案例
1>.查看hdfs文件系统信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs fsck / Connecting to namenode via http://node101.yinzhengjie.org.cn:50070/fsck?ugi=hdfs&path=%2F FSCK started by hdfs (auth:SIMPLE) from /10.1.2.101 for path / at Thu May 23 14:32:41 CST 2019 ....................................... /user/yinzhengjie/data/day001/test_output/_partition.lst: Under replicated BP-1230584423-10.1.2.101-1558513980919:blk_1073742006_1182. Target Replicas is 10 but found 4 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). ..................Status: HEALTHY #代表这次HDFS上block检查结果 Total size: 2001318792 B (Total open files size: 498 B) #代表根目录下文件总大小 Total dirs: 189 #代表检测的目录下总共有多少目录 Total files: 57 #代表检测的目录下总共有多少文件 Total symlinks: 0 (Files currently being written: 7) #代表检测的目录下有多少个符号链接 Total blocks (validated): 58 (avg. block size 34505496 B) (Total open file blocks (not validated): 6) #代表检测的目录下有多少的block是有效的。 Minimally replicated blocks: 58 (100.0 %) #代表拷贝的最小block块数。 Over-replicated blocks: 0 (0.0 %) #代表当前副本数大于指定副本数的block数量。 Under-replicated blocks: 1 (1.7241379 %) #代表当前副本书小于指定副本数的block数量。 Mis-replicated blocks: 0 (0.0 %) #代表丢失的block块数量。 Default replication factor: 3 #代表默认的副本数（自身一份，默认拷贝两份）。 Average block replication: 2.3965516 #代表块的平均复制数，即平均备份的数目，Default replication factor 的值为3，因此需要备份在备份2个，这里的平均备份数等于2是理想值，如果大于2说明可能有多余的副本数存在。 Corrupt blocks: 0 #代表坏的块数，这个指不为0，说明当前集群有不可恢复的块，即数据丢失啦！ Missing replicas: 6 (4.137931 %) #代表丢失的副本数 Number of data-nodes: 4 #代表有多好个DN节点 Number of racks: 1 #代表有多少个机架 FSCK ended at Thu May 23 14:32:41 CST 2019 in 7 milliseconds The filesystem under path '/' is HEALTHY [hdfs@node101.yinzhengjie.org.cn ~]$
2>.fsck指令显示HDFS块信息
[hdfs@node101.yinzhengjie.org.cn ~]$ hdfs fsck / -files -blocks Connecting to namenode via http://node101.yinzhengjie.org.cn:50070/fsck?ugi=hdfs&files=1&blocks=1&path=%2F FSCK started by hdfs (auth:SIMPLE) from /10.1.2.101 for path / at Thu May 23 14:30:51 CST 2019 / <dir> /hbase <dir> /hbase/.tmp <dir> /hbase/MasterProcWALs <dir> /hbase/MasterProcWALs/state-00000000000000000002.log 30 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743172_2348 len=30 Live_repl=3 /hbase/WALs <dir> /hbase/WALs/node102.yinzhengjie.org.cn,60020,1558589829098 <dir> /hbase/WALs/node102.yinzhengjie.org.cn,60020,1558590692594 <dir> /hbase/WALs/node103.yinzhengjie.org.cn,60020,1558589826957 <dir> /hbase/WALs/node103.yinzhengjie.org.cn,60020,1558590692071 <dir> /hbase/WALs/node104.yinzhengjie.org.cn,60020,1558590690690 <dir> /hbase/WALs/node105.yinzhengjie.org.cn,60020,1558589830953 <dir> /hbase/WALs/node105.yinzhengjie.org.cn,60020,1558590695092 <dir> /hbase/data <dir> /hbase/data/default <dir> /hbase/data/hbase <dir> /hbase/data/hbase/meta <dir> /hbase/data/hbase/meta/.tabledesc <dir> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 398 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743148_2324 len=398 Live_repl=3 /hbase/data/hbase/meta/.tmp <dir> /hbase/data/hbase/meta/1588230740 <dir> /hbase/data/hbase/meta/1588230740/.regioninfo 32 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743147_2323 len=32 Live_repl=3 /hbase/data/hbase/meta/1588230740/info <dir> /hbase/data/hbase/meta/1588230740/info/4502037817cf408da4c31f38632d386e 1389 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743185_2361 len=1389 Live_repl=3 /hbase/data/hbase/meta/1588230740/info/cc017533033a4b57904c694bf156d9a6 1529 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743168_2344 len=1529 Live_repl=3 /hbase/data/hbase/meta/1588230740/recovered.edits <dir> /hbase/data/hbase/meta/1588230740/recovered.edits/20.seqid 0 bytes, 0 block(s): OK /hbase/data/hbase/namespace <dir> /hbase/data/hbase/namespace/.tabledesc <dir> /hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 312 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743156_2332 len=312 Live_repl=3 /hbase/data/hbase/namespace/.tmp <dir> /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7 <dir> /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7/.regioninfo 42 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743157_2333 len=42 Live_repl=3 /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7/info <dir> /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7/info/2efca8e894a4419f9d6e86bb8c8c736b 1079 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743166_2342 len=1079 Live_repl=3 /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7/recovered.edits <dir> /hbase/data/hbase/namespace/27b26f72bc8dabfb5f8dae587ad9cda7/recovered.edits/11.seqid 0 bytes, 0 block(s): OK /hbase/hbase.id 42 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743146_2322 len=42 Live_repl=3 /hbase/hbase.version 7 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743145_2321 len=7 Live_repl=3 /hbase/oldWALs <dir> /jobtracker <dir> /jobtracker/jobsInfo <dir> /jobtracker/jobsInfo/job_201905221917_0001.info 1013 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743091_2267 len=1013 Live_repl=3 /jobtracker/jobsInfo/job_201905221917_0002.info 1013 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743117_2293 len=1013 Live_repl=3 /tmp <dir> /tmp/.cloudera_health_monitoring_canary_files <dir> /tmp/hbase-staging <dir> /tmp/hbase-staging/DONOTERASE <dir> /tmp/hive <dir> /tmp/hive/hive <dir> /tmp/hive/hive/2cd86efc-ec86-40ac-8472-d78c9e6b90a4 <dir> /tmp/hive/hive/2cd86efc-ec86-40ac-8472-d78c9e6b90a4/_tmp_space.db <dir> /tmp/hive/root <dir> /tmp/logs <dir> /tmp/logs/root <dir> /tmp/logs/root/logs <dir> /tmp/logs/root/logs/application_1558520562958_0001 <dir> /tmp/logs/root/logs/application_1558520562958_0001/node102.yinzhengjie.org.cn_8041 3197 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742001_1177 len=3197 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0001/node103.yinzhengjie.org.cn_8041 3197 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742002_1178 len=3197 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0001/node104.yinzhengjie.org.cn_8041 35641 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742003_1179 len=35641 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0002 <dir> /tmp/logs/root/logs/application_1558520562958_0002/node102.yinzhengjie.org.cn_8041 68827 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742034_1210 len=68827 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0002/node103.yinzhengjie.org.cn_8041 191005 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742037_1213 len=191005 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0002/node104.yinzhengjie.org.cn_8041 80248 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742036_1212 len=80248 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0002/node105.yinzhengjie.org.cn_8041 64631 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742035_1211 len=64631 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0003 <dir> /tmp/logs/root/logs/application_1558520562958_0003/node102.yinzhengjie.org.cn_8041 23256 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742056_1232 len=23256 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0003/node103.yinzhengjie.org.cn_8041 35498 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742055_1231 len=35498 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0003/node104.yinzhengjie.org.cn_8041 131199 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742057_1233 len=131199 Live_repl=3 /tmp/logs/root/logs/application_1558520562958_0003/node105.yinzhengjie.org.cn_8041 19428 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742054_1230 len=19428 Live_repl=3 /tmp/mapred <dir> /tmp/mapred/system <dir> /tmp/mapred/system/jobtracker.info 4 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073741947_1123 len=4 Live_repl=3 /tmp/mapred/system/seq-000000000002 <dir> /tmp/mapred/system/seq-000000000002/jobtracker.info 4 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743173_2349 len=4 Live_repl=3 /user <dir> /user/history <dir> /user/history/done <dir> /user/history/done/2019 <dir> /user/history/done/2019/05 <dir> /user/history/done/2019/05/22 <dir> /user/history/done/2019/05/22/000000 <dir> /user/history/done/2019/05/22/000000/job_1558520562958_0001-1558525119627-root-TeraGen-1558525149229-2-0-SUCCEEDED-root.users.root-1558525126064.jhist 18715 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073741999_1175 len=18715 Live_repl=3 /user/history/done/2019/05/22/000000/job_1558520562958_0001_conf.xml 153279 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742000_1176 len=153279 Live_repl=3 /user/history/done/2019/05/22/000000/job_1558520562958_0002-1558525279858-root-TeraSort-1558525343312-8-16-SUCCEEDED-root.users.root-1558525284386.jhist 102347 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742032_1208 len=102347 Live_repl=1 /user/history/done/2019/05/22/000000/job_1558520562958_0002_conf.xml 154575 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742033_1209 len=154575 Live_repl=1 /user/history/done/2019/05/22/000000/job_1558520562958_0003-1558525587458-root-TeraValidate-1558525623716-16-1-SUCCEEDED-root.users.root-1558525591653.jhist 71381 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742052_1228 len=71381 Live_repl=3 /user/history/done/2019/05/22/000000/job_1558520562958_0003_conf.xml 153701 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742053_1229 len=153701 Live_repl=3 /user/history/done_intermediate <dir> /user/history/done_intermediate/root <dir> /user/hive <dir> /user/hive/warehouse <dir> /user/hive/warehouse/page_view <dir> /user/hive/warehouse/page_view/PageViewData.csv 1584 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073743063_2239 len=1584 Live_repl=3 /user/hue <dir> /user/hue/.Trash <dir> /user/hue/.Trash/190523130000 <dir> /user/hue/.Trash/190523130000/user <dir> /user/hue/.Trash/190523130000/user/hue <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0/p2=420 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1/p2=421 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586450195 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586450195/p1=p0 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586450195/p1=p0/p2=420 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586450195/p1=p1 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586450195/p1=p1/p2=421 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586750132 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586750132/p1=p0 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586750132/p1=p0/p2=420 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586750132/p1=p1 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558586750132/p1=p1/p2=421 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587050151 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587050151/p1=p0 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587050151/p1=p0/p2=420 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587050151/p1=p1 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587050151/p1=p1/p2=421 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587350133 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587350133/p1=p0 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587350133/p1=p0/p2=420 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587350133/p1=p1 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558587350133/p1=p1/p2=421 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558586146864 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558586450229 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558586750160 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558587050181 <dir> /user/hue/.Trash/190523130000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558587350162 <dir> /user/hue/.Trash/190523140000 <dir> /user/hue/.Trash/190523140000/user <dir> /user/hue/.Trash/190523140000/user/hue <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558589871630 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558589871630/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558589871630/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558589871630/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558589871630/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590170701 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590170701/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590170701/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590170701/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590170701/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590470694 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590470694/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590470694/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590470694/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590470694/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590772147 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590772147/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590772147/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590772147/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558590772147/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591070813 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591070813/p1=p0 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591070813/p1=p0/p2=420 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591070813/p1=p1 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591070813/p1=p1/p2=421 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558587650383 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558589871818 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558590170843 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558590470827 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558590772354 <dir> /user/hue/.Trash/190523140000/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558591070964 <dir> /user/hue/.Trash/Current <dir> /user/hue/.Trash/Current/user <dir> /user/hue/.Trash/Current/user/hue <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591670694 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591670694/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591670694/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591670694/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591670694/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591970721 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591970721/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591970721/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591970721/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558591970721/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592270766 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592270766/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592270766/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592270766/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592270766/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592570728 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592570728/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592570728/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592570728/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592570728/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592870675 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592870675/p1=p0 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592870675/p1=p0/p2=420 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592870675/p1=p1 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df10872/cm_test_table1558592870675/p1=p1/p2=421 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558591370839 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558591670884 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558591970864 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558592270913 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558592570869 <dir> /user/hue/.Trash/Current/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_4e4cb72650ff61877f815b077df108721558592870812 <dir> /user/hue/.cloudera_manager_hive_metastore_canary <dir> /user/impala <dir> /user/root <dir> /user/root/.staging <dir> /user/yinzhengjie <dir> /user/yinzhengjie/data <dir> /user/yinzhengjie/data/day001 <dir> /user/yinzhengjie/data/day001/test_input <dir> /user/yinzhengjie/data/day001/test_input/_SUCCESS 0 bytes, 0 block(s): OK /user/yinzhengjie/data/day001/test_input/part-m-00000 500000000 bytes, 4 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073741990_1166 len=134217728 Live_repl=3 1. BP-1230584423-10.1.2.101-1558513980919:blk_1073741992_1168 len=134217728 Live_repl=3 2. BP-1230584423-10.1.2.101-1558513980919:blk_1073741994_1170 len=134217728 Live_repl=3 3. BP-1230584423-10.1.2.101-1558513980919:blk_1073741996_1172 len=97346816 Live_repl=3 /user/yinzhengjie/data/day001/test_input/part-m-00001 500000000 bytes, 4 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073741989_1165 len=134217728 Live_repl=3 1. BP-1230584423-10.1.2.101-1558513980919:blk_1073741991_1167 len=134217728 Live_repl=3 2. BP-1230584423-10.1.2.101-1558513980919:blk_1073741993_1169 len=134217728 Live_repl=3 3. BP-1230584423-10.1.2.101-1558513980919:blk_1073741995_1171 len=97346816 Live_repl=3 /user/yinzhengjie/data/day001/test_output <dir> /user/yinzhengjie/data/day001/test_output/_SUCCESS 0 bytes, 0 block(s): OK /user/yinzhengjie/data/day001/test_output/_partition.lst 165 bytes, 1 block(s): Under replicated BP-1230584423-10.1.2.101-1558513980919:blk_1073742006_1182. Target Replicas is 10 but found 4 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742006_1182 len=165 Live_repl=4 /user/yinzhengjie/data/day001/test_output/part-r-00000 62307000 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742015_1191 len=62307000 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00001 62782700 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742016_1192 len=62782700 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00002 61993900 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742017_1193 len=61993900 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00003 63217700 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742019_1195 len=63217700 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00004 62628600 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742018_1194 len=62628600 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00005 62884100 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742020_1196 len=62884100 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00006 63079700 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742021_1197 len=63079700 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00007 61421800 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742022_1198 len=61421800 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00008 61319800 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742023_1199 len=61319800 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00009 61467300 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742025_1201 len=61467300 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00010 62823400 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742024_1200 len=62823400 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00011 63392200 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742026_1202 len=63392200 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00012 62889200 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742027_1203 len=62889200 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00013 62953000 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742028_1204 len=62953000 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00014 62072800 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742029_1205 len=62072800 Live_repl=1 /user/yinzhengjie/data/day001/test_output/part-r-00015 62766800 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742030_1206 len=62766800 Live_repl=1 /user/yinzhengjie/data/day001/ts_validate <dir> /user/yinzhengjie/data/day001/ts_validate/_SUCCESS 0 bytes, 0 block(s): OK /user/yinzhengjie/data/day001/ts_validate/part-r-00000 24 bytes, 1 block(s): OK 0. BP-1230584423-10.1.2.101-1558513980919:blk_1073742050_1226 len=24 Live_repl=3 Status: HEALTHY Total size: 2001318792 B (Total open files size: 498 B) Total dirs: 189 Total files: 57 Total symlinks: 0 (Files currently being written: 7) Total blocks (validated): 58 (avg. block size 34505496 B) (Total open file blocks (not validated): 6) Minimally replicated blocks: 58 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 1 (1.7241379 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.3965516 Corrupt blocks: 0 Missing replicas: 6 (4.137931 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Thu May 23 14:30:51 CST 2019 in 8 milliseconds The filesystem under path '/' is HEALTHY [hdfs@node101.yinzhengjie.org.cn ~]$ [hdfs@node101.yinzhengjie.org.cn ~]$
3>.
六.hdfs与oiv结合我使用案例
1>.查看hdfs oiv的帮助信息
[yinzhengjie@s101 ~]$ hdfs oiv Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE Offline Image Viewer View a Hadoop fsimage INPUTFILE using the specified PROCESSOR, saving the results in OUTPUTFILE. The oiv utility will attempt to parse correctly formed image files and will abort fail with mal-formed image files. The tool works offline and does not require a running cluster in order to process an image file. The following image processors are available: * XML: This processor creates an XML document with all elements of the fsimage enumerated, suitable for further analysis by XML tools. * FileDistribution: This processor analyzes the file size distribution in the image. -maxSize specifies the range [0, maxSize] of file sizes to be analyzed (128GB by default). -step defines the granularity of the distribution. (2MB by default) * Web: Run a viewer to expose read-only WebHDFS API. -addr specifies the address to listen. (localhost:5978 by default) * Delimited (experimental): Generate a text file with all of the elements common to both inodes and inodes-under-construction, separated by a delimiter. The default delimiter is \t, though this may be changed via the -delimiter argument. Required command line arguments: -i,--inputFile <arg> FSImage file to process. Optional command line arguments: -o,--outputFile <arg> Name of output file. If the specified file exists, it will be overwritten. (output to stdout by default) -p,--processor <arg> Select which type of processor to apply against image file. (XML|FileDistribution|Web|Delimited) (Web by default) -delimiter <arg> Delimiting string to use with Delimited processor. -t,--temp <arg> Use temporary dir to cache intermediate result to generate Delimited outputs. If not set, Delimited processor constructs the namespace in memory before outputting text. -h,--help Display usage information and exit [yinzhengjie@s101 ~]$
2>.使用oiv命令查询hadoop镜像文件
[yinzhengjie@s101 ~]$ ll total 0 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ ls -lh /home/yinzhengjie/hadoop/dfs/name/current/ | grep fsimage -rw-rw-r--. 1 yinzhengjie yinzhengjie 1.2K May 27 06:02 fsimage_0000000000000000767 -rw-rw-r--. 1 yinzhengjie yinzhengjie 62 May 27 06:02 fsimage_0000000000000000767.md5 -rw-rw-r--. 1 yinzhengjie yinzhengjie 1.4K May 27 07:58 fsimage_0000000000000000932 -rw-rw-r--. 1 yinzhengjie yinzhengjie 62 May 27 07:58 fsimage_0000000000000000932.md5 [yinzhengjie@s101 ~]$ hdfs oiv -i ./hadoop/dfs/name/current/fsimage_0000000000000000767 -o yinzhengjie.xml -p XML [yinzhengjie@s101 ~]$ ll total 8 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell -rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml [yinzhengjie@s101 ~]$
3>.
七.hdfs与oev结合我使用案例
1>.查看hdfs oev的帮助信息
[yinzhengjie@s101 ~]$ hdfs oev Usage: bin/hdfs oev [OPTIONS] -i INPUT_FILE -o OUTPUT_FILE Offline edits viewer Parse a Hadoop edits log file INPUT_FILE and save results in OUTPUT_FILE. Required command line arguments: -i,--inputFile <arg> edits file to process, xml (case insensitive) extension means XML format, any other filename means binary format -o,--outputFile <arg> Name of output file. If the specified file exists, it will be overwritten, format of the file is determined by -p option Optional command line arguments: -p,--processor <arg> Select which type of processor to apply against image file, currently supported processors are: binary (native binary format that Hadoop uses), xml (default, XML format), stats (prints statistics about edits file) -h,--help Display usage information and exit -f,--fix-txids Renumber the transaction IDs in the input, so that there are no gaps or invalid transaction IDs. -r,--recover When reading binary edit logs, use recovery mode. This will give you the chance to skip corrupt parts of the edit log. -v,--verbose More verbose output, prints the input and output filenames, for processors that write to a file, also output to screen. On large image files this will dramatically increase processing time (default is false). Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] [yinzhengjie@s101 ~]$
2>.使用oev命令查询hadoop的编辑日志文件
[yinzhengjie@s101 ~]$ ls -lh /home/yinzhengjie/hadoop/dfs/name/current/ | grep edits | tail -5 -rw-rw-r--. 1 yinzhengjie yinzhengjie 42 May 27 08:33 edits_0000000000000001001-0000000000000001002 -rw-rw-r--. 1 yinzhengjie yinzhengjie 42 May 27 08:34 edits_0000000000000001003-0000000000000001004 -rw-rw-r--. 1 yinzhengjie yinzhengjie 42 May 27 08:35 edits_0000000000000001005-0000000000000001006 -rw-rw-r--. 1 yinzhengjie yinzhengjie 42 May 27 08:36 edits_0000000000000001007-0000000000000001008 -rw-rw-r--. 1 yinzhengjie yinzhengjie 1.0M May 27 08:36 edits_inprogress_0000000000000001009 [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ ll total 8 drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell -rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ hdfs oev -i ./hadoop/dfs/name/current/edits_0000000000000001007-0000000000000001008 -o edits.xml -p XML [yinzhengjie@s101 ~]$ ll total 12 -rw-rw-r--. 1 yinzhengjie yinzhengjie 315 May 27 08:39 edits.xml drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell -rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml [yinzhengjie@s101 ~]$ [yinzhengjie@s101 ~]$ cat edits.xml <?xml version="1.0" encoding="UTF-8"?> <EDITS> <EDITS_VERSION>-63</EDITS_VERSION> <RECORD> <OPCODE>OP_START_LOG_SEGMENT</OPCODE> <DATA> <TXID>1007</TXID> </DATA> </RECORD> <RECORD> <OPCODE>OP_END_LOG_SEGMENT</OPCODE> <DATA> <TXID>1008</TXID> </DATA> </RECORD> </EDITS> [yinzhengjie@s101 ~]$
3>.
八.hadoop命令介绍
在上面我们以及提到过，"hadoop fs"其实就等价于“hdfs dfs”,但是hadoop有些命令是hdfs 命令所不支持的，我们举几个例子：
1>.检查压缩库本地安装情况
[yinzhengjie@s101 ~]$ hadoop checknative 18/05/27 04:40:13 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version 18/05/27 04:40:13 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /soft/hadoop-2.7.3/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: false lz4: true revision:99 bzip2: false openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)! [yinzhengjie@s101 ~]$
2>.格式化名称节点
[root@yinzhengjie ~]# hadoop namenode -format DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. 18/05/27 17:24:29 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = yinzhengjie/211.98.71.195 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.3 STARTUP_MSG: classpath = /soft/hadoop-2.7.3/etc/hadoop:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jets3t-0.9.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpclient-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/httpcore-4.2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-configuration-1.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-digester-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/gson-2.2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-framework-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-client-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsch-0.1.42.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/mockito-all-1.8.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-httpclient-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-net-3.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jsp-api-2.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/common/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/common/hadoop-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/soft/hadoop-2.7.3/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-lang-2.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guava-11.0.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-cli-1.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/activation-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/servlet-api-2.5.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-codec-1.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-client-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-json-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jettison-1.1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/lib/jetty-6.1.26.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/xz-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/asm-3.2.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/javax.inject-1.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/junit-4.11.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/soft/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by 'root' on 2016-08-18T01:41Z STARTUP_MSG: java = 1.8.0_131 ************************************************************/ 18/05/27 17:24:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 18/05/27 17:24:29 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-36f63542-a60c-46d0-8df1-f8fa32730764 18/05/27 17:24:30 INFO namenode.FSNamesystem: No KeyProvider found. 18/05/27 17:24:30 INFO namenode.FSNamesystem: fsLock is fair:true 18/05/27 17:24:30 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 18/05/27 17:24:30 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 18/05/27 17:24:30 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 18/05/27 17:24:30 INFO blockmanagement.BlockManager: The block deletion will start around 2018 May 27 17:24:30 18/05/27 17:24:30 INFO util.GSet: Computing capacity for map BlocksMap 18/05/27 17:24:30 INFO util.GSet: VM type = 64-bit 18/05/27 17:24:30 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 18/05/27 17:24:30 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/05/27 17:24:30 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 18/05/27 17:24:30 INFO blockmanagement.BlockManager: defaultReplication = 1 18/05/27 17:24:30 INFO blockmanagement.BlockManager: maxReplication = 512 18/05/27 17:24:30 INFO blockmanagement.BlockManager: minReplication = 1 18/05/27 17:24:30 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 18/05/27 17:24:30 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 18/05/27 17:24:30 INFO blockmanagement.BlockManager: encryptDataTransfer = false 18/05/27 17:24:30 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 18/05/27 17:24:30 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) 18/05/27 17:24:30 INFO namenode.FSNamesystem: supergroup = supergroup 18/05/27 17:24:30 INFO namenode.FSNamesystem: isPermissionEnabled = true 18/05/27 17:24:30 INFO namenode.FSNamesystem: HA Enabled: false 18/05/27 17:24:30 INFO namenode.FSNamesystem: Append Enabled: true 18/05/27 17:24:30 INFO util.GSet: Computing capacity for map INodeMap 18/05/27 17:24:30 INFO util.GSet: VM type = 64-bit 18/05/27 17:24:30 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 18/05/27 17:24:30 INFO util.GSet: capacity = 2^20 = 1048576 entries 18/05/27 17:24:30 INFO namenode.FSDirectory: ACLs enabled? false 18/05/27 17:24:30 INFO namenode.FSDirectory: XAttrs enabled? true 18/05/27 17:24:30 INFO namenode.FSDirectory: Maximum size of an xattr: 16384 18/05/27 17:24:30 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/05/27 17:24:30 INFO util.GSet: Computing capacity for map cachedBlocks 18/05/27 17:24:30 INFO util.GSet: VM type = 64-bit 18/05/27 17:24:30 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 18/05/27 17:24:30 INFO util.GSet: capacity = 2^18 = 262144 entries 18/05/27 17:24:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 18/05/27 17:24:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 18/05/27 17:24:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 18/05/27 17:24:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 18/05/27 17:24:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/05/27 17:24:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/05/27 17:24:30 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/05/27 17:24:30 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/05/27 17:24:30 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/05/27 17:24:30 INFO util.GSet: VM type = 64-bit 18/05/27 17:24:30 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 18/05/27 17:24:30 INFO util.GSet: capacity = 2^15 = 32768 entries 18/05/27 17:24:30 INFO namenode.FSImage: Allocated new BlockPoolId: BP-430965362-211.98.71.195-1527467070404 18/05/27 17:24:30 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted. 18/05/27 17:24:30 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/05/27 17:24:30 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 351 bytes saved in 0 seconds. 18/05/27 17:24:30 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/05/27 17:24:30 INFO util.ExitUtil: Exiting with status 0 18/05/27 17:24:30 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at yinzhengjie/211.98.71.195 ************************************************************/ [root@yinzhengjie ~]#
3>.执行自定义jar包
[yinzhengjie@s101 data]$ hadoop jar YinzhengjieMapReduce-1.0-SNAPSHOT.jar cn.org.yinzhengjie.mapreduce.wordcount.WordCountApp /world.txt /out 18/06/13 17:31:45 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 18/06/13 17:31:45 INFO input.FileInputFormat: Total input paths to process : 1 18/06/13 17:31:45 INFO mapreduce.JobSubmitter: number of splits:1 18/06/13 17:31:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1528935621892_0001 18/06/13 17:31:46 INFO impl.YarnClientImpl: Submitted application application_1528935621892_0001 18/06/13 17:31:46 INFO mapreduce.Job: The url to track the job: http://s101:8088/proxy/application_1528935621892_0001/ 18/06/13 17:31:46 INFO mapreduce.Job: Running job: job_1528935621892_0001 18/06/13 17:31:54 INFO mapreduce.Job: Job job_1528935621892_0001 running in uber mode : false 18/06/13 17:31:54 INFO mapreduce.Job: map 0% reduce 0% 18/06/13 17:32:00 INFO mapreduce.Job: map 100% reduce 0% 18/06/13 17:32:07 INFO mapreduce.Job: map 100% reduce 100% 18/06/13 17:32:08 INFO mapreduce.Job: Job job_1528935621892_0001 completed successfully 18/06/13 17:32:08 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=1081 FILE: Number of bytes written=244043 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=644 HDFS: Number of bytes written=613 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=3382 Total time spent by all reduces in occupied slots (ms)=4417 Total time spent by all map tasks (ms)=3382 Total time spent by all reduce tasks (ms)=4417 Total vcore-milliseconds taken by all map tasks=3382 Total vcore-milliseconds taken by all reduce tasks=4417 Total megabyte-milliseconds taken by all map tasks=3463168 Total megabyte-milliseconds taken by all reduce tasks=4523008 Map-Reduce Framework Map input records=1 Map output records=87 Map output bytes=901 Map output materialized bytes=1081 Input split bytes=91 Combine input records=0 Combine output records=0 Reduce input groups=67 Reduce shuffle bytes=1081 Reduce input records=87 Reduce output records=67 Spilled Records=174 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=205 CPU time spent (ms)=1570 Physical memory (bytes) snapshot=363290624 Virtual memory (bytes) snapshot=4190236672 Total committed heap usage (bytes)=211574784 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=553 File Output Format Counters Bytes Written=613 [yinzhengjie@s101 data]$
关于“Hadoop fs”更多相关命令请参考我的笔记：https://www.cnblogs.com/yinzhengjie/p/9906360.html
本文来自博客园，作者：尹正杰，转载请注明原文链接：https://www.cnblogs.com/yinzhengjie/p/9074730.html，个人微信: "JasonYin2020"（添加时请备注来源及意图备注，有偿付费）
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
