hadoop 操作

官方文档：https://hadoop.apache.org/docs/r3.1.2/

hadoop fs -ls /   看根目录下的所有文件

hdfs dfs -put alice.txt 上传

hdfs dfs -get /hdfsPath /localPath  下载到本地

hdfs dfsadmin -report.

 

Edits + Fsimage过程在 hdfs-site.xml 中指定路径。

启动服务时可能会删除一些文件：sudo rm -r /usr/local/hadoop_store/hdfs/datanode/current

current文件夹下对应的文件：

 

1. Fsimage文件：HDFS文件系统元数据的一个永久性的检查点，其中包含HDFS文件系统的所有目录和文件idnode的序列化信息；
2. Fsimage.md5文件：是镜像文件的 md5 校验文件，这个校验文件是为了判断镜像文件是否被修改；
3. Edits文件：存放HDFS文件系统的所有更新操作，文件系统客户端执行的所有写操作首先会被记录到Edits文件中。
4. seen_txid文件：它代表的是 namenode 里面的 edits_* 文件的尾数，namenode 重启的时候，会按照 seen_txid 的数字， 循序从头跑 edits_0000001~ 到 seen_txid 的数字。
5. VERSION文件：记录了当前NameNode的一些信息。

显示HDFS块信息

hadoop@node-master:~$ hdfs fsck / -files
Connecting to namenode via http://node-master:9870/fsck?ugi=hadoop&files=1&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.56.2 for path / at Sun Oct 27 17:37:28 AEDT 2019
/ <dir>
/alice.txt 173595 bytes, replicated: replication=2, 1 block(s):  OK
/readme.txt 152 bytes, replicated: replication=2, 1 block(s):  OK

Status: HEALTHY
 Number of data-nodes:    3
 Number of racks:        1
 Total dirs:            1
 Total symlinks:        0

Replicated Blocks:
 Total size:    173747 B
 Total files:    2
 Total blocks (validated):    2 (avg. block size 86873 B)
 Minimally replicated blocks:    2 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:    0 (0.0 %)
 Mis-replicated blocks:        0 (0.0 %)
 Default replication factor:    2
 Average block replication:    2.0
 Missing blocks:        0
 Corrupt blocks:        0
 Missing replicas:        0 (0.0 %)

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:    0
 Total block groups (validated):    0
 Minimally erasure-coded block groups:    0
 Over-erasure-coded block groups:    0
 Under-erasure-coded block groups:    0
 Unsatisfactory placement block groups:    0
 Average block group size:    0.0
 Missing block groups:        0
 Corrupt block groups:        0
 Missing internal blocks:    0
FSCK ended at Sun Oct 27 17:37:28 AEDT 2019 in 3 milliseconds


The filesystem under path '/' is HEALTHY

 

hadoop@node-master:~$ hdfs fsck / -blocks
Connecting to namenode via http://node-master:9870/fsck?ugi=hadoop&blocks=1&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.56.2 for path / at Sun Oct 27 17:40:29 AEDT 2019

Status: HEALTHY
 Number of data-nodes:    3
 Number of racks:        1
 Total dirs:            1
 Total symlinks:        0

Replicated Blocks:
 Total size:    173747 B
 Total files:    2
 Total blocks (validated):    2 (avg. block size 86873 B)
 Minimally replicated blocks:    2 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:    0 (0.0 %)
 Mis-replicated blocks:        0 (0.0 %)
 Default replication factor:    2
 Average block replication:    2.0
 Missing blocks:        0
 Corrupt blocks:        0
 Missing replicas:        0 (0.0 %)

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:    0
 Total block groups (validated):    0
 Minimally erasure-coded block groups:    0
 Over-erasure-coded block groups:    0
 Under-erasure-coded block groups:    0
 Unsatisfactory placement block groups:    0
 Average block group size:    0.0
 Missing block groups:        0
 Corrupt block groups:        0
 Missing internal blocks:    0
FSCK ended at Sun Oct 27 17:40:29 AEDT 2019 in 4 milliseconds


The filesystem under path '/' is HEALTHY

hadoop@node-master:~$ hdfs fsck /readme.txt -files
Connecting to namenode via http://node-master:9870/fsck?ugi=hadoop&files=1&path=%2Freadme.txt
FSCK started by hadoop (auth:SIMPLE) from /192.168.56.2 for path /readme.txt at Sun Oct 27 17:43:58 AEDT 2019
/readme.txt 152 bytes, replicated: replication=2, 1 block(s):  OK

Status: HEALTHY
 Number of data-nodes:    3
 Number of racks:        1
 Total dirs:            0
 Total symlinks:        0

Replicated Blocks:
 Total size:    152 B
 Total files:    1
 Total blocks (validated):    1 (avg. block size 152 B)
 Minimally replicated blocks:    1 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:    0 (0.0 %)
 Mis-replicated blocks:        0 (0.0 %)
 Default replication factor:    2
 Average block replication:    2.0
 Missing blocks:        0
 Corrupt blocks:        0
 Missing replicas:        0 (0.0 %)

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:    0
 Total block groups (validated):    0
 Minimally erasure-coded block groups:    0
 Over-erasure-coded block groups:    0
 Under-erasure-coded block groups:    0
 Unsatisfactory placement block groups:    0
 Average block group size:    0.0
 Missing block groups:        0
 Corrupt block groups:        0
 Missing internal blocks:    0
FSCK ended at Sun Oct 27 17:43:58 AEDT 2019 in 1 milliseconds


The filesystem under path '/readme.txt' is HEALTHY