hdfs操作手册
hdfscli 命令行
# hdfscli --help
HdfsCLI: a command line interface for HDFS.
Usage:
hdfscli [interactive] [-a ALIAS] [-v...]
hdfscli download [-fsa ALIAS] [-v...] [-t THREADS] HDFS_PATH LOCAL_PATH
hdfscli upload [-sa ALIAS] [-v...] [-A | -f] [-t THREADS] LOCAL_PATH HDFS_PATH
hdfscli -L | -V | -h
Commands:
download Download a file or folder from HDFS. If a
single file is downloaded, - can be
specified as LOCAL_PATH to stream it to
standard out.
interactive Start the client and expose it via the python
interpreter (using iPython if available).
upload Upload a file or folder to HDFS. - can be
specified as LOCAL_PATH to read from standard
in.
Arguments:
HDFS_PATH Remote HDFS path.
LOCAL_PATH Path to local file or directory.
Options:
-A --append Append data to an existing file. Only supported
if uploading a single file or from standard in.
-L --log Show path to current log file and exit.
-V --version Show version and exit.
-a ALIAS --alias=ALIAS Alias of namenode to connect to.
-f --force Allow overwriting any existing files.
-s --silent Don't display progress status.
-t THREADS --threads=THREADS Number of threads to use for parallelization.
0 allocates a thread per file. [default: 0]
-v --verbose Enable log output. Can be specified up to three
times (increasing verbosity each time).
Examples:
hdfscli -a prod /user/foo
hdfscli download features.avro dat/
hdfscli download logs/1987-03-23 - >>logs
hdfscli upload -f - data/weights.tsv <weights.tsv
HdfsCLI exits with return status 1 if an error occurred and 0 otherwise.
要使用hdfscli,首先需要设置hdfscli的默认配置文件
# cat ~/.hdfscli.cfg [global] default.alias = dev [dev.alias] url = http://hadoop:50070 user = root
python可用的客户端类：
InsecureClient(default)
TokenClient
上传或下载文件
使用hdfscli上传文件或文件夹（将hadoop文件夹上传到/hdfs）
# hdfscli upload --alias=dev -f /hadoop-2.4.1/etc/hadoop/ /hdfs
使用hdfscli下载/logs目录到操作系统的/root/test目录下
# hdfscli download /logs /root/test/
hdfscli 交互模式
[root@hadoop ~]# hdfscli --alias=dev
Welcome to the interactive HDFS python shell.
The HDFS client is available as `CLIENT`.
>>> CLIENT.list("/")
[u'Demo', u'hdfs', u'logs', u'logss']
>>> CLIENT.status("/Demo")
{u'group': u'supergroup', u'permission': u'755', u'blockSize': 0,
u'accessTime': 0, u'pathSuffix': u'', u'modificationTime': 1495123035501L,
u'replication': 0, u'length': 0, u'childrenNum': 1, u'owner': u'root',
u'type': u'DIRECTORY', u'fileId': 16389}
>>> CLIENT.delete("logs/install.log")
False
>>> CLIENT.delete("/logs/install.log")
True
与python接口的绑定
初始化客户端
1、导入client类，然后调用它的构造函数
>>> from hdfs import InsecureClient
>>> client = InsecureClient("http://172.10.236.21:50070",user='ann')
>>> client.list("/")
[u'Demo', u'hdfs', u'logs', u'logss']
2、导入config类，加载一个已存在的配置文件并且从已存在的alias创建一个client,配置文件默认的读取文件为~/.hdfs_config.cfg
>>> from hdfs import Config
>>> client=Config().get_client("dev")
>>> client.list("/")
[u'Demo', u'hdfs', u'logs', u'logss']
读文件
read()方法可从hdfs系统读取一个文件，但是它必须放在with块中，以确保每次都能正确关闭连接
>>> with client.read("/logs/yarn-env.sh"，encoding="utf-8") as reader:
... features=reader.read()
...
>>> print features
chunk_size参数将返回一个生成器，它使文件的内容变成流数据
>>> with client.read("/logs/yarn-env.sh",chunk_size=1024) as reader:
... for chunk in reader:
... print chunk
...
delimiter参数同样返回一个生成器，文件内容是被指定符号分隔的
>>> with client.read("/logs/yarn-env.sh", encoding="utf-8", delimiter="\n") as reader:
... for line in reader:
... time.sleep(1)
... print line
写文件
write方法用于写文件到hdfs(将本地文件kong.txt写入hdfs的/logs/kongtest.txt文件中)
>>> with open("/root/test/kong.txt") as reader, client.write("/logs/kongtest.txt") as writer:
... for line in reader:
... if line.startswith("-"):
... writer.write(line)
