Hadoop 之Pig的安装的与配置之遇到的问题---待解决
1. 前提是hadoop集群已经配置完成并且可以正常启动;以下是我的配置方案:
首先配置vim /etc/hosts
192.168.1.64 xuegod64
192.168.1.65 xuegod65
192.168.1.63 xuegod63
(将配置好的文件拷贝到其他两台机器,我是在xuegod64上配置的,使用scp /etc/hosts xuegod63:/etc/进行拷贝,进行该步骤前提是已经配置好SSH免密码登录;关于SSH免密码登录在此就不再详说了)
2.准备安装包如下图
[hadoop@xuegod64 ~]$ ls
hadoop-2.4.1.tar.gz
pig-0.15.0.tar.gz
jdk-8u66-linux-x64.rpm
zookeeper-3.4.7.tar.gz(可以不用)
![image image](https://images2015.cnblogs.com/blog/804543/201703/804543-20170311093938420-68169077.png)
3.配置/etc/profile
[hadoop@xuegod64 ~]$ vim /etc/profile #前提是使用root用户将编辑此文件的权限赋予hadoop用户
export JAVA_HOME=/usr/java/jdk1.8.0_66/
export HADOOP_HOME=/home/hadoop/hadoop-2.4.1/
export HBASE_HOME=/home/hadoop/hbase-1.1.2/
export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.4.7/
export PIG_HOME=/home/hadoop/pig-0.15.0/
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOO
KEEPER_HOME/bin:$HBASE_HOME/bin:$PIG_HOME/bin:$PATH
4.检验pig是否配置成功
[hadoop@xuegod64 ~]$ pig -help
Apache Pig version 0.15.0 (r1682971)
compiled Jun 01 2015, 11:44:35
USAGE: Pig [options] [-] : Run interactively in grunt shell.
Pig [options] -e[xecute] cmd [cmd ...] : Run cmd(s).
Pig [options] [-f[ile]] file : Run cmds found in file.
。
。
。
。
5.Pig执行模式
Pig有两种执行模式,分别为:
1)本地模式(Local)本地模式下,Pig运行在单一的JVM中,可访问本地文件。该模式适用于处理小规模数据或学习之用。
运行以下命名设置为本地模式:
pig –x local
2)MapReduce模式在MapReduce模式下,Pig将查询转换为MapReduce作业提交给Hadoop(可以说群集 ,也可以说伪分布式)。应该检查当前Pig版本是否支持你当前所用的Hadoop版本。某一版本的Pig仅支持特定版本的Hadoop,你可以通过访问Pig官网获取版本支持信息。
Pig会用到HADOOP_HOME环境变量。如果该变量没有设置,Pig也可以利用自带的Hadoop库,但是这样就无法保证其自带肯定库和你实际使用的HADOOP版本是否兼容,所以建议显式设置HADOOP_HOME变量。且还需要设置如下变量:
export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop
Pig默认模式是mapreduce,你也可以用以下命令进行设置:
[hadoop@xuegod64 ~]$ pig -x mapreduce
(中间略)
grunt>
下一步,需要告诉Pig它所用Hadoop集群的Namenode和Jobtracker。一般情况下,正确安装配置Hadoop后,这些配置信息就已经可用了,不需要做额外的配置
6.运行Pig程序
Pig程序执行方式有三种
1) 脚本方式
直接运行包含Pig脚本的文件,比如以下命令将运行本地scripts.pig文件中的所有命令:
pig scripts.pig
2)Grunt方式
a) Grunt提供了交互式运行环境,可以在命令行编辑执行命令
b) Grund同时支持命令的历史记录,通过上下方向键访问。
c) Grund支持命令的自动补全功能。比如当你输入a = foreach b g时,按下Tab键,则命令行自动变成a = foreach b generate。你甚至可以自定义命令自动补全功能的详细方式。具体请参阅相关文档。
3) 嵌入式方式
可以在java中运行Pig程序,类似于使用JDBC运行SQL程序
(不熟悉)
6.启动集群
[hadoop@xuegod64 ~]$ start-all.sh
[hadoop@xuegod64 ~]$ jps
4722 DataNode
5062 DFSZKFailoverController
5159 ResourceManager
4905 JournalNode
5321 Jps
4618 NameNode
2428 QuorumPeerMain
5279 NodeManager
[hadoop@xuegod64 ~]$ ssh xuegod63
Last login: Sat Jan 2 23:10:21 2016 from xuegod64
[hadoop@xuegod63 ~]$ jps
2130 QuorumPeerMain
3125 Jps
2982 NodeManager
2886 JournalNode
2795 DataNode
[hadoop@xuegod64 ~]$ ssh xuegod65
Last login: Sat Jan 2 15:11:33 2016 from xuegod64
[hadoop@xuegod65 ~]$ jps
3729 Jps
2401 QuorumPeerMain
3415 JournalNode
3484 DFSZKFailoverController
3325 DataNode
3583 NodeManager
3590 SecondNameNode
7.简单示例
我们以查找最高气温为例,演示如何利用Pig统计每年的最高气温。假设数据文件内容如下(每行一个记录,tab分割)
以local模式进入pig,依次输入以下命令(注意以分号结束语句):
[hadoop@xuegod64 ~]$ pig -x local
grunt> records = load'/home/hadoop/zuigaoqiwen.txt'as(year:chararray,temperature:int);
2016-01-02 16:12:05,700 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2016-01-02 16:12:05,701 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
grunt> dump records;
(1930:28:1,)
(1930:0:1,)
(1930:22:1,)
(1930:22:1,)
(1930:22:1,)
(1930:22:1,)
(1930:28:1,)
(1930:0:1,)
(1930:0:1,)
(1930:0:1,)
(1930:11:1,)
(1930:0:1,)
。
。
。
(过程略)
grunt> describe records;
records: {year: chararray,temperature: int}
grunt> valid_records = filter records by temperature!=999;
grunt> grouped_records = group valid_records by year;
grunt> dump grouped_records;
grunt> describe grouped_records;
grouped_records: {group: chararray,valid_records: {(year: chararray,temperature: int)}}
grunt> grouped_records = group valid_records by year;
grunt> dump grouped_records;
.
.
2016-01-02 16:16:02,974 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 7347344
Input split[0]:
Length = 7347344
ClassName: org.apache.hadoop.mapreduce.lib.input.FileSplit
Locations:
2016-01-02 16:16:08,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-01-02 16:16:08,012 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAFeatures
2.4.1 0.15.0 hadoop 2016-01-02 16:16:02 2016-01-02 16:16:08 GROUP_BY,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTimMedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs
job_local798558500_0002 1 1 n/a n/a n/a n/a n/a n/a n/a n/a grouped_records,records,valid_records GROUP_BY file:/tmp/temp-206603117/tmp-1002834084,
Input(s):
Successfully read 642291 records from: "/home/hadoop/zuigaoqiwen.txt"
Output(s):
Successfully stored 0 records in: "file:/tmp/temp-206603117/tmp-1002834084"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_local798558500_0002
grunt> describe grouped_records;
grouped_records: {group: chararray,valid_records: {(year: chararray,temperature: int)}}
grunt> max_temperature = foreach grouped_records generate group,MAX(valid_records.temperature);
grunt> dump max_temperature;
(1990,23)
(1991,21)
(1992,30)
grunt> quit
2016-01-02 16:24:25,303 [main] INFO org.apache.pig.Main - Pig script completed in 14 minutes, 27 seconds and 123 milliseconds (867123 ms)
中间有些问题,搞不定:
错误提示:
2016-01-02 16:18:28,049 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-01-02 16:18:28,050 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-01-02 16:18:28,050 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-01-02 16:18:28,055 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 642291 time(s).
2016-01-02 16:18:28,055 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2016-01-02 16:18:28,055 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2016-01-02 16:18:28,056 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-01-02 16:18:28,056 [main] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
2016-01-02 16:18:28,246 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-01-02 16:18:28,246 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
错误日志:
at java.lang.reflect.Method.invoke(Method.java:497)
Pig Stack Trace
at org.apache.pig.tools.grunt.GruntParser.processPig(Grunt
Parser.java:1082)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.p
arse(PigScriptParser.java:505)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError
(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError
(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:565)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Met
hod)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMetho
dAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegat
ingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
==============