pig入门案例

测试数据位于：/home/hadoop/luogankun/workspace/sync_data/pig
person.txt中的数据以逗号分隔

1,zhangsan,112
2,lisi,113
3,wangwu,114
4,zhaoliu,115

 

score.txt中的数据以制表符分隔

1       20
2       30
3       40
5       50

 

pig只能针对HDFS上的文件进行操作，所以需要将文件先上传到HDFS中

cd /home/hadoop/luogankun/workspace/sync_data/pig
hadoop fs -put person.txt input/pig/person.txt
hadoop fs -put score.txt input/pig/score.txt

 

 

load文件(HDFS系统上的)

a = load 'input/pig/person.txt' using PigStorage(',') as (id:int, name:chararray, age:int);
b = load 'input/pig/score.txt' using PigStorage('\t') as (id:int, score:int);

 

 

查看表结构

describe a
a: {id: int,name: chararray,age: int}

describe b
b: {id: int,score: int}

 

查看表数据

dump a
(1,zhangsan,112)
(2,lisi,113)
(3,wangwu,114)
(4,zhaoliu,115)

dump b
(1,20)
(2,30)
(3,40)
(5,50)

dump 会跑mapreduce任务。

 

条件过滤

查询person中id小于4的人

aa = filter a by id < 4;

dump aa;
(1,zhangsan,112)
(2,lisi,113)
(3,wangwu,114)

pig中等号使用==， 例如：aa = filter a by id == 4;

 

表关联

c = join a by id left , b by id;

describe c
c: {a::id: int,a::name: chararray,a::age: int,b::id: int,b::score: int}
#表名字段名之间两个冒号，字段与字段类型之间一个冒号

dump c
(1,zhangsan,112,1,20)
(2,lisi,113,2,30)
(3,wangwu,114,3,40)
(4,zhaoliu,115,,)

由于采用的是left join，所以只有四条数据，而且第四条数据是没有分数的。

 

迭代数据

d =foreach c generate a::id as id, a::name as name, b::score as score, a::age as age;

describe d;
d: {id: int,name: chararray,score: int,age: int}

dump d
(1,zhangsan,20,112)
(2,lisi,30,113)
(3,wangwu,40,114)
(4,zhaoliu,,115)

注意：foreach使用时只要等号前或者后有一个空格即可，如果等号两端都没有空格的话会报错。

 

处理结果存储到HDFS系统上

store d into 'output/pig/person_score' using PigStorage(',');   #导出到HDFS上的文件分隔符是逗号
hadoop fs -ls output/pig/person_score
hadoop fs -cat output/pig/person_score/part-r-00000
1,zhangsan,20,112
2,lisi,30,113
3,wangwu,40,114
4,zhaoliu,,115

hadoop fs -rmr output/pig/person_score
store d into 'output/pig/person_score';     #导出到HDFS上的文件分隔符是制表符
hadoop fs -ls output/pig/person_score
hadoop fs -cat output/pig/person_score/part-r-00000
1 zhangsan 20 112
2 lisi 30 113
3 wangwu 40 114
4 zhaoliu 115

 

 

pig执行文件

将上面的所有pig shell脚本放到一个sh脚本中执行
/home/hadoop/luogankun/workspace/shell/pig/person_score.pig

a = load 'input/pig/person.txt' using PigStorage(',') as (id:int, name:chararray, age:int);
b = load 'input/pig/score.txt' using PigStorage('\t') as (id:int, score:int);
c = join a by id left , b by id;
d =foreach c generate a::id as id, a::name as name, b::score as score, a::age as age;
store d into 'output/pig/person_score2' using PigStorage(',');   

 

执行person.score.pig脚本：

/home/hadoop/luogankun/workspace/shell/pig

pig person_score.pig

 

 

pig脚本传递参数

pig脚本位置：/home/hadoop/luogankun/workspace/shell/pig/mulit_params_demo01.pig

log = LOAD '$input' AS (user:chararray, time:long, query:chararray);
lmt = LIMIT log $size;
DUMP lmt;

 

上传数据到hdfs文件中

cd /home/hadoop/luogankun/workspace/shell/pig
hadoop fs -put excite-small.log input/pig/excite-small.log

传递方式一：逐个参数传递

pig -param input=input/pig/excite-small.log -param size=4 mulit_params_demo01.pig

传递方式二：将参数保存在txt文件中

/home/hadoop/luogankun/workspace/shell/pig/mulit_params.txt

input=input/pig/excite-small.log
size=5
pig -param_file mulit_params.txt mulit_params_demo01.pig