Hive手写SQL案例

1-请详细描述将一个有结构的文本文件student.txt导入到一个hive表中的步骤，及其关键字

• 假设student.txt 有以下几列：id,name,gender三列
• 1-创建数据库 create database student_info;
• 2-创建hive表 student

create external table student_info.student(
id string comment '学生id',
name string comment '学生姓名',
gender string comment '学生性别'
) comment "学生信息表"
row format delimited fields terminated by '\t'
line terminated by '\n'
stored as textfile
location "/user/root/student";

• 3-加载数据

load data local inpath '/root/student.txt' into table student_info.student  location "/user/root/student" ;

• 4- 进入hive-cli，查看相应的表结构
  select * from student_info.student limit 10；

划重点：要回手写代码

2-利用HQL实现以下功能

2-1-创建表

• 创建员工基本信息表(EmployeeInfo)，字段包括(员工 ID，员工姓名，员工身份证号，性别，年龄，所属部门，岗位，入职公司时间，离职公司时间)，分区字段为入职公司时间，其行分隔符为”\n “，字段分隔符为”\t “。其中所属部门包括行政部、财务部、研发部、教学部，其对应岗位包括行政经理、行政专员、财务经理、财务专员、研发工程师、测试工程师、实施工程师、讲师、助教、班主任等，时间类型值如：2018-05-10 11:00:00

• 创建员工收入表(IncomeInfo)，字段包括(员工 ID，员工姓名，收入金额，收入所属
  月份，收入类型，收入薪水的时间)，分区字段为发放薪水的时间，其中收入类型包括薪资、奖金、公司福利、罚款四种情况 ; 时间类型值如：2018-05-10 11:00:00。

注意：时间类型是2018-05-10 11:00:00，需要对字段进行处理

• 创建员工基本信息表

create external table test.employee_info(
id string comment '员工id',
name string comment '员工姓名',
indentity_card string comment '身份证号',
gender string comment '性别',
department string comment '所属部门',
post string comment '岗位',
hire_date string comment '入职时间',
departure_date string comment '离职时间'
) comment "员工基本信息表"
partitioned by (day string comment "员工入职时间")
row format delimited fields terminated by '\t'
lines terminated by  '\n'
stored as textfile 
location '/user/root/employee';

• 创建员工收入表

create external table test.income_info(
id string comment '员工id',
name string comment '员工姓名',
income_data string comment '收入',
income_month string comment '收入所属月份',
income_type string comment '收入类型',
income_datetime string comment '收入薪水时间'
) comment '员工收入表'
partitioned by (day string comment "员工发放薪水时间")
row format delimited fields terminated by '\t'
lines terminated by  '\n'
stored as textfile 
location '/user/root/income';

2-2用 HQL 实现，求公司每年的员工费用总支出各是多少，并按年份降序排列?

• 重点对时间类型 2018-05-10 11:00:00 进行内置函数处理
• 需要读取income_info全量表，按照分区时间进行聚合，因为收入类型里面有罚款一项，所以需要在员工发放的钱中扣除罚款的钱。
• 不采用join、对数据一次遍历输出结果，
• 对于大数据量的情况下，要考虑对数据进行一次遍历求出结果

select 
    income_year,(income_data-(nvl(penalty_data,0))) as company_cost
from
(
    -- 统计员工收入金额和罚款金额，输出 2019 500 10
    select 
        income_year,
        sum(case when income_type!='罚款' then data_total else 0 end) as income_data,
        sum(case when income_type='罚款' then data_total else 0 end) as penalty_data
    from
    (
    -- 按照年份、收入类型求收入金额
    select 
        year(to_date(income_datetime)) as income_year,
        income_type,
        sum(income_data) as data_total
    from
        test.income_info
    group by 
        year(to_date(income_datetime)) ,income_type
    ) tmp_a
    group by  tmp_a.income_year
) as  temp
order by income_year desc;

2-3用 HQL 实现，求各部门每年的员工费用总支出各是多少，并按年份降序，按部门的支出升序排列？

• 保证对数据的一次遍历

--根据id关联得出department,和消费类型
select 
    income_year,department,
    (sum(case when income_type!='罚款' then income_data else 0 end) - sum(case when income_type='罚款' then income_data else 0 end) ) as department_cost
from
(
    -- 先对员工进行薪资类别的聚合统计
    select 
        id,year(to_date(income_datetime)) as income_year,income_type,sum(income_data) as income_data
    from 
        test.income_info
    group by 
    year(to_date(income_datetime)),id,income_type
) temp_a
inner join
    test.employee_info b
on
    temp_a.id=b.id
group by
    department,income_year
order by income_year desc , department_cost asc;

2-4用 HQL 实现，求各部门历史所有员工费用总支出各是多少，按总支出多少排名降序，遇到值相等情况，不留空位。

• 根据2-3中的中间结果进行修改
• 注意历史上所有的数据

select department,department_cost,dense_rank() over(order by department_cost desc) as cost_rank
from
(
--根据id关联得出department,和消费类型
select 
    department,
    (sum(case when income_type!='罚款' then income_data else 0 end) - sum(case when income_type='罚款' then income_data else 0 end) ) as department_cost
from
(
    -- 先对员工进行薪资类别的聚合统计
    select 
        id,income_type,sum(income_data) as income_data
    from 
        test.income_info
    group by 
    id,income_type
) temp_a
inner join
    test.employee_info b
on
    temp_a.id=b.id
group by
    department
) tmp_c ;

2-5 用 HQL 实现，创建并生成员工薪资收入动态变化表，即员工 ID，员工姓名，员工本月薪资，本月薪资发放时间，员工上月薪资，上月薪资发放时间。分区字段为本月薪资发放时间。

• 感觉应该使用动态分区插入的特性？-但是不知道该怎么写
• 先创建表，再采用insert into table **** select ***
• 要考虑到离职和入职的员工，这一点需要考虑到，full join
• 两张表进行full join，过滤day is null
• 需要concat year month to_date内置函数处理
• 这个题需要考虑的比较多

create external table test.income_dynamic(
id string comment '员工id',
name string comment '员工姓名',
income_data_current string comment '本月收入',
income_datetime_current string comment 本月'收入薪水时间',
income_data_last   string comment '上月收入',
income_datetime_last string comment '上月收入薪水时间',
) comment '员工收入动态表'
partitioned by (day string comment "员工本月发放薪水时间")
row format delimited fields terminated by '\t'
lines terminated by  '\n'
stored as textfile 
location '/user/root/income';
-- ------------------------------------------------------------------------------
-- 动态分区插入
-- 插入语句
-- 采用full join
insert into table test.income_dynamic partition(day)
select 
    (case when id_a is not null then id_a else id_b end ) as id,
    (case when name_a is not null then name_a else name_b end )  as name ,
    income_data,income_datetime,income_data_b,income_datetime_b,day
from
    (
    -- 选出表中所有的数据
    select
        id as id_a,name as name_a,income_data,income_datetime,day,concat(year(to_date(day)),month(to_date(day))) as day_flag
    from 
        test.income_info
    where 
        income_type='薪资' ) tmp_a
full outer join
    (
    -- 将表中的收到薪水的日期整体加一个月
    select
        id as id_b,name as name_b,income_data as income_data_b,income_datetime as  income_datetime_b,concat(year(add_months(to_date(day),1)),month(add_months(to_date(day),1))) as   month_flag
    from 
        test.income_info
    where 
        income_type='薪资'
    ) tmp_b
    on 
        tmp_a.day_flag=tmp_b.month_flag
    and 
        tmp_a.id_a=tmp_b.id_b
where day is not null
;

2-6 用 HQL 实现，薪资涨幅方面，2018 年 5 月份谁的工资涨的最多，谁的涨幅最大？

• 再2-5的基础上做比较简单，仅仅利用select部分即可；或者是再2-5的基础上做就行

Hive行列转换

１、问题
hive如何将
a       b       1
a       b       2
a       b       3
c       d       4
c       d       5
c       d       6
变为：
a       b       1,2,3
c       d       4,5,6
-------------------------------------------------------------------------------------------
２、数据
test.txt
a       b       1 
a       b       2 
a       b       3 
c       d       4 
c       d       5 
c       d       6
------------------------------------------------------------------------------------------- 
３、答案
1.建表
drop table tmp_jiangzl_test;
create table tmp_jiangzl_test
(
col1 string,
col2 string,
col3 string
)
row format delimited fields terminated by '\t'
stored as textfile;
-- 加载数据
load data local inpath '/home/jiangzl/shell/test.txt' into table tmp_jiangzl_test;
2.处理
select col1,col2,concat_ws(',',collect_set(col3)) 
from tmp_jiangzl_test  
group by col1,col2;
---------------------------------------------------------------------------------------
collect_set/concat_ws语法参考链接：https://blog.csdn.net/waiwai3/article/details/79071544
https://blog.csdn.net/yeweiouyang/article/details/41286469   [Hive]用concat_w实现将多行记录合并成一行
---------------------------------------------------------------------------------------
二、列转行
１、问题
hive如何将
a       b       1,2,3
c       d       4,5,6
变为：
a       b       1
a       b       2
a       b       3
c       d       4
c       d       5
c       d       6
---------------------------------------------------------------------------------------------
2、答案
1.建表

drop table tmp_jiangzl_test;
create table tmp_jiangzl_test
(
col1 string,
col2 string,
col3 string
)
row format delimited fields terminated by '\t'
stored as textfile;
处理：
select col1, col2, col5
from tmp_jiangzl_test a 
lateral  view explode(split(col3,',')) b AS col5;
---------------------------------------------------------------------------------------
lateral  view 语法参考链接：
https://blog.csdn.net/clerk0324/article/details/58600284

Hive实现wordcount

1.创建数据库
create database wordcount;
2.创建外部表
create external table word_data(line string) row format delimited fields terminated by ',' location '/home/hadoop/worddata';
3.映射数据表
load data inpath '/home/hadoop/worddata' into table word_data;
4.这里假设我们的数据存放在hadoop下，路径为：/home/hadoop/worddata，里面主要是一些单词文件，内容大概为：
hello man
what are you doing now
my running
hello
kevin
hi man
执行了上述hql就会创建一张表src_wordcount，内容是这些文件的每行数据，每行数据存在字段line中，select * from word_data;就可以看到这些数据

5.根据MapReduce的规则，我们需要进行拆分，把每行数据拆分成单词，这里需要用到一个hive的内置表生成函数（UDTF）：explode(array)，参数是array，其实就是行变多列：

create table words(word string);
insert into table words select explode(split(line, " ")) as word from word_data;

6.查看words表内容
OK
hello
man
what
are
you
doing
now
my
running
hello
kevin
hi
man
split是拆分函数，跟java的split功能一样，这里是按照空格拆分，所以执行完hql语句，words表里面就全部保存的单个单词
7.group by统计单词
    select word, count(*) from wordcount.words group by word;
wordcount.words 库名称.表名称，group by word这个word是create table words(word string) 命令创建的word string

结果：
are     1
doing   1
hello   2
hi      1
kevin   1
man     2
my      1
now     1
running 1
what    1
you     1

Hive取TopN

• rank() over()
• dense_rank() over()
• row_number() over()

求取指定状态下的订单id

• 给一张订单表，统计只购买过面粉的用户；（重点在于仅仅购买过面粉的客户）
  eg：order:order_id,buyer_id,order_time.....

在保证一次遍历的情况下,重点是O(1)复杂度

select buyer_id
from
(
select buyer_id,sum(case when order_id='面粉' then 0 else 1 end) as flag
from order
) as tmp
where flag=0;

微博体系中互粉的有多少组

• 在微博粉丝表中，互相关注的人有多少组，例如：A-->B;B-->A；A和B互粉，称为一组。
  表结构：id,keep_id,time.... (id,keep_id可作为联合主键)
• 借助Hive进行实现

select count(*)/2 as weibo_relation_number
from
(
  (select concat(id,keep_id) as flag from weibo_relation)
  union all  --全部合并到一起，不能提前去重
  (select concat(keep_id,id) as flag from weibo_relation)
) as tmp
having count(flag) =2
group by flag;

购买了香蕉的人买了多少东西

• 这个是一个很经典的问题，购买了香蕉的人买了多少东西
• 数据还是延用上一个问题的数据和表结构，即理解为关注C的人总共关注了多少人
• 仔细理解是需要对关注的人进行去重统计

select count(distinct keep_id) as total_keep_id
from weibo_relation
where id
  in
(select id from weibo_relation where keep_id='c')