|NO.Z.00035|——————————|BigDataEnd|——|Hadoop&PB级数仓.V02|——|PB数仓.v02|广告分析|事件日志采集&ODS层数据处理|

一、广告分析——事件日志采集

### --- 启动Flume Agent（适当的修改参数，128M滚动一次）
~~~     # 启动flume

[root@hadoop02 ~]# flume-ng agent --conf /opt/yanqi/servers/flume-1.9.0/conf \
--conf-file /data/yanqidw/conf/flume-log2hdfs4.conf \
-name a1 -Dflume.root.logger=INFO,console

### --- 生成数据（文件大小约640M，100W条事件日志）

[root@hadoop02 ~]# cd /data/yanqidw/jars/
[root@hadoop02 ~]# java -cp data-generator-1.1-SNAPSHOT-jar-with-dependencies.jar \
com.yanqi.ecommerce.AppEvent 1000000 2020-07-21 \
> /data/yanqidw/logs/event/events0802.log

### --- 数据采集完成后，检查HDFS结果

[root@hadoop02 ~]# hdfs dfs -ls /user/data/logs/event
drwxr-xr-x   - root supergroup          0 2021-10-03 20:11 /user/data/logs/event/dt=2020-07-21
drwxr-xr-x   - root supergroup          0 2021-10-03 20:11 /user/data/logs/event/dt=Unknown

二、ODS层加载数据

### --- ODS层建表和数据加载
~~~     # 语法：在ODS层建表

drop table if exists ods.ods_log_event;
CREATE EXTERNAL TABLE ods.ods_log_event(`str` string)
PARTITIONED BY (`dt` string)
STORED AS TEXTFILE
LOCATION '/user/data/logs/event';

~~~     # 操作实例

hive (default)> drop table if exists ods.ods_log_event;

hive (default)> CREATE EXTERNAL TABLE ods.ods_log_event(`str` string)
              > PARTITIONED BY (`dt` string)
              > STORED AS TEXTFILE
              > LOCATION '/user/data/logs/event';

### --- 创建ODS层向HDFS建立关联数据脚本

[root@hadoop02 ~]# vim /data/yanqidw/script/advertisement/ods_load_event_log.sh
#!/bin/bash

source /etc/profile

if [ -n "$1" ]
then
    do_date=$1
else
    do_date=`date -d "-1 day" +%F`
fi

sql="
alter table ods.ods_log_event add partition (dt='$do_date');
"

hive -e "$sql"

### --- 把HDFS层文件与ODS层建立关联
~~~     # 执行脚本

[root@hadoop02 ~]# sh /data/yanqidw/script/advertisement/ods_load_event_log.sh 2020-07-21

~~~     # 在ODS层下查看数据是否与hdfs文件建立关联

hive (default)> show partitions ods.ods_log_event;
partition
dt=2020-07-21                        

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---