Impala 加载Hive的UDF

Impala的UDF有两种:

  Native Imapal UDF:使用C++开发的,性能极高,官方性能测试比第二种高出将近10倍

  Hive的UDF:是Hive中的UDF,直接加载到Impala中,优点是不需要任何改动,完全跟Hive中用法相同

第一种方式请参考我转载的文章【转】Impala安装json解析udf插件

本文介绍第二种方式,在Impala中直接加载Hive的UDF

如在Hive中有一个UDF为get_json_object,用于解析Json,但是Imapla中没有类似的函数。

1. 该function所在的jar包是/usr/lib/hive/lib/hive-exec-1.1.0-cdh5.13.0.jar,

[cloudera@quickstart lib]$ jar tf hive-exec-1.1.0-cdh5.13.0.jar|grep UDFJson
org/apache/hadoop/hive/ql/udf/UDFJson$AddingList.class
org/apache/hadoop/hive/ql/udf/UDFJson.class
org/apache/hadoop/hive/ql/udf/UDFJson$HashCache.class
org/apache/hadoop/hive/ql/udf/UDFJson$1.class

2.把jar包上传到hdfs的目录中,如下:

 hdfs dfs -put /usr/lib/hive/lib/hive-exec-1.1.0-cdh5.13.0.jar /user/cloudera/lib/hive-udf.jar

3.在Impala Shell中创建function,其中Symbol指向类名称:https://github.com/apache/hive/blob/branch-0.14/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFJson.java

create function if not exists get_json_object(String,String) returns String location "/user/cloudera/lib/hive-udf.jar" SYMBOL="org.apache.hadoop.hive.ql.udf.UDFJson"; 
[quickstart.cloudera:21000] > show functions;
Query: show functions
+-------------+---------------------------------+-------------+---------------+
| return type | signature                       | binary type | is persistent |
+-------------+---------------------------------+-------------+---------------+
| STRING      | get_json_object(STRING, STRING) | JAVA        | false         |
+-------------+---------------------------------+-------------+---------------+

  

4.在Impala shell中使用

[quickstart.cloudera:21000] > select get_json_object(test1.content,'$.userId') from test1;
Query: select get_json_object(test1.content,'$.userId') from test1
Query submitted at: 2018-06-28 04:19:44 (Coordinator: http://quickstart.cloudera:25000)
Query progress can be monitored at: http://quickstart.cloudera:25000/query_plan?query_id=4241f9deab0498e2:ab9c00fd00000000
+--------------------------------------------------------------------+
| get_json_object(report_data.content, '$.userid') |
+--------------------------------------------------------------------+
| 16                                                                 |
| 15                                                                 |
| 8                                                                  |
+--------------------------------------------------------------------+ 

验证可以使用

posted @ 2018-06-28 20:15  骑着龙的羊  阅读(3955)  评论(0编辑  收藏  举报