presto集成iceberg
一、Presto服务下新建catelog
cd /usr/local/service/presto/etc/catalog
vim iceberg.properties
connector.name=iceberg
#iceberg.catalog.type=hive_metastore
hive.metastore.uri=thrift://127.0.0.1:7004,thrift://......
hive.config.resources=/usr/local/service/hadoop/etc/hadoop/core-site.xml,/usr/local/service/hadoop/etc/hadoop/hdfs-site.xml
hive.hdfs.authentication.type=KERBEROS
hive.hdfs.impersonation.enabled=true
hive.hdfs.presto.keytab=/var/krb5kdc/emr.keytab
hive.hdfs.presto.principal=hadoop/_HOST@EMR....
hive.metastore.authentication.type=KERBEROS
hive.metastore.client.keytab=/var/krb5kdc/emr.keytab
hive.metastore.client.principal=hadoop/_HOST@EMR....
hive.metastore.service.principal=hadoop/_HOST@EMR....
hive.recursive-directories=true
二、复制jar包到presto的iceberg catalog下
可以从hive目录下拷贝,没有的话需要去下载(用了腾讯chdfs没用这些技术的话可以少拷贝)
cp goosefs-1.1.0-SNAPSHOT-client.jar ../iceberg/
cp cosn-ranger-interface-1.0.4.jar ../iceberg/
cp hadoop-ranger-client-for-presto-2.8.5-5.0.jar ../iceberg/
cp hadoop-temrfs-2.8.5-8.1.4-1.0.6.jar ../iceberg/
cp chdfs_hadoop_plugin_network-* ../iceberg/
cp gson-2.2.4.jar ../iceberg/
三、重启presto的所有节点(测试时可以先用PrestoSQL-Coordinator)
三、测试
/usr/local/service/presto/presto-client/presto --server 127.0.0.1:9000 --catalog iceberg --schema data_lake_ods --user hadoop