presto + slider 提交计算至yarn

10.112.28.240 prestocli

10.183.225.158 perstoser

 

hive-site.xml

useUnicode=true&characterEncoding=UTF-8&autoReconnect=true 

 

kinit -k -t /etc/presto/presto.keytab presto

hadoop fs -mkdir -p /user/presto/.slider/package/PRESTO/
hadoop fs -put ./presto-yarn-package-1.3-SNAPSHOT-0.176.zip hdfs://ns/user/presto/.slider/package/PRESTO/

hadoop fs -mkdir -p /etc/presto/
hadoop fs -put /etc/presto/presto.keytab /etc/presto/

　　

 

启动协调者：

/usr/bin/nohup /xxx/soft/lesports-presto-server-0.176/bin/launcher run --node-config /etc/presto/01/node.properties --jvm-config /etc/presto/01/jvm.config --config /etc/presto/01/config.properties --log-levels-file /etc/presto/01/log.properties >/xxx/soft/lesports-presto-server-0.176/stdout.log 2>&1 &

　　

netstat -nlp | grep 20000

启动slider:

/usr/bin/nohup /letv/soft/xxx-slider-0.80.0-incubating/bin/startPresto.sh cloud_presto01 /etc/presto/01/slider/appConfig.json /etc/presto/01/slider/resources.json > /xxx/soft/lecloud-slider-0.80.0-incubating/stdout.log 2>&1 &

　　

container-executor.cfg有min.user.id限制1000，需要修改user id:

usermod -u 1001 presto
cat /etc/passwd

在hdfs上建文件夹：
1. kinit 获取相应票据。

kinit -k -t /var/kerberos/krb5kdc/keytab/hadoop3_root.keytab presto/hadoop3@JENKIN.COM

hdfs dfs -mkdir -p /user/presto/.slider/cluster/

　查看hdfs文件：

[root@hadoop3 ~]# hadoop fs -ls /user/presto/.slider/cluster/
Found 1 items
drwxr-xr-x   - presto supergroup          0 2017-09-15 15:25 /user/presto/.slider/cluster/cloud_presto01

　　

presto-cli查询：

1. 下载presto-cli

运行： ./presto-cli --server hadoop1:20000 --catalog hive --schema jenkindb

 

问题解决：

问题：failed: No worker nodes available　

解决：presto coordinate config.properties增加 node-scheduler.include-coordinator=true

问题：org.apache.hadoop.ipc.RemoteException: User: presto@JENKIN.COM is not allowed to impersonate root

解决：hive.properties增加 hive.hdfs.impersonation.enabled=false