《Hive编程指南》14.3 投影变换的实践出错原因分析
自己在学习14.3节投影变换执行SQL语句hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA, newB FROM a;时出现了这个错误
Ended Job = job_local1231989520_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
跟作者的输出不一样。
自己一开始时没有管这个错误，直接跳过这个问题，继续往下看了。
但在执行接下来的语句SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA FROM a;时又出现了这个错误
Ended Job = job_local1771018374_0006 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
没有明确的错误信息，自己先是另开了一个终端尝试了一下：
duxing@duxing-X550JK:~$ /bin/cut
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" | /bin/cut -f1
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
第一次没有找的出错原因。
又在终端尝试了一下，在执行这行语句的时候SELECT TRANSFORM(col1, col2) USING 'cut -f1' AS newA, newB FROM a;得到了预期的结果，同时我又执行ls /bin | grep "cut"并没有cut文件。
现在看来是因为Ubuntu中cut程序并没有放在/bin/目录下导致语句执行出错。
在执行which cut后得到验证，cut放在了/usr/bin/目录下。
相关命令执行记录
- Hive终端
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA, newB FROM a;
Ended Job = job_local1231989520_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA, newB FROM a;
Ended Job = job_local1383279613_0005 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA FROM a;
Ended Job = job_local1771018374_0006 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/cut -f1' AS newA FROM a;
Ended Job = job_local81582517_0007 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/sed s/4/10' AS newA, newB AS a;
NoViableAltException(37@[])
at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormat(HiveParser.java:34626)
at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectTrfmClause(HiveParser_SelectClauseParser.java:2021)
at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1216)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:51850)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:45661)
at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:45568)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:44584)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:44454)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1696)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1178)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:444)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1242)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1384)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:67 cannot recognize input near 'AS' 'a' '<EOF>' in serde specification
hive (default)> SELECT TRANSFORM(col1, col2) USING '/bin/sed s/4/10' AS newA, newB FROM a;
/bin/sed: -e expression #1, char 6: unterminated `s' command
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Ended Job = job_local1180910273_0008 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING 'cut -f1' AS newA, newB FROM a;
newA newB
4 NULL
3 NULL
hive (default)> SELECT TRANSFORM(col1, col2) USING 'cut -f1' AS newA FROM a;
newA
4
3
终端1：
duxing@duxing-X550JK:~$ /bin/cut
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" | /bin/cut -f1
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |sed s/4/10
sed: -e expression #1, char 6: unterminated `s' command
duxing@duxing-X550JK:~$ sed s/4/10
sed: -e expression #1, char 6: unterminated `s' command
终端2：
duxing@duxing-X550JK:~$ ls /bin | grep "cut"
duxing@duxing-X550JK:~$ ls /bin
bash bzmore dd fgrep kbd_mode ls nc ntfsfallocate ps sh systemd-inhibit uname zfgrep
bunzip2 cat df findmnt kill lsblk nc.openbsd ntfsfix pwd sh.distrib systemd-machine-id-setup uncompress zforce
busybox cgroups-mount dir fuser kmod lsmod netcat ntfsinfo rbash sleep systemd-notify unicode_start zgrep
bzcat cgroups-umount dmesg fusermount less mkdir netstat ntfsls readlink ss systemd-tmpfiles vdir zless
bzcmp chacl dnsdomainname getfacl lessecho mknod networkctl ntfsmove red static-sh systemd-tty-ask-password-agent vmmouse_detect zmore
bzdiff chgrp domainname grep lessfile mktemp nisdomainname ntfstruncate rm stty tailf wdctl znew
bzegrep chmod dumpkeys gunzip lesskey more ntfs-3g ntfswipe rmdir su tar which
bzexe chown echo gzexe lesspipe mount ntfs-3g.probe open rnano sync tempfile whiptail
bzfgrep chvt ed gzip ln mountpoint ntfs-3g.secaudit openvt run-parts systemctl touch ypdomainname
bzgrep cp efibootmgr hciconfig loadkeys mt ntfs-3g.usermap pidof sed systemd true zcat
bzip2 cpio egrep hostname login mt-gnu ntfscat ping setfacl systemd-ask-password udevadm zcmp
bzip2recover dash false ip loginctl mv ntfscluster ping6 setfont systemd-escape ulockmgr_server zdiff
bzless date fgconsole journalctl lowntfs-3g nano ntfscmp plymouth setupcon systemd-hwdb umount zegrep
duxing@duxing-X550JK:~$ which cut
/usr/bin/cut
duxing@duxing-X550JK:~$ /usr/bin/cut
/usr/bin/cut: you must specify a list of bytes, characters, or fields
Try '/usr/bin/cut --help' for more information.
补充：在写本文的时候发现其实在终端执行/bin/cut的时候已经可以得到出错原因了，bash: /bin/cut: No such file or directory已经提示了/bin/cut不存在，而执行/usr/bin/cut的提示是没有参数。
