hive bug
- SHELL$ hive -S -e “set” | grep warehouse
- hive.metastore.warehouse.dir=/user/hive/warehouse
- hive.warehouse.subdir.inherit.perms=false
Status: Running (Executing on YARN cluster with App id application_1510723060011_0082)
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 1 1 0 0 0 0
Map 10 ......... SUCCEEDED 1 1 0 0 0 0
Map 2 .......... SUCCEEDED 1 1 0 0 0 0
Map 3 .......... SUCCEEDED 1 1 0 0 0 0
Map 4 .......... SUCCEEDED 1 1 0 0 0 0
Map 5 .......... SUCCEEDED 1 1 0 0 0 0
Map 6 .......... SUCCEEDED 1 1 0 0 0 0
Map 8 .......... SUCCEEDED 21 21 0 0 0 0
Map 9 .......... SUCCEEDED 2 2 0 0 0 0
Reducer 7 ...... SUCCEEDED 1 1 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 10/10 [==========================>>] 100% ELAPSED TIME: 13.72 s
--------------------------------------------------------------------------------
Loading data to table bridge.sods_to_cio_libor_hold partition (biz_date=20170628)
Failed with exception java.io.IOException: rename for src path: hdfs://mycluster/apps/hive/warehouse/bridge.db/sods_to_cio_libor_hold/biz_date=20170628/.hive-staging_hive_2017-11-16_10-06-19_248_4976600282215103377-1/-ext-10000/000008_0 to dest path:hdfs://mycluster/apps/hive/warehouse/bridge.db/sods_to_cio_libor_hold/biz_date=20170628/000008_0 returned false
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
om bridge got value #279
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: submitDAG took 370ms
17/11/18 16:55:48 [main]: INFO client.TezClient: Submitted dag to TezSession, sessionName=HIVE-f8697cdf-5cb6-4d3b-9ea6-db5bc976d944, applicationId=application_1510723060011_0609, dagName=Insert overwrite table bridge.sod...'X_CNBD'(Stage-1)
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG ssl.FileBasedKeyStoresFactory: CLIENT TrustStore: /etc/security/clientKeys/all.jks
17/11/18 16:55:48 [main]: DEBUG impl.TimelineClientImpl: Cannot load customized ssl related configuration. Fallback to system-generic settings.
java.io.FileNotFoundException: /etc/security/clientKeys/all.jks (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:168)
at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.<init>(ReloadingX509TrustManager.java:86)
at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:211)
at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.initSslConnConfigurator(TimelineClientImpl.java:510)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.initConnConfigurator(TimelineClientImpl.java:486)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:272)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.tez.client.TezYarnClient.init(TezYarnClient.java:46)
at org.apache.tez.dag.api.client.DAGClientImpl.<init>(DAGClientImpl.java:90)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:581)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:466)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:446)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:181)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: INFO impl.TimelineClientImpl: Timeline service address: http://cnsz131375:8188/ws/v1/timeline/
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.AHSClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:163)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: INFO client.AHSProxy: Connecting to Application History server at cnsz131375/10.25.167.173:10200
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.AHSProxy.getProxy(AHSProxy.java:49)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationHistoryProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.AHSClientImpl is started
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /seabox/var/lib/hadoop-hdfs/dn_socket
17/11/18 16:55:48 [main]: DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://mycluster
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /seabox/var/lib/hadoop-hdfs/dn_socket
17/11/18 16:55:48 [main]: DEBUG retry.RetryUtils: multipleLinearRandomRetry = MultipleLinearRandomRetry[500x2000ms]
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131375/10.25.167.173:8020
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge: starting, having connections 8
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #280
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #280
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 2ms
17/11/18 16:55:48 [main]: DEBUG impl.FileSystemTimelineWriter: yarn.timeline-service.client.fd-flush-interval-secs=10, yarn.timeline-service.client.fd-clean-interval-secs=60, yarn.timeline-service.client.fd-retain-secs=300, yarn.timeline-service.entity-file.fs-support-append=true, yarn.timeline-service.entity-group-fs-store.active-dir=/ats/active
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl is started
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl is started
17/11/18 16:55:48 [main]: INFO log.PerfLogger: </PERFLOG method=TezSubmitDag start=1510995348115 end=1510995348697 duration=582 from=org.apache.hadoop.hive.ql.exec.tez.TezTask>
17/11/18 16:55:48 [main]: INFO SessionState:
17/11/18 16:55:48 [main]: INFO log.PerfLogger: <PERFLOG method=TezRunDag from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:48 [main]: INFO log.PerfLogger: <PERFLOG method=TezSubmitToRunningDag from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131375/10.25.167.173:8032
17/11/18 16:55:48 [main]: WARN ipc.Client: Failed to connect to server: cnsz131375/10.25.167.173:8032: retries get failed due to exceeded maximum allowed retries number: 0
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: closing ipc connection to cnsz131375/10.25.167.173:8032: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8032 from bridge: closed
17/11/18 16:55:48 [main]: DEBUG retry.RetryInvocationHandler: Exception while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Trying to failover immediately.
java.net.ConnectException: Call From cnsz131372/10.25.167.170 to cnsz131375:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
... 42 more
17/11/18 16:55:48 [main]: INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:163)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge sending #282
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge got value #282
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms
17/11/18 16:55:48 [main]: DEBUG rpc.DAGClientRPCImpl: App: application_1510723060011_0609 in state: RUNNING
17/11/18 16:55:48 [main]: DEBUG client.TezClientUtils: Connecting to Tez AM at cnsz131372/10.25.167.170:43571
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.tez.client.TezClientUtils.getAMProxy(TezClientUtils.java:908)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131372/10.25.167.170:43571
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge: starting, having connections 9
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #283
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #283
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getDAGStatus took 165ms
17/11/18 16:55:48 [main]: INFO log.PerfLogger: </PERFLOG method=TezSubmitToRunningDag start=1510995348707 end=1510995348894 duration=187 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
Status: Running (Executing on YARN cluster with App id application_1510723060011_0609)
17/11/18 16:55:48 [main]: INFO SessionState: Status: Running (Executing on YARN cluster with App id application_1510723060011_0609)
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITIALIZING -1 0 0 -1 0 0
Map 2 INITIALIZING -1 0 0 -1 0 0
Map 3 INITIALIZING -1 0 0 -1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
Map 2 INITED 1 0 0 1 0 0
Map 3 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Map 2 INITED 1 0 0 1 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #284
Map 3 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0/1 Map 3: 0/1 Map 4: 0/1 Map 5: 0/1 Map 6: 0/1 Map 8: 0/21 Map 9: 0/2 Reducer
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
Map 2 INITED 1 0 0 1 0 0
Map 3 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Map 4 INITED 1 0 0 1 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #294
Map 5 INITED 1 0 0 1 0 0
Map 6 INITED 1 0 0 1 0 0
Map 8 INITED 21 0 0 21 0 0
Map 9 INITED 2 0 0 2 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Reducer 7 INITED 2 0 0 2 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #295
--------------------------------------------------------------------------------
VERTICES: 00/10 [>>--------------------------] 0% ELAPSED TIME: 2.44 s
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 1
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #298
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #298T handler]: DEBUG rpc.DAGClientRPCImpl: TryKill for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 19ms0421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #296
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 10z131372/10.25.167.170:43571 fr17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #299
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #299: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms90421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #297
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 2sz131372/10.25.167.170:43571 fr17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #300
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #300
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 3
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #301
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #301
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 4
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #302
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #302
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 5
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #303
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #303
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 6
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #304
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #304
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 8
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #305
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #305
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 9
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #306
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #306
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Reducer 7
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #307
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #307
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
Map 1 KILLED -1 0 0 -1 0 0
Map 10 KILLED 1 0 0 1 0 0
Map 2 KILLED 1 0 0 1 0 0
Map 3 KILLED 1 0 0 1 0 0
Map 4 KILLED 1 0 0 1 0 0
Map 5 KILLED 1 0 0 1 0 0
Map 6 KILLED 1 0 0 1 0 0
Map 8 KILLED 21 0 0 21 0 0
Map 9 KILLED 2 0 0 2 0 0
Reducer 7 KILLED 2 0 0 2 0 0
--------------------------------------------------------------------------------
VERTICES: 00/10 [>>--------------------------] 0% ELAPSED TIME: 2.69 s
--------------------------------------------------------------------------------
Status: Killed
17/11/18 16:55:51 [main]: INFO SessionState: Status: Killed
Job received Kill while in RUNNING state.
17/11/18 16:55:51 [main]: ERROR SessionState: Job received Kill while in RUNNING state.
Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]
DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: ERROR SessionState: DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=TezRunDag start=1510995348707 end=1510995351582 duration=2875 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #308
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #308
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getDAGStatus took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #309
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #309
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #310
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #310
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 0ms
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.AHSClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG impl.FileSystemTimelineWriter: Closing cache
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: INFO hooks.ATSHook: Created ATS Hook
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1510995351618 end=1510995351618 duration=0 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [ATS Logger 0]: INFO hooks.ATSHook: Received post-hook notification for :bridge_20171118165543_29ad0d2b-ae72-466b-9139-210b176ff364
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Job received Kill while in RUNNING state.Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Job received Kill while in RUNNING state.Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: DEBUG ql.Driver: Shutting down query Insert overwrite table bridge.sods_to_cio_libor_hold partition(biz_date=20170606)
SELECT 'CNY',
'3002' AS COMPANY_REPORT, --数据上报公司
'3002' AS COMPANY_MANAGER, --投资管理公司
'3002' AS COMPANY_OWNER, --投资主体公司
P1.MAPPING_KEY AS BUSINESS_LINE, --业务线
CASE
WHEN T1.TRDTYPE in ('42', '82', '84') THEN '01001002'
WHEN T1.TRDTYPE in ('43', '83', '85') THEN '01001003'
END AS DETAIL_ASSET_TYPE, --明细资产分类
CASE p2.mapping_key
WHEN '005' THEN '001'
ELSE '001'
END AS FUND_SOURCE, --资金来源, 把005修改成001后上报，001保持不变
P.PORTFOLIO_UNITS AS PORTFOLIO_UNITS, --投资组合
T1.I_CODE AS SECURITY_ID, --证券代码,给流水号
T1.I_CODE AS SECURITY_NAME, --证券名称,给流水号
nvl(T20.TARGET_CODE,'OTHER') AS MARKET_NO, --市场代码
nvl(T20.TARGET_NAME,'其他') AS MARKET_NAME, --市场名称
oc.partyname AS COUNTER_PARTY, --交易对手名称
T1.partyid AS COUNTER_PARTY_ID, --交易对手组织机构代码
T2.AMOUNT AS HOLDING_UNITS, --持仓数量PAR_VALUE
T2.AMOUNT AS BOOK_BALANCE_N, --财务账面余额_本币AMOUNT
T2.AMOUNT AS BOOK_BALANCE_B, --财务账面余额_本币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_N, --参考财务成本_原币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_B, --参考财务成本_本币AMOUNT
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE('2017-06-30'),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_N, --应计利息_原币
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE('2017-06-30'),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_B --应计利息_本币
FROM bridge.FBMS_A_TTRD_OTC_TRADE_IB T1
INNER JOIN bridge.FBMS_A_TTRD_OTC_IBLB_IB T2 ON T2.I_CODE = T1.INTORDID
AND T2.M_TYPE = T1.M_TYPE
AND T2.A_TYPE = T1.A_TYPE
AND T2.ENDSETDATE > TO_DATE('2017-06-30') --投资组合映射主表
left join (select v1.accountid, v2.murex_leaf, v2.Third_Business_Line as business_line, v2.portfolio_units, v2.fund_source, v2.account_class
from bridge.BRIDGE_APP_PORTFOLIOACCOUNT_ib v1
inner join bridge.bridge_murex_leaf_config v2 on v1.portfolioid = v2.murex_leaf
and ascii(v2.is_send_group) = -26
and v1.biz_date = v2.biz_date
where '20170630' = v2.biz_date) p on p.accountid = t1.secu_accid
--业务线
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'BUSINESS_LINE' and '20170630' = biz_date) p1 on p.business_line = p1.chinese_name
--资金来源
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'FUND_SOURCE' and '20170630' = biz_date) p2 on p.fund_source = p2.chinese_name
--会计分类
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'ACCOUNT_CLASS' and '20170630' = biz_date) p3 on p.account_class = p3.chinese_name
--交易对手(交易对手SUMMIT使用的是衡泰的数据源，衡泰是从交易中心获取的，可以通用)
left join (select max(partyname) as partyname, partyid from bridge.FBMS_A_TTRD_OTC_COUNTERPARTY_IB where '20170630' = biz_date group by partyid) oc on t1.partyid = oc.partyid
left join bridge.bridge_security_to_cio_mapping_ib T20
on T1.M_TYPE = T20.SOURCE_CODE
and T20.SOURCE_SYSTEM_CODE = 'FBMS'
and T20.MAPPING_TYPE ='MARKET_NO'
and T20.SOURCE_COLUMN_NAME = 'M_TYPE'
and T20.SOURCE_TABLE_NAME = 'FBMS_A_TTRD_OTC_TRADE_IB'
and T20.biz_date='20170630'
WHERE T1.ORDSOURCE NOT IN ('4', '11') --过滤掉投顾指令单
AND T1.ORDSTATUS = '7'
AND T1.A_TYPE in ('SPT_IBOR')
AND T1.M_TYPE = 'X_CNBD'
17/11/18 16:55:51 [ATS Logger 0]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)
17/11/18 16:55:51 [main]: INFO ql.Driver: Resetting the caller context to
17/11/18 16:55:51 [ATS Logger 0]: DEBUG impl.TimelineWriter: POST to http://cnsz131375:8188/ws/v1/timeline/
17/11/18 16:55:51 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
17/11/18 16:55:51 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=0}
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1510995347627 end=1510995351619 duration=3992 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1510995351619 end=1510995351619 duration=0 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: DEBUG ql.Driver: Shutting down query Insert overwrite table bridge.sods_to_cio_libor_hold partition(biz_date=20170606)
SELECT 'CNY',
'3002' AS COMPANY_REPORT, --数据上报公司
'3002' AS COMPANY_MANAGER, --投资管理公司
'3002' AS COMPANY_OWNER, --投资主体公司
P1.MAPPING_KEY AS BUSINESS_LINE, --业务线
CASE
WHEN T1.TRDTYPE in ('42', '82', '84') THEN '01001002'
WHEN T1.TRDTYPE in ('43', '83', '85') THEN '01001003'
END AS DETAIL_ASSET_TYPE, --明细资产分类
CASE p2.mapping_key
WHEN '005' THEN '001'
ELSE '001'
END AS FUND_SOURCE, --资金来源, 把005修改成001后上报，001保持不变
P.PORTFOLIO_UNITS AS PORTFOLIO_UNITS, --投资组合
T1.I_CODE AS SECURITY_ID, --证券代码,给流水号
T1.I_CODE AS SECURITY_NAME, --证券名称,给流水号
nvl(T20.TARGET_CODE,'OTHER') AS MARKET_NO, --市场代码
nvl(T20.TARGET_NAME,'其他') AS MARKET_NAME, --市场名称
oc.partyname AS COUNTER_PARTY, --交易对手名称
T1.partyid AS COUNTER_PARTY_ID, --交易对手组织机构代码
T2.AMOUNT AS HOLDING_UNITS, --持仓数量PAR_VALUE
T2.AMOUNT AS BOOK_BALANCE_N, --财务账面余额_本币AMOUNT
T2.AMOUNT AS BOOK_BALANCE_B, --财务账面余额_本币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_N, --参考财务成本_原币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_B, --参考财务成本_本币AMOUNT
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE('2017-06-30'),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_N, --应计利息_原币
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE('2017-06-30'),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_B --应计利息_本币
FROM bridge.FBMS_A_TTRD_OTC_TRADE_IB T1
INNER JOIN bridge.FBMS_A_TTRD_OTC_IBLB_IB T2 ON T2.I_CODE = T1.INTORDID
AND T2.M_TYPE = T1.M_TYPE
AND T2.A_TYPE = T1.A_TYPE
AND T2.ENDSETDATE > TO_DATE('2017-06-30') --投资组合映射主表
left join (select v1.accountid, v2.murex_leaf, v2.Third_Business_Line as business_line, v2.portfolio_units, v2.fund_source, v2.account_class
from bridge.BRIDGE_APP_PORTFOLIOACCOUNT_ib v1
inner join bridge.bridge_murex_leaf_config v2 on v1.portfolioid = v2.murex_leaf
and ascii(v2.is_send_group) = -26
and v1.biz_date = v2.biz_date
where '20170630' = v2.biz_date) p on p.accountid = t1.secu_accid
--业务线
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'BUSINESS_LINE' and '20170630' = biz_date) p1 on p.business_line = p1.chinese_name
--资金来源
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'FUND_SOURCE' and '20170630' = biz_date) p2 on p.fund_source = p2.chinese_name
--会计分类
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = 'ACCOUNT_CLASS' and '20170630' = biz_date) p3 on p.account_class = p3.chinese_name
--交易对手(交易对手SUMMIT使用的是衡泰的数据源，衡泰是从交易中心获取的，可以通用)
left join (select max(partyname) as partyname, partyid from bridge.FBMS_A_TTRD_OTC_COUNTERPARTY_IB where '20170630' = biz_date group by partyid) oc on t1.partyid = oc.partyid
left join bridge.bridge_security_to_cio_mapping_ib T20
on T1.M_TYPE = T20.SOURCE_CODE
and T20.SOURCE_SYSTEM_CODE = 'FBMS'
and T20.MAPPING_TYPE ='MARKET_NO'
and T20.SOURCE_COLUMN_NAME = 'M_TYPE'
and T20.SOURCE_TABLE_NAME = 'FBMS_A_TTRD_OTC_TRADE_IB'
and T20.biz_date='20170630'
WHERE T1.ORDSOURCE NOT IN ('4', '11') --过滤掉投顾指令单
AND T1.ORDSTATUS = '7'
AND T1.A_TYPE in ('SPT_IBOR')
AND T1.M_TYPE = 'X_CNBD'
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #311
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #311
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #312
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #312
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1510995351625 end=1510995351625 duration=0 from=org.apache.hadoop.hive.ql.Driver>
hive> 17/11/18 16:56:11 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 executed
17/11/18 16:56:13 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 expired
17/11/18 16:56:13 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 exited
