重启hadoop集群报错Cannot obtain block length for LocatedBlock故障分析和解决
重启hadoop集群报错Cannot obtain block length for LocatedBlock故障分析和解决
数据存放在Hive中，通过hue工具查询hive中的表报Cannot obtain block length for LocatedBlock异常，这样无法访问hdfs文件的问题必须解决
一.问题背景
问题产生的原因可能是由于前几日Hadoop集群维护的时候，操作不当，重启Hadoop集群，没有关闭写入hdfs程序的入口导致的hdfs文件写入后状态不一致。排查和解决过程如下.
二.解决过程
1.既然是hdfs文件出问题,用fsck检查一下
su - hadoop
hdfs fsck /
当然你可以具体到指定的hdfs路径,检查完打印结果没有发现任何异常，没有发现损坏或者Corrupt的block，继续排查
2.那么加上其他参数细查
hdfs fsck / –openforwrite
ok,这次检查出来不少文件打印显示都是 openforwrite状态,而且我测试相应文件确实不能读取,这很不正常不是吗？Flume已经写过的hdfs文件居然还处于openforwrite状态，而且无法cat和get
所以这里的”Cannot obtain block length for LocatedBlock”结合字面意思讲应该是当前有文件处于写入状态尚未关闭，无法与对应的datanode通信来成功标识其block长度.
那么分析其产生的可能性，举栗子如下
1>Flume客户端写入hdfs文件时的网络连接被不正常的关闭了
或者
2>Flume或者其他客户端写入hdfs失败了，而且其replication副本也丢失了
我这里应该属于第一种，总结一下就是Flume写入的hdfs文件由于什么原因没有被正常close，状态不一致随后无法正常访问.继续排查
3.推断:HDFS文件租约未释放
了解过HDFS租约后我们知道,客户端在每次读写HDFS文件的时候获取租约对文件进行读写，文件读取完毕了，然后再释放此租约.文件状态就是关闭的了。
但是结合当前场景由于先关闭的hadoop集群，后关闭的Flume sink hdfs,那么hadoop集群都关了，Flume还在对hdfs文件写入，那么租约最后释放了吗？答案是肯定没释放.
4.恢复租约
对于这些状态损坏的文件来讲，rm掉的话是很暴力的做法，万一上游对应日期的数据已经没有rention呢？所以，既然没有释放租约，那么恢复租约close掉文件就是了,如下命令
hdfs debug recoverLease -path <path-of-the-file> -retries <retry times>
请将<path-of-the-file>修改成你需要恢复的租约状态不一致的hdfs文件的具体路径,如果要恢复的很多，可以写个自动化脚本来找出需要恢复的所有文件然后统一恢复租约.
ok，执行完命令后再次cat对应hdfs文件已无异常，顺利显示内容，问题解决
**************************
也有如下的解决办法(记录一下，不推荐，这样会把数据删除，如果上游数据没有了这些数据就丢失了)：
在hadoop测试集群运行job的过程中发现部分运行失败，有Cannot obtain block length for LocatedBlock，使用hdfs dfs -cat ${文件}的时候也报这个错，看过代码后发现是文件未被正常关闭(flume传到hdfs上的，可能由于Namenode长时间gc或者切换或者重启会留下很多.tmp结尾的文件)，为此，使用hadoop命令显示处于打开状态的文件，然后删除这些文件：
hadoop fsck / -openforwrite | egrep -v '^\.+$' | egrep "MISSING|OPENFORWRITE" | grep -o "/[^ ]*" | sed -e "s/:$//" | xargs -i hadoop fs -rmr {};
然后重传这些文件到hdfs
**************************
具体分析过程：
1.过滤出现问题的hdfs文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hdfs fsck / -openforwrite | egrep -v '^\.+$' | egrep "MISSING|OPENFORWRITE"
Connecting to namenode via http://uhadoop-bwgkeu-master2:50070
/hbase/MasterProcWALs/state-00000000000000002635.log 0 bytes, 0 block(s), OPENFORWRITE: /hbase/WALs/uhadoop-bwgkeu-core1,60020,1498641938863/uhadoop-bwgkeu-core1%2C60020%2C1498641938863.default.1508117402394 83 bytes, 1 block(s), OPENFORWRITE:
/hbase/WALs/uhadoop-bwgkeu-core1,60020,1498641938863/uhadoop-bwgkeu-core1%2C60020%2C1498641938863.default.1508117402394: MISSING 1 blocks of total size 83 B./hbase/WALs/uhadoop-bwgkeu-core2,60020,1498641949156/uhadoop-bwgkeu-core2%2C60020%2C1498641949156.default.1508117425337 83 bytes, 1 block(s), OPENFORWRITE:
/hbase/WALs/uhadoop-bwgkeu-core2,60020,1498641949156/uhadoop-bwgkeu-core2%2C60020%2C1498641949156.default.1508117425337: MISSING 1 blocks of total size 83 B./hbase/WALs/uhadoop-bwgkeu-core3,60020,1498641936308/uhadoop-bwgkeu-core3%2C60020%2C1498641936308..meta.1508117439450.meta 83 bytes, 1 block(s), OPENFORWRITE:
/hbase/WALs/uhadoop-bwgkeu-core3,60020,1498641936308/uhadoop-bwgkeu-core3%2C60020%2C1498641936308..meta.1508117439450.meta: MISSING 1 blocks of total size 83 B./hbase/WALs/uhadoop-bwgkeu-core3,60020,1498641936308/uhadoop-bwgkeu-core3%2C60020%2C1498641936308.default.1508117439417 83 bytes, 1 block(s), OPENFORWRITE:
/hbase/WALs/uhadoop-bwgkeu-core3,60020,1498641936308/uhadoop-bwgkeu-core3%2C60020%2C1498641936308.default.1508117439417: MISSING 1 blocks of total size 83 B./hbase/WALs/uhadoop-bwgkeu-core4,60020,1498641947912/uhadoop-bwgkeu-core4%2C60020%2C1498641947912.default.1508117427255 83 bytes, 1 block(s), OPENFORWRITE:
/hbase/WALs/uhadoop-bwgkeu-core4,60020,1498641947912/uhadoop-bwgkeu-core4%2C60020%2C1498641947912.default.1508117427255: MISSING 1 blocks of total size 83 B...............................................................................................
................................................................./user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/0865f0ae-aaa4-4485-9a29-b4a0c1d962fc.avro 19676160 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/0865f0ae-aaa4-4485-9a29-b4a0c1d962fc.avro: MISSING 1 blocks of total size 19676160 B.../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/15032fea-033a-412e-afa3-f1aeaf6bdc6c.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/15032fea-033a-412e-afa3-f1aeaf6bdc6c.avro: MISSING 1 blocks of total size 64512 B......................../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/8a19bcde-b786-47b4-8f56-4842e7f7768e.avro 22127616 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/8a19bcde-b786-47b4-8f56-4842e7f7768e.avro: MISSING 1 blocks of total size 22127616 B...../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/a5c0bb67-0c16-42bc-b01a-3607bcbf344e.avro 6193152 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/a5c0bb67-0c16-42bc-b01a-3607bcbf344e.avro: MISSING 1 blocks of total size 6193152 B...
./user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/b213cb77-841e-4f4a-8b3b-9cadfa7a5281.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/b213cb77-841e-4f4a-8b3b-9cadfa7a5281.avro: MISSING 1 blocks of total size 64512 B......./user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/c199e94c-360d-45c5-9e1f-3625846d1697.avro 6128640 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/c199e94c-360d-45c5-9e1f-3625846d1697.avro: MISSING 1 blocks of total size 6128640 B........./user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/eb0896b1-7e7b-419e-8da2-0b2cb5cae4f1.avro 11031552 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/eb0896b1-7e7b-419e-8da2-0b2cb5cae4f1.avro: MISSING 1 blocks of total size 11031552 B../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/ec3d545b-f1d1-4a65-a7fd-cdee404e1c65.avro 17095680 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-13/ec3d545b-f1d1-4a65-a7fd-cdee404e1c65.avro: MISSING 1 blocks of total size 17095680 B..../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-14/01012999-d259-4459-b1e4-1d8d827a3497.avro 20579328 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-14/01012999-d259-4459-b1e4-1d8d827a3497.avro: MISSING 1 blocks of total size 20579328 B../user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-14/9f09655c-7954-468d-94f9-2c0c6f09014c.avro 8064000 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-14/9f09655c-7954-468d-94f9-2c0c6f09014c.avro: MISSING 1 blocks of total size 8064000 B./user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-15/d07f0ab7-2dcc-4d0a-8649-0d5a06e7fa2b.avro 2580480 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_login/dt=2017-10-15/d07f0ab7-2dcc-4d0a-8649-0d5a06e7fa2b.avro: MISSING 1 blocks of total size 2580480 B.................................................................../user/hive/warehouse/tbl_aaa_user_register/dt=2017-10-14/0809d75f-7ea2-443f-9810-249869c8fa23.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_register/dt=2017-10-14/0809d75f-7ea2-443f-9810-249869c8fa23.avro: MISSING 1 blocks of total size 64512 B./user/hive/warehouse/tbl_aaa_user_register/dt=2017-10-15/6887ac77-e0c3-4ae3-b52d-5f63a5198d98.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_aaa_user_register/dt=2017-10-15/6887ac77-e0c3-4ae3-b52d-5f63a5198d98.avro: MISSING 1 blocks of total size 64512 B......
...................../user/hive/warehouse/tbl_bin_task_report/dt=2017-10-14/73838a8b-5cef-42df-b8aa-7a49b5e46450.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_task_report/dt=2017-10-14/73838a8b-5cef-42df-b8aa-7a49b5e46450.avro: MISSING 1 blocks of total size 64512 B................................/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-12/58e2e0d2-0d8c-40eb-a118-5427a57580c2.avro 1183477248 bytes, 9 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-12/58e2e0d2-0d8c-40eb-a118-5427a57580c2.avro: MISSING 1 blocks of total size 109735424 B../user/hive/warehouse/tbl_bin_user_req/dt=2017-10-13/8b65fa0e-0068-4cfa-a063-6d3eae2ddc4e.avro 691152384 bytes, 6 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-13/8b65fa0e-0068-4cfa-a063-6d3eae2ddc4e.avro: MISSING 1 blocks of total size 20063744 B./user/hive/warehouse/tbl_bin_user_req/dt=2017-10-13/d8be28b2-5bb8-410a-b756-fa200172dbde.avro 616092160 bytes, 5 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-13/d8be28b2-5bb8-410a-b756-fa200172dbde.avro: MISSING 1 blocks of total size 79221248 B./user/hive/warehouse/tbl_bin_user_req/dt=2017-10-14/3f639aeb-fd1c-46b6-9876-39481092ca79.avro 1621515776 bytes, 13 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-14/3f639aeb-fd1c-46b6-9876-39481092ca79.avro: MISSING 1 blocks of total size 10903040 B./user/hive/warehouse/tbl_bin_user_req/dt=2017-10-15/65cba6b6-c360-4b5e-9e9a-08c6c59a2c58.avro 1668480512 bytes, 13 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_req/dt=2017-10-15/65cba6b6-c360-4b5e-9e9a-08c6c59a2c58.avro: MISSING 1 blocks of total size 57867776 B...................../user/hive/warehouse/tbl_bin_user_task/dt=2017-10-12/71284a97-c749-42af-85fa-2f7eb1dd9410.avro 4191070720 bytes, 32 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_task/dt=2017-10-12/71284a97-c749-42af-85fa-2f7eb1dd9410.avro: MISSING 1 blocks of total size 30321152 B../user/hive/warehouse/tbl_bin_user_task/dt=2017-10-13/54ad8cc1-30b0-4ba3-a61a-40009f1efdaf.avro 2658904576 bytes, 20 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_task/dt=2017-10-13/54ad8cc1-30b0-4ba3-a61a-40009f1efdaf.avro: MISSING 1 blocks of total size 108767744 B./user/hive/warehouse/tbl_bin_user_task/dt=2017-10-13/e15699de-7075-497a-9419-8f3af578243f.avro 3386861056 bytes, 26 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_task/dt=2017-10-13/e15699de-7075-497a-9419-8f3af578243f.avro: MISSING 1 blocks of total size 31417856 B./user/hive/warehouse/tbl_bin_user_task/dt=2017-10-14/8fa833d9-f112-4e47-8f2b-f406a4bd7409.avro 7924200960 bytes, 60 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_task/dt=2017-10-14/8fa833d9-f112-4e47-8f2b-f406a4bd7409.avro: MISSING 1 blocks of total size 5355008 B./user/hive/warehouse/tbl_bin_user_task/dt=2017-10-15/e1027b60-b637-48bd-8759-742732c21d57.avro 8167960064 bytes, 61 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_bin_user_task/dt=2017-10-15/e1027b60-b637-48bd-8759-742732c21d57.avro: MISSING 1 blocks of total size 114896384 B................
............................................................................../user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/061f1b3e-c35d-4a7e-9b86-4a23bb748a5f.avro 682830336 bytes, 6 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/061f1b3e-c35d-4a7e-9b86-4a23bb748a5f.avro: MISSING 1 blocks of total size 11741696 B../user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro 437425664 bytes, 4 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro: MISSING 1 blocks of total size 34772480 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/c8286332-d0e7-4243-bca1-eb3cd92dcf1a.avro 516130304 bytes, 4 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/c8286332-d0e7-4243-bca1-eb3cd92dcf1a.avro: MISSING 1 blocks of total size 113477120 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-14/d2406a1c-922b-4e56-a699-8f713e113dc1.avro 1057517056 bytes, 8 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-14/d2406a1c-922b-4e56-a699-8f713e113dc1.avro: MISSING 1 blocks of total size 117992960 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-15/35a23aac-afe1-4f95-97ea-73e660310b85.avro 937233920 bytes, 7 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-15/35a23aac-afe1-4f95-97ea-73e660310b85.avro: MISSING 1 blocks of total size 131927552 B.................
............................................................................../user/hive/warehouse/tbl_push_user_req/dt=2017-10-12/a1fbd02a-2385-4453-9c4d-eb805c6bb332.avro 9527362048 bytes, 71 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-12/a1fbd02a-2385-4453-9c4d-eb805c6bb332.avro: MISSING 1 blocks of total size 132121088 B../user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/6fd1601f-c20c-4437-ae19-622c2f5397a7.avro 6598087168 bytes, 50 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/6fd1601f-c20c-4437-ae19-622c2f5397a7.avro: MISSING 1 blocks of total size 21418496 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/90e513ed-71cb-40a5-ad7d-081065cebed0.avro 5747396096 bytes, 43 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/90e513ed-71cb-40a5-ad7d-081065cebed0.avro: MISSING 1 blocks of total size 110251520 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-14/73eb7c87-6fba-4dde-91f3-52a08ab89f54.avro 13647727104 bytes, 102 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-14/73eb7c87-6fba-4dde-91f3-52a08ab89f54.avro: MISSING 1 blocks of total size 91736576 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-15/8775a629-2aad-48a1-a015-7ac8a64861c2.avro 13515573760 bytes, 101 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-15/8775a629-2aad-48a1-a015-7ac8a64861c2.avro: MISSING 1 blocks of total size 93800960 B.................
................................................................../user/hive/warehouse/tbl_script_task_report/dt=2017-10-13/335ed956-1115-4ee0-8bbb-670ec7bcb794.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_task_report/dt=2017-10-13/335ed956-1115-4ee0-8bbb-670ec7bcb794.avro: MISSING 1 blocks of total size 129024 B./user/hive/warehouse/tbl_script_task_report/dt=2017-10-14/128ad816-26fe-49ac-8a74-5ff4ec70b379.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_task_report/dt=2017-10-14/128ad816-26fe-49ac-8a74-5ff4ec70b379.avro: MISSING 1 blocks of total size 129024 B./user/hive/warehouse/tbl_script_task_report/dt=2017-10-15/ad1b7e4c-97c5-4fe6-a6db-c04642343d8d.avro 2064384 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_task_report/dt=2017-10-15/ad1b7e4c-97c5-4fe6-a6db-c04642343d8d.avro: MISSING 1 blocks of total size 2064384 B................................
./user/hive/warehouse/tbl_script_user_req/dt=2017-10-12/24dbb22a-a556-4d27-bdac-765f0de4cfab.avro 60447744 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_req/dt=2017-10-12/24dbb22a-a556-4d27-bdac-765f0de4cfab.avro: MISSING 1 blocks of total size 60447744 B./user/hive/warehouse/tbl_script_user_req/dt=2017-10-13/26f2401e-6afe-404a-849d-2c7b668904d4.avro 1677312 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_req/dt=2017-10-13/26f2401e-6afe-404a-849d-2c7b668904d4.avro: MISSING 1 blocks of total size 1677312 B./user/hive/warehouse/tbl_script_user_req/dt=2017-10-13/c4f5ef87-fda9-4aa2-ba33-ce8c884423a9.avro 22514688 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_req/dt=2017-10-13/c4f5ef87-fda9-4aa2-ba33-ce8c884423a9.avro: MISSING 1 blocks of total size 22514688 B./user/hive/warehouse/tbl_script_user_req/dt=2017-10-14/5e92588f-3ec3-4895-8d07-cecc9cfbb527.avro 22256640 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_req/dt=2017-10-14/5e92588f-3ec3-4895-8d07-cecc9cfbb527.avro: MISSING 1 blocks of total size 22256640 B./user/hive/warehouse/tbl_script_user_req/dt=2017-10-15/bb27dcce-b146-4a5a-a404-356f620092c8.avro 233373184 bytes, 2 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_req/dt=2017-10-15/bb27dcce-b146-4a5a-a404-356f620092c8.avro: MISSING 1 blocks of total size 99155456 B................................../user/hive/warehouse/tbl_script_user_task/dt=2017-10-12/f6f779a7-5fa7-42e5-9d6b-b799a9ee363c.avro 42642432 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_task/dt=2017-10-12/f6f779a7-5fa7-42e5-9d6b-b799a9ee363c.avro: MISSING 1 blocks of total size 42642432 B./user/hive/warehouse/tbl_script_user_task/dt=2017-10-13/94879c02-1515-46e2-87ae-4959fffcb60f.avro 1161216 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_task/dt=2017-10-13/94879c02-1515-46e2-87ae-4959fffcb60f.avro: MISSING 1 blocks of total size 1161216 B./user/hive/warehouse/tbl_script_user_task/dt=2017-10-13/b090f1fc-b0c9-4438-9725-86cf483fc0a4.avro 15547392 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_task/dt=2017-10-13/b090f1fc-b0c9-4438-9725-86cf483fc0a4.avro: MISSING 1 blocks of total size 15547392 B./user/hive/warehouse/tbl_script_user_task/dt=2017-10-14/882f80c1-f1e1-45a1-925a-04cb32533860.avro 15353856 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_task/dt=2017-10-14/882f80c1-f1e1-45a1-925a-04cb32533860.avro: MISSING 1 blocks of total size 15353856 B./user/hive/warehouse/tbl_script_user_task/dt=2017-10-15/adbca2bb-6fe2-4024-88d5-6564757b8a7c.avro 157120000 bytes, 2 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_script_user_task/dt=2017-10-15/adbca2bb-6fe2-4024-88d5-6564757b8a7c.avro: MISSING 1 blocks of total size 22902272 B.........................................................
..../user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-12/02a1bf7a-b9a2-4d24-a115-e26de6d0f3c1.avro 829176320 bytes, 7 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-12/02a1bf7a-b9a2-4d24-a115-e26de6d0f3c1.avro: MISSING 1 blocks of total size 23869952 B../user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-13/14614409-f509-4ba4-9524-94c48f65c4a7.avro 580933120 bytes, 5 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-13/14614409-f509-4ba4-9524-94c48f65c4a7.avro: MISSING 1 blocks of total size 44062208 B./user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-13/8e005354-4adf-4f1b-a1d2-e880f8771b0e.avro 353462784 bytes, 3 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-13/8e005354-4adf-4f1b-a1d2-e880f8771b0e.avro: MISSING 1 blocks of total size 85027328 B./user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-14/14bca449-6b9c-479c-ac71-2ccfee333849.avro 1116449280 bytes, 9 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-14/14bca449-6b9c-479c-ac71-2ccfee333849.avro: MISSING 1 blocks of total size 42707456 B./user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-15/732cff8d-fcdb-46d4-8f26-87e727e11834.avro 1115352576 bytes, 9 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_result/dt=2017-10-15/732cff8d-fcdb-46d4-8f26-87e727e11834.avro: MISSING 1 blocks of total size 41610752 B............................................................./user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-12/8e69c855-08bd-45c7-9b86-73306cce6110.avro 5144593920 bytes, 39 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-12/8e69c855-08bd-45c7-9b86-73306cce6110.avro: MISSING 1 blocks of total size 44320256 B../user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-13/2462ac9d-913f-4023-943f-6e56576834c9.avro 3378797056 bytes, 26 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-13/2462ac9d-913f-4023-943f-6e56576834c9.avro: MISSING 1 blocks of total size 23353856 B./user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-13/c3cbaf93-f84d-4fa2-bb76-497b411f66ae.avro 2056876544 bytes, 16 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-13/c3cbaf93-f84d-4fa2-bb76-497b411f66ae.avro: MISSING 1 blocks of total size 43610624 B./user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-14/1cef6a08-2226-449c-ac80-df8e694465c1.avro 5945577984 bytes, 45 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-14/1cef6a08-2226-449c-ac80-df8e694465c1.avro: MISSING 1 blocks of total size 39997952 B./user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-15/a109bd87-dd4a-4e7d-891e-9fbd19bd7fd5.avro 6118631936 bytes, 46 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_speech_voice_upload/dt=2017-10-15/a109bd87-dd4a-4e7d-891e-9fbd19bd7fd5.avro: MISSING 1 blocks of total size 78834176 B.........................
........................................................................................./user/hive/warehouse/tbl_voice_troops_text_msg/dt=2017-10-13/c82904dc-d9b2-4ec4-9e47-747e50055886.avro 64512 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_text_msg/dt=2017-10-13/c82904dc-d9b2-4ec4-9e47-747e50055886.avro: MISSING 1 blocks of total size 64512 B...........
......................................................................../user/hive/warehouse/tbl_voice_troops_user_login/dt=2017-10-14/bfa03c0e-dd87-4167-9150-5b72e9b01f42.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_user_login/dt=2017-10-14/bfa03c0e-dd87-4167-9150-5b72e9b01f42.avro: MISSING 1 blocks of total size 129024 B...../user/hive/warehouse/tbl_voice_troops_user_login/dt=2017-10-15/822470c3-7944-45d2-9c9b-95a73e58974f.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_user_login/dt=2017-10-15/822470c3-7944-45d2-9c9b-95a73e58974f.avro: MISSING 1 blocks of total size 129024 B.......................
............................./user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-13/640e1314-4ad4-4393-8a2b-35d4e4f5c85f.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-13/640e1314-4ad4-4393-8a2b-35d4e4f5c85f.avro: MISSING 1 blocks of total size 129024 B./user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-13/6da6f234-bb2b-4d1e-b400-4b04d6471cd3.avro 196020736 bytes, 2 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-13/6da6f234-bb2b-4d1e-b400-4b04d6471cd3.avro: MISSING 1 blocks of total size 61803008 B............................/user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-15/41614fe9-3421-4285-b227-605ca6f2edbf.avro 129024 bytes, 1 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_voice_troops_user_logout/dt=2017-10-15/41614fe9-3421-4285-b227-605ca6f2edbf.avro: MISSING 1 blocks of total size 129024 B..........................................
............................................................................./yunva/log/aaa/user_login/loginlog/day=20171008/.681eb3fd-81f1-4451-8e3c-6781e1526e6a.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/aaa/user_login/loginlog/day=20171008/.681eb3fd-81f1-4451-8e3c-6781e1526e6a.avro.tmp: MISSING 1 blocks of total size 64512 B../yunva/log/aaa/user_login/loginlog/day=20171008/.961c8f89-228f-467a-a42f-c765faa77446.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/aaa/user_login/loginlog/day=20171008/.961c8f89-228f-467a-a42f-c765faa77446.avro.tmp: MISSING 1 blocks of total size 64512 B................./yunva/log/aaa/user_login/loginlog/day=20171016/.51c960d0-0cc2-41ce-8f0f-e98e258c6ef0.avro.tmp 40126464 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/aaa/user_login/loginlog/day=20171016/.51c960d0-0cc2-41ce-8f0f-e98e258c6ef0.avro.tmp: MISSING 1 blocks of total size 40126464 B./yunva/log/aaa/user_login/loginlog/day=20171016/.5ee9adb5-21de-4707-9e9f-cab76c906931.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/aaa/user_login/loginlog/day=20171016/.5ee9adb5-21de-4707-9e9f-cab76c906931.avro.tmp: MISSING 1 blocks of total size 0 B...
................................................................./yunva/log/aaa/user_register/registerlog/day=20171016/.1e8ab233-69eb-4108-9889-321b73af2f11.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/aaa/user_register/registerlog/day=20171016/.624df6ea-e8a4-4246-91e6-2158e03bb7c9.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/aaa/user_register/registerlog/day=20171016/.624df6ea-e8a4-4246-91e6-2158e03bb7c9.avro.tmp: MISSING 1 blocks of total size 64512 B............./yunva/log/bin/bin_user_report/reportbinresultlog/day=20171016/.f99a6c9b-a417-4f8e-804c-619ca2f5d39f.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/bin/bin_user_report/reportbinresultlog/day=20171016/.f99a6c9b-a417-4f8e-804c-619ca2f5d39f.avro.tmp: MISSING 1 blocks of total size 64512 B..../yunva/log/bin/bin_user_req/getbinreqlog/day=20171016/.cb3085cd-df25-4647-81e7-08a4769ae97d.avro.tmp 134217728 bytes, 1 block(s), OPENFORWRITE: ................
/yunva/log/bin/bin_user_task/getbinresultlog/day=20171016/.ce48a779-54fd-49b5-af08-7cda2ea2c8f5.avro.tmp 1073741824 bytes, 8 block(s), OPENFORWRITE: ......................................................................../yunva/log/push/push_task_result/pushresultlog/day=20171012/.5280080a-c36d-4c09-8cb5-39a4160400e2.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/push/push_task_result/pushresultlog/day=20171013/.3548f984-22d8-48b0-aa25-b20c2e8fc527.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/push/push_task_result/pushresultlog/day=20171014/.5e5d1895-cb8c-4e3c-a0ce-557af2fb8647.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/push/push_task_result/pushresultlog/day=20171015/.858c9d1a-02cc-457e-970a-eaec62090baf.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/push/push_task_result/pushresultlog/day=20171016/.93d23248-2d05-4ce4-8ad4-9c7b278ac868.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ......................
....................../yunva/log/push/push_user_app/userapplog/day=20171016/.67c579f7-3345-46f1-96d3-9103ed8d5e61.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_app/userapplog/day=20171016/.67c579f7-3345-46f1-96d3-9103ed8d5e61.avro.tmp: MISSING 1 blocks of total size 0 B..../yunva/log/push/push_user_req/pushreqlog/day=20170627/.0cde60aa-a82e-43d2-bcf4-ee6e43760940.avro.tmp 4838400 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.0cde60aa-a82e-43d2-bcf4-ee6e43760940.avro.tmp: MISSING 1 blocks of total size 4838400 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.324522cc-9338-465f-9f91-3a2359403551.avro.tmp 9870336 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.324522cc-9338-465f-9f91-3a2359403551.avro.tmp: MISSING 1 blocks of total size 9870336 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.7eb22588-e800-4852-9065-c5fce68640fe.avro.tmp 2386944 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.7eb22588-e800-4852-9065-c5fce68640fe.avro.tmp: MISSING 1 blocks of total size 2386944 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.a5f3ec8c-ce9e-4975-98c3-0b989b6d8e8a.avro.tmp 9870336 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.a5f3ec8c-ce9e-4975-98c3-0b989b6d8e8a.avro.tmp: MISSING 1 blocks of total size 9870336 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.b11a3062-e083-4da8-a4cc-1ab3995b0f70.avro.tmp 223115776 bytes, 2 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.b11a3062-e083-4da8-a4cc-1ab3995b0f70.avro.tmp: MISSING 1 blocks of total size 88898048 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.f583128f-c477-4423-a42c-3ee36cc08504.avro.tmp 1870848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.f583128f-c477-4423-a42c-3ee36cc08504.avro.tmp: MISSING 1 blocks of total size 1870848 B.....................................................................
........................../yunva/log/push/push_user_req/pushreqlog/day=20170628/.28707d47-03ae-4d66-8587-035b3da91a23.avro.tmp 9934848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.28707d47-03ae-4d66-8587-035b3da91a23.avro.tmp: MISSING 1 blocks of total size 9934848 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.42f3cd58-9c1f-402d-89df-f730dda91042.avro.tmp 8902656 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.42f3cd58-9c1f-402d-89df-f730dda91042.avro.tmp: MISSING 1 blocks of total size 8902656 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.6e076565-87db-4c2d-a3e8-f83a649906c0.avro.tmp 8455716864 bytes, 63 block(s), OPENFORWRITE: /yunva/log/push/push_user_req/pushreqlog/day=20170628/.87a2448c-cbb7-4fd7-a4da-86a6ffbfe6c8.avro.tmp 9934848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.87a2448c-cbb7-4fd7-a4da-86a6ffbfe6c8.avro.tmp: MISSING 1 blocks of total size 9934848 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.b2f2a6ed-58da-49c8-9788-5432cd94d2c5.avro.tmp 11870208 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.b2f2a6ed-58da-49c8-9788-5432cd94d2c5.avro.tmp: MISSING 1 blocks of total size 11870208 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.fba2ae7d-3595-44e7-969d-4de0cfd6c500.avro.tmp 4967424 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.fba2ae7d-3595-44e7-969d-4de0cfd6c500.avro.tmp: MISSING 1 blocks of total size 4967424 B.....................................................................
.............................../yunva/log/push/push_user_req/pushreqlog/day=20171016/.1961835f-f57b-48b7-939c-67cde3f8c4c6.avro.tmp 3087007744 bytes, 23 block(s), OPENFORWRITE: ................................................................/yunva/log/script/script_user_report/reportscriptresultlog/day=20171016/.22fc73b3-8b89-4308-ba7c-2d301db6966e.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/script/script_user_report/reportscriptresultlog/day=20171016/.22fc73b3-8b89-4308-ba7c-2d301db6966e.avro.tmp: MISSING 1 blocks of total size 64512 B./yunva/log/script/script_user_report/reportscriptresultlog/day=20171016/.37da2812-84aa-4802-87d9-8c9d07c1df1b.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/script/script_user_report/reportscriptresultlog/day=20171016/.37da2812-84aa-4802-87d9-8c9d07c1df1b.avro.tmp: MISSING 1 blocks of total size 0 B...
./yunva/log/script/script_user_req/getscriptreqlog/day=20171016/.3415a275-1c89-4df1-a119-d7c384daead6.avro.tmp 209181184 bytes, 2 block(s), OPENFORWRITE:
/yunva/log/script/script_user_req/getscriptreqlog/day=20171016/.3415a275-1c89-4df1-a119-d7c384daead6.avro.tmp: MISSING 1 blocks of total size 74963456 B./yunva/log/script/script_user_req/getscriptreqlog/day=20171016/.64ee4460-de2b-417a-b553-59bf6a7c00e7.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/script/script_user_req/getscriptreqlog/day=20171016/.64ee4460-de2b-417a-b553-59bf6a7c00e7.avro.tmp: MISSING 1 blocks of total size 0 B..../yunva/log/script/script_user_task/getscriptresultlog/day=20171016/.47281cd2-c17a-43c9-9536-fbe92d2f1278.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/script/script_user_task/getscriptresultlog/day=20171016/.47281cd2-c17a-43c9-9536-fbe92d2f1278.avro.tmp: MISSING 1 blocks of total size 0 B./yunva/log/script/script_user_task/getscriptresultlog/day=20171016/.d55fcce8-f608-42bf-b15e-51de2e66f983.avro.tmp 148797952 bytes, 2 block(s), OPENFORWRITE:
/yunva/log/script/script_user_task/getscriptresultlog/day=20171016/.d55fcce8-f608-42bf-b15e-51de2e66f983.avro.tmp: MISSING 1 blocks of total size 14580224 B.........../yunva/log/speech/speech_voice_result/speechfilelog/day=20171016/.73303f6f-f304-4174-900b-327b075c7392.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/speech/speech_voice_result/speechfilelog/day=20171016/.73303f6f-f304-4174-900b-327b075c7392.avro.tmp: MISSING 1 blocks of total size 0 B.........../yunva/log/speech/speech_voice_upload/uploadfilelog/day=20171016/.dad4b76f-4a56-44e3-a25c-13ca9f6c8e1e.avro.tmp 402653184 bytes, 3 block(s), OPENFORWRITE: ......................................................../yunva/log/voice/live/voice_live_login/userloginreq/day=20171012/.3f2a56f4-225e-4c5a-a078-adbc58463cdf.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_login/userloginreq/day=20171013/.4b2e3bfe-50c7-4328-9e49-1ee231ec01d9.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_login/userloginreq/day=20171014/.a9ed2c01-1d3b-44c3-9aeb-71a02a06cc30.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_login/userloginreq/day=20171015/.487042de-8f35-4cc9-9ee2-ec0d2a1569bd.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ..........
...................................................................../yunva/log/voice/live/voice_live_logout/userlogoutreq/day=20171012/.213af552-78dd-4af9-a60d-f9d98b291f7b.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ./yunva/log/voice/live/voice_live_logout/userlogoutreq/day=20171013/.c559de67-bd83-4402-8bec-2900b820dd9b.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_logout/userlogoutreq/day=20171014/.bf94cdb2-0ed7-4982-9bdc-77a52095c872.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_logout/userlogoutreq/day=20171015/.04a8371b-3dd6-44c1-a46a-049bf43e5a7a.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ..........................
../yunva/log/voice/live/voice_live_msg/usermsgreq/day=20171012/.dc23f808-a2df-4339-8d53-bcedb430a17f.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/live/voice_live_msg/usermsgreq/day=20171013/.3b494920-fe17-4c3e-b4b3-91edc1297654.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: .................................../yunva/log/voice/troops/voice_mic_lose/usermicloselog/day=20171016/.1a5dd569-75fd-4e61-b033-28ab8ebe00dc.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ....................................../yunva/log/voice/troops/voice_mic_used/usermicusedlog/day=20171016/.7d9e04d8-33af-482e-83b6-de3601d1e0ad.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: .....................
........./yunva/log/voice/troops/voice_send_text/usertextlog/day=20171012/.ae027c60-5202-4eb5-b508-cafe5fdab24b.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_send_text/usertextlog/day=20171012/.ae027c60-5202-4eb5-b508-cafe5fdab24b.avro.tmp: MISSING 1 blocks of total size 64512 B..../yunva/log/voice/troops/voice_send_text/usertextlog/day=20171016/.217655d3-ca83-430d-b57c-453a49150a97.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: ........./yunva/log/voice/troops/voice_user_login/userloginlog/day=20171014/.a3c087b3-e84f-4b92-b0ab-83b3614a4122.avro.tmp 129024 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_login/userloginlog/day=20171014/.a3c087b3-e84f-4b92-b0ab-83b3614a4122.avro.tmp: MISSING 1 blocks of total size 129024 B./yunva/log/voice/troops/voice_user_login/userloginlog/day=20171015/.78df15ac-9675-4d9b-bf41-b1938da619d4.avro.tmp 0 bytes, 0 block(s), OPENFORWRITE: /yunva/log/voice/troops/voice_user_login/userloginlog/day=20171016/.22b4a1c1-5a6c-4fe1-a526-700d3716eac6.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_login/userloginlog/day=20171016/.22b4a1c1-5a6c-4fe1-a526-700d3716eac6.avro.tmp: MISSING 1 blocks of total size 0 B./yunva/log/voice/troops/voice_user_login/userloginlog/day=20171016/.336d0899-4753-472b-958e-fd0a81b02850.avro.tmp 3354624 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_login/userloginlog/day=20171016/.336d0899-4753-472b-958e-fd0a81b02850.avro.tmp: MISSING 1 blocks of total size 3354624 B................/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171012/.0a1431c2-dcae-4480-9f0d-0079e759187b.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171012/.0a1431c2-dcae-4480-9f0d-0079e759187b.avro.tmp: MISSING 1 blocks of total size 64512 B./yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171013/.5ca01fba-1e29-432e-9210-eba47f10b765.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171013/.5ca01fba-1e29-432e-9210-eba47f10b765.avro.tmp: MISSING 1 blocks of total size 64512 B../yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171015/.a1992c28-3648-49c2-8dcf-0255c504f331.avro.tmp 129024 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171015/.a1992c28-3648-49c2-8dcf-0255c504f331.avro.tmp: MISSING 1 blocks of total size 129024 B./yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171016/.6c1af28f-6d11-4c6b-902a-1e8c221716e8.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171016/.6c1af28f-6d11-4c6b-902a-1e8c221716e8.avro.tmp: MISSING 1 blocks of total size 0 B./yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171016/.9abc1f33-c001-40a0-9d72-6efe75cb43b7.avro.tmp 2967552 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/voice/troops/voice_user_logout/userlogoutlog/day=20171016/.9abc1f33-c001-40a0-9d72-6efe75cb43b7.avro.tmp: MISSING 1 blocks of total size 2967552 B.....................................................
MISSING BLOCKS: 102
MISSING SIZE: 2677460383 B
2.进一步筛选定位具体出问题的Hive数据表信息
[hadoop@uhadoop-bwgkeu-master1 ~]$ hdfs fsck / -openforwrite | egrep -v '^\.+$' | egrep "push_user_app|push_user_req|push_user_task"
Connecting to namenode via http://uhadoop-bwgkeu-master2:50070
............................................................................../user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/061f1b3e-c35d-4a7e-9b86-4a23bb748a5f.avro 682830336 bytes, 6 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/061f1b3e-c35d-4a7e-9b86-4a23bb748a5f.avro: MISSING 1 blocks of total size 11741696 B../user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro 437425664 bytes, 4 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro: MISSING 1 blocks of total size 34772480 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/c8286332-d0e7-4243-bca1-eb3cd92dcf1a.avro 516130304 bytes, 4 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/c8286332-d0e7-4243-bca1-eb3cd92dcf1a.avro: MISSING 1 blocks of total size 113477120 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-14/d2406a1c-922b-4e56-a699-8f713e113dc1.avro 1057517056 bytes, 8 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-14/d2406a1c-922b-4e56-a699-8f713e113dc1.avro: MISSING 1 blocks of total size 117992960 B./user/hive/warehouse/tbl_push_user_app/dt=2017-10-15/35a23aac-afe1-4f95-97ea-73e660310b85.avro 937233920 bytes, 7 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_app/dt=2017-10-15/35a23aac-afe1-4f95-97ea-73e660310b85.avro: MISSING 1 blocks of total size 131927552 B.................
............................................................................../user/hive/warehouse/tbl_push_user_req/dt=2017-10-12/a1fbd02a-2385-4453-9c4d-eb805c6bb332.avro 9527362048 bytes, 71 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-12/a1fbd02a-2385-4453-9c4d-eb805c6bb332.avro: MISSING 1 blocks of total size 132121088 B../user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/6fd1601f-c20c-4437-ae19-622c2f5397a7.avro 6598087168 bytes, 50 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/6fd1601f-c20c-4437-ae19-622c2f5397a7.avro: MISSING 1 blocks of total size 21418496 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/90e513ed-71cb-40a5-ad7d-081065cebed0.avro 5747396096 bytes, 43 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-13/90e513ed-71cb-40a5-ad7d-081065cebed0.avro: MISSING 1 blocks of total size 110251520 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-14/73eb7c87-6fba-4dde-91f3-52a08ab89f54.avro 13647727104 bytes, 102 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-14/73eb7c87-6fba-4dde-91f3-52a08ab89f54.avro: MISSING 1 blocks of total size 91736576 B./user/hive/warehouse/tbl_push_user_req/dt=2017-10-15/8775a629-2aad-48a1-a015-7ac8a64861c2.avro 13515573760 bytes, 101 block(s), OPENFORWRITE:
/user/hive/warehouse/tbl_push_user_req/dt=2017-10-15/8775a629-2aad-48a1-a015-7ac8a64861c2.avro: MISSING 1 blocks of total size 93800960 B.................
....................../yunva/log/push/push_user_app/userapplog/day=20171016/.67c579f7-3345-46f1-96d3-9103ed8d5e61.avro.tmp 0 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_app/userapplog/day=20171016/.67c579f7-3345-46f1-96d3-9103ed8d5e61.avro.tmp: MISSING 1 blocks of total size 0 B..../yunva/log/push/push_user_req/pushreqlog/day=20170627/.0cde60aa-a82e-43d2-bcf4-ee6e43760940.avro.tmp 4838400 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.0cde60aa-a82e-43d2-bcf4-ee6e43760940.avro.tmp: MISSING 1 blocks of total size 4838400 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.324522cc-9338-465f-9f91-3a2359403551.avro.tmp 9870336 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.324522cc-9338-465f-9f91-3a2359403551.avro.tmp: MISSING 1 blocks of total size 9870336 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.7eb22588-e800-4852-9065-c5fce68640fe.avro.tmp 2386944 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.7eb22588-e800-4852-9065-c5fce68640fe.avro.tmp: MISSING 1 blocks of total size 2386944 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.a5f3ec8c-ce9e-4975-98c3-0b989b6d8e8a.avro.tmp 9870336 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.a5f3ec8c-ce9e-4975-98c3-0b989b6d8e8a.avro.tmp: MISSING 1 blocks of total size 9870336 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.b11a3062-e083-4da8-a4cc-1ab3995b0f70.avro.tmp 223115776 bytes, 2 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.b11a3062-e083-4da8-a4cc-1ab3995b0f70.avro.tmp: MISSING 1 blocks of total size 88898048 B./yunva/log/push/push_user_req/pushreqlog/day=20170627/.f583128f-c477-4423-a42c-3ee36cc08504.avro.tmp 1870848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170627/.f583128f-c477-4423-a42c-3ee36cc08504.avro.tmp: MISSING 1 blocks of total size 1870848 B.....................................................................
........................../yunva/log/push/push_user_req/pushreqlog/day=20170628/.28707d47-03ae-4d66-8587-035b3da91a23.avro.tmp 9934848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.28707d47-03ae-4d66-8587-035b3da91a23.avro.tmp: MISSING 1 blocks of total size 9934848 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.42f3cd58-9c1f-402d-89df-f730dda91042.avro.tmp 8902656 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.42f3cd58-9c1f-402d-89df-f730dda91042.avro.tmp: MISSING 1 blocks of total size 8902656 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.6e076565-87db-4c2d-a3e8-f83a649906c0.avro.tmp 8455716864 bytes, 63 block(s), OPENFORWRITE: /yunva/log/push/push_user_req/pushreqlog/day=20170628/.87a2448c-cbb7-4fd7-a4da-86a6ffbfe6c8.avro.tmp 9934848 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.87a2448c-cbb7-4fd7-a4da-86a6ffbfe6c8.avro.tmp: MISSING 1 blocks of total size 9934848 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.b2f2a6ed-58da-49c8-9788-5432cd94d2c5.avro.tmp 11870208 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.b2f2a6ed-58da-49c8-9788-5432cd94d2c5.avro.tmp: MISSING 1 blocks of total size 11870208 B./yunva/log/push/push_user_req/pushreqlog/day=20170628/.fba2ae7d-3595-44e7-969d-4de0cfd6c500.avro.tmp 4967424 bytes, 1 block(s), OPENFORWRITE:
/yunva/log/push/push_user_req/pushreqlog/day=20170628/.fba2ae7d-3595-44e7-969d-4de0cfd6c500.avro.tmp: MISSING 1 blocks of total size 4967424 B.....................................................................
.............................../yunva/log/push/push_user_req/pushreqlog/day=20171016/.1961835f-f57b-48b7-939c-67cde3f8c4c6.avro.tmp 3087007744 bytes, 23 block(s), OPENFORWRITE: ................................................................/yunva/log/script/script_user_report/reportscriptresultlog/day=20171016/.22fc73b3-8b89-4308-ba7c-2d301db6966e.avro.tmp 64512 bytes, 1 block(s), OPENFORWRITE:
3.切换到hadoop用户下
su - hadoop
# 列出hdfs中的10行文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hadoop fs -ls /user/hive/warehouse/tbl_push_user_req/ | head -10
Found 82 items
drwxrwxrwt - root supergroup 0 2017-07-28 10:31 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-21
drwxrwxrwt - root supergroup 0 2017-07-28 11:23 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-22
drwxrwxrwt - root supergroup 0 2017-07-28 11:24 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-23
drwxrwxrwt - root supergroup 0 2017-07-28 11:30 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-24
drwxrwxrwt - hadoop supergroup 0 2017-08-08 15:53 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-25
drwxrwxrwt - hadoop supergroup 0 2017-08-08 15:53 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-26
drwxrwxrwt - hadoop supergroup 0 2017-08-08 15:53 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-27
drwxrwxrwt - hadoop supergroup 0 2017-08-08 15:53 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-28
drwxrwxrwt - hadoop supergroup 0 2017-08-08 15:53 /user/hive/warehouse/tbl_push_user_req/dt=2017-07-29
对比正常的数据和不正常的数据：
# 列出某个具体文件的10行内容
有问题的文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hadoop fs -cat /user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/061f1b3e-c35d-4a7e-9b86-4a23bb748a5f.avro | head -10
cat: Cannot obtain block length for LocatedBlock{BP-1799054461-10.19.128.248-1498641674986:blk_1073900198_194650; getBlockSize()=11741696; corrupt=false; offset=671088640; locs=[DatanodeInfoWithStorage[10.19.91.236:50010,DS-d6edd2d2-8efc-43d4-b9ec-3e96ae5877e3,DISK], DatanodeInfoWithStorage[10.19.196.141:50010,DS-8363259d-7e1b-451b-9b1d-94352d6c5733,DISK], DatanodeInfoWithStorage[10.19.80.123:50010,DS-4e7d89be-8694-4039-aa30-60e16f3290e1,DISK]]}
没问题的文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hadoop fs -cat /user/hive/warehouse/tbl_push_user_app/dt=2017-10-12/67e87e3f-aa47-48cf-8c98-daf5e25ac1d8.avro | head -10
Objavro.schema
"type":"record","name":"UserAppLog","namespace":"com.yunva.hadoop.kafka.hdfs.push.model","fields":[{"name":"yunvaId","type":["null","string"],"default":null},{"name":"osType","type":["null","string"],"default":null},{"name":"root","type":["null","boolean"],"default":null},{"name":"time","type":["null","string"],"default":null},{"name":"apps","type":["null",{"type":"array","items":{"type":"record","name":"App","fields":[{"name":"n","type":["null","string"],"default":null},{"name":"v","type":["null","string"],"default":null}]},"java-class":"java.util.List"}],"default":null},{"name":"esPath","type":["null","string"],"default":null},{"name":"tag","type":["null","string"],"default":null},{"name":"date","type":["null","long"],"default":null}]}avro.codec
snappy"Ȉ´.iT.cc6899014da04fed9130a5be158268971:2017-10-12 03:00:00.284.+0800$com.t<tmgp.gbits.atmU09050(c?|mobileqq692push:user:app/// ¸ԋߡW@2f06975e80c94a99a512f90d40d39f5595.
具体解决办法：
找到前台通过hue查询Hive关于push_user_app dt=2017-10-13报的错
定位到具体文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hadoop fs -ls /user/hive/warehouse/tbl_push_user_app/dt=2017-10-13
Found 2 items
-rwxrwxrwt 3 root supergroup 437425664 2017-10-13 03:00 /user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro
-rwxrwxrwt 3 root supergroup 516130304 2017-10-13 17:12 /user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/c8286332-d0e7-4243-bca1-eb3cd92dcf1a.avro
# 通过cat命令找出有问题的文件，以下特征即为有问题的文件
[hadoop@uhadoop-bwgkeu-master1 ~]$ hadoop fs -cat /user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro | head -10
cat: Cannot obtain block length for LocatedBlock{BP-1799054461-10.19.128.248-1498641674986:blk_1073900850_195992; getBlockSize()=34772480; corrupt=false; offset=402653184; locs=[DatanodeInfoWithStorage[10.19.196.141:50010,DS-8363259d-7e1b-451b-9b1d-94352d6c5733,DISK], DatanodeInfoWithStorage[10.19.185.160:50010,DS-d5dd2c3f-d410-404a-be07-bdaf5e31cb5f,DISK], DatanodeInfoWithStorage[10.19.91.236:50010,DS-d6edd2d2-8efc-43d4-b9ec-3e96ae5877e3,DISK]]}
# 对有问题的文件进行修复
[hadoop@uhadoop-bwgkeu-master1 ~]$ hdfs debug recoverLease -path /user/hive/warehouse/tbl_push_user_app/dt=2017-10-13/5cdcd93f-4a74-480b-b603-491e089d6a69.avro -retries 3
# 验证
通过前台查询tbl_push_user_app这张hive的表，可以看到可以正常查询了
select * from tbl_push_user_app where dt='2017-10-13' limit 10
