hadoop 2.7.7 ERROR datanode.DataNode: BlockSender.sendChunks() exception: java.io.IOException: 你的主机中的软件中止了一个已建立的连接。

最近在测试Hbase在windows上的单机版的功能。

版本:hadoop 2.7.7  hbase 2.0.0

错误:

ERROR datanode.DataNode: BlockSender.sendChunks() exception: java.io.IOException: 你的主机中的软件中止了一个已建立的连接。

  

       功能都是可以正常的跑通,但是在hbase 写到hadoop的过程中,一直报这个错,虽然不影响使用,但是这个异常一直刷屏看着非常的不舒服。各种百度 google,居然很多的给的方法都是修改服务器的语言为英文,其他的各种很多都是互相复制根本没有作用,害的搞了半天都没有搞定。

在翻github的时候,突然想起来可以去 apache的jira上去搜索下,一搜果然有人遇到相同的问题。

      附上链接:https://issues.apache.org/jira/browse/HDFS-12514,按照里面给的path修改对应版本的hadoop代码,然后再重新编译hadoop,编译好了后覆盖hadoop-hdfs-2.7.7.jar,重新启动hadoop hbase 运行了一段时间,下面就是对应的修改内容:

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
index ff81b5aae70..474e28d06f6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
@@ -176,6 +176,18 @@
    */
   private static final long LONG_READ_THRESHOLD_BYTES = 256 * 1024;
 
+  /** Message of exception thrown when client closes hedged read. */
+  private static final String[] IGNORED_CLIENT_EXCEPTION_MESSAGES =
+      new String[]{
+          "Broken pipe",
+          "Connection reset",
+          // WSAECONNRESET
+          "An existing connection was forcibly closed by the remote host",
+          // WSAECONNABORTED
+          "An established connection was aborted by the software in your " +
+              "host machine",
+          "The stream is closed"};
+
   // The number of bytes per checksum here determines the alignment
   // of reads: we always start reading at a checksum chunk boundary,
   // even if the checksum type is NULL. So, choosing too big of a value
@@ -646,8 +658,7 @@ private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
          * coding example. NEVER do it to drive a program logic. NEVER.
          * It was done here because the NIO throws an IOException for EPIPE.
          */
-        String ioem = e.getMessage();
-        if (!ioem.startsWith("Broken pipe") && !ioem.startsWith("Connection reset")) {
+        if (!ignoreExceptionInSendBlock(e)) {
           LOG.error("BlockSender.sendChunks() exception: ", e);
           datanode.getBlockScanner().markSuspectBlock(
               ris.getVolumeRef().getVolume().getStorageID(),
@@ -663,6 +674,16 @@ private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
 
     return dataLen;
   }
+
+  private boolean ignoreExceptionInSendBlock(Exception e) {
+    String ioem = e.getMessage();
+    for (String ignoredMsg : IGNORED_CLIENT_EXCEPTION_MESSAGES) {
+      if (ioem.startsWith(ignoredMsg)) {
+        return true;
+      }
+    }
+    return false;
+  }
   
   /**
    * Read checksum into given buffer

   记录下给需要的同学,有时候网上这些不靠谱挺耽误时间

posted @ 2023-05-20 22:40  liuzx8888  阅读(315)  评论(0编辑  收藏  举报