Window中调试HBase问题小结

　　1.好久没用log4j了，转到logback好多年了，hbase程序运行时，报缺少log4j配置，那么，就转去logback吧（以下的XXX表示版本号）。

　　原先lib包里面有log4j-XXX.jar、slf4j-api-XXX.jar、slf4j-log4j12-XXX.jar，干掉log4j-XXX.jar和slf4j-log4j12-XXX.jar，加入jcl-over-slf4j-XXX.jar，log4j-over-slf4j-XXX.jar、logback-classic-XXX.jar、logback-core-XXX.jar，至于原先的slf4j-api这个包，可以顺手替换成新的，然后，加入一个logback.xml到classpath下

<?xml version="1.0" encoding="UTF-8"?>

<!-- For assistance related to logback-translator or configuration  -->
<!-- files in general, please contact the logback user mailing list -->
<!-- at http://www.qos.ch/mailman/listinfo/logback-user             -->
<!--                                                                -->
<!-- For professional support please see                            -->
<!--    http://www.qos.ch/shop/products/professionalSupport         -->
<!--                                                                -->
<configuration scan="true" scanPeriod="120 seconds" debug="false">
  <appender name="FileApp" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <!--See also http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> 
      <level>WARN</level> 
    </filter>
    <File>log/HBaseClient.log</File>
    <encoder>
      <pattern>%d [%t] %-5p %c - %m%n</pattern>
    </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <maxIndex>10</maxIndex>
      <FileNamePattern>log/HBaseClient.log.%i</FileNamePattern>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <MaxFileSize>1024KB</MaxFileSize>
    </triggeringPolicy>
  </appender>
  <appender name="ConApp" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%d [%t] %-5p %c - %m%n</pattern>
    </encoder>
  </appender> 
  <root level="ERROR">
      <appender-ref ref="ConApp"/>
    <appender-ref ref="FileApp"/>
  </root>
</configuration>

　　这样，log4j和系统的日志系统就被logback接管了，至于为啥不用log4j，用logback，这个。。。自己去看吧，有空我可能会写点啥，logback有点小bug，就是有的linux系统下会找不到配置文件，以前修复过，不知道新版本有处理掉没。

　　2.Could not locate executable null\bin\winutils.exe in the Hadoop binaries

　　老掉牙的问题了，系统变量设置HADOOP_HOME，我还是不想去设置环境变量，还是一行代码来得快

System.setProperty("hadoop.home.dir", "G:/hadoop/hadoop-2.4.1");

　　3.Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

　　本地库是为了提高性能和包含一些java不支持的实现，比如集群中设置支持gzip lzo压缩后，在对压缩文件进行读取或者对输入文件压缩的时候要使用到hadoop的本地库。

　　找不到本地库，linux下可以自己编译一下，官方文档中也有介绍http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/NativeLibraries.html。

　　windows下。。。。我以前是在https://github.com/srccodes/hadoop-common-2.2.0-bin/tree/master/bin下载的，但是我win7 64位，就用不了，注册dll文件的时候，就报错了。官方文档中写着“The native hadoop library is supported on *nix platforms only.”。。。

　　找到一个windows下编译hadoop本地库的地址http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os

　　看了一遍，想想不划算，算了，到时候肯定是丢去linux系统下跑的，折腾这个干嘛啊，linux下又没有问题。

　　4.程序运行正常，但是当log级别改为Info时，突然发现了这么一条

2014-08-04 16:19:22,942 [main-SendThread(Master:2222)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server Master/192.168.117.128:2222. Will not attempt to authenticate using SASL (java.lang.SecurityException: 无法定位登录配置)

　　看到个Exception，什么情况？觉得哪里不对，但是程序又是一切正常的。

　　搜索了一通，发现好多一样的问题，但是貌似和我的没啥关系，别人是程序跑不通，我是程序正常得很。

　　有说是zookeeper版本不一致的，我看了下hadoop和hbase中的确实有区别，一个zookeeper-3.4.5.jar，一个zookeeper-3.4.6.jar，但是我总觉得不对，新旧怎么可能会不兼容啊。那就直接统一了试试，连客户端程序的都统一了，结果那个info信息还是存在，就和不换是一样的，果然不是这里的问题！

　　有说是配置和hosts的，我这里都没这问题的，都配置过了的。加参数？我在配置中都写好的，直接打印一看就知道有了。

　　这只是一个info信息，很奇怪为啥是个info，照说info级别的都是不影响啥的信息，程序确实是很正常的跑，但是看到这个信息就是不舒服。

　　自己查代码去算了，根据日志，是在ClientCnxn.java中出现的信息，好吧，查看源码

private void logStartConnect(InetSocketAddress addr) {
            String msg = "Opening socket connection to server " + addr;
            if (zooKeeperSaslClient != null) {
              msg += ". " + zooKeeperSaslClient.getConfigStatus();
            }
            LOG.info(msg);
        }

　　汗，还得换去查ZooKeeperSaslClient.java，继续吧

public ZooKeeperSaslClient(final String serverPrincipal)
            throws LoginException {
        /**
         * ZOOKEEPER-1373: allow system property to specify the JAAS
         * configuration section that the zookeeper client should use.
         * Default to "Client".
         */
        String clientSection = System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client");
        // Note that 'Configuration' here refers to javax.security.auth.login.Configuration.
        AppConfigurationEntry entries[] = null;
        RuntimeException runtimeException = null;
        try {
            entries = Configuration.getConfiguration().getAppConfigurationEntry(clientSection);
        } catch (SecurityException e) {
            // handle below: might be harmless if the user doesn't intend to use JAAS authentication.
            runtimeException = e;
        } catch (IllegalArgumentException e) {
            // third party customized getAppConfigurationEntry could throw IllegalArgumentException when JAAS
            // configuration isn't set. We can reevaluate whether to catch RuntimeException instead when more 
            // different types of RuntimeException found
            runtimeException = e;
        }
        if (entries != null) {
            this.configStatus = "Will attempt to SASL-authenticate using Login Context section '" + clientSection + "'";
            this.saslClient = createSaslClient(serverPrincipal, clientSection);
        } else {
            // Handle situation of clientSection's being null: it might simply because the client does not intend to 
            // use SASL, so not necessarily an error.
            saslState = SaslState.FAILED;
            String explicitClientSection = System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY);
            if (explicitClientSection != null) {
                // If the user explicitly overrides the default Login Context, they probably expected SASL to
                // succeed. But if we got here, SASL failed.
                if (runtimeException != null) {
                    throw new LoginException("Zookeeper client cannot authenticate using the " + explicitClientSection +
                            " section of the supplied JAAS configuration: '" +
                            System.getProperty(Environment.JAAS_CONF_KEY) + "' because of a " +
                            "RuntimeException: " + runtimeException);
                } else {
                    throw new LoginException("Client cannot SASL-authenticate because the specified JAAS configuration " +
                            "section '" + explicitClientSection + "' could not be found.");
                }
            } else {
                // The user did not override the default context. It might be that they just don't intend to use SASL,
                // so log at INFO, not WARN, since they don't expect any SASL-related information.
                String msg = "Will not attempt to authenticate using SASL ";
                if (runtimeException != null) {
                    msg += "(" + runtimeException + ")";
                } else {
                    msg += "(unknown error)";
                }
                this.configStatus = msg;
                this.isSASLConfigured = false;
            }
            if (System.getProperty(Environment.JAAS_CONF_KEY)  != null) {
                // Again, the user explicitly set something SASL-related, so they probably expected SASL to succeed.
                if (runtimeException != null) {
                    throw new LoginException("Zookeeper client cannot authenticate using the '" +
                            System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client") +
                            "' section of the supplied JAAS configuration: '" +
                            System.getProperty(Environment.JAAS_CONF_KEY) + "' because of a " +
                            "RuntimeException: " + runtimeException);
                } else {
                    throw new LoginException("No JAAS configuration section named '" +
                            System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client") +
                            "' was found in specified JAAS configuration file: '" +
                            System.getProperty(Environment.JAAS_CONF_KEY) + "'.");
                }
            }
        }
    }

　　根据上面的代码，可知问题在于红色部分，再注意一下上面的注释，我靠，果然只是一个普通的说明文字，但是，尼玛能不能不在info信息里面搞个Exception提示出来？？？

　　好吧，那确实是一个info信息，是无关紧要的info信息，其实只是说明了。。。。。由于客户端没用SASL，所以不使用SASL进行验证。。。

　　能不能换个好点的说明文字？我这些操作都是多余的了！人家就那样显示的，唉。看得不爽也没办法了，难道为了这点东西还去改源码不成？够汗的。　　

　　解决办法，客户端添加SASL信息，然后服务端也改掉，添加验证，这个官方文档http://hbase.apache.org/book.html#security有说明了，直接看去就是了，这里不说明了。

　　唉，处女座追求完美的心，果然是不变的。。。加上，看不到烦心的文字，舒坦了。。