关于hadoop登陆kerberos时设置环境变量问题的思考

　　中心思想，设置kerberos环境变量时，发现JDK源码当中的一个问题，故描述如下。

　　在平时的使用中，如果hadoop集群配置kerberos认证的话，使用java访问hdfs或者hive时，需要先进行认证登陆，之后才可以访问。登陆代码大致如下：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

package demo.kerberos;   import java.io.IOException;   import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.security.UserGroupInformation;   public class KerberosLoginTest {       public static void main(String[] args) throws IOException {         // 设置kerberos相关登陆信息         // 方法1：设置krb5.conf配置文件         System.setProperty("java.security.krb5.conf", "/home/eabour/hadoop/conf/krb5.conf");         // 方法2：设置kerberos环境变量         // System.setProperty("java.security.krb5.realm", "HADOOP.COM");         // System.setProperty("java.security.krb5.kdc", "kdc.server.com");         // 开启登陆调试日志         System.setProperty("sun.security.krb5.debug", "true");         Configuration conf = new Configuration();         conf.addResource(new Path("/home/eabour/hadoop/conf/core-site.xml"));         conf.addResource(new Path("/home/eabour/hadoop/conf/hdfs-site.xml"));         UserGroupInformation.setConfiguration(conf);         // 登陆kerberos         UserGroupInformation.loginUserFromKeytab("test@HADOOP.COM", "/home/eabour/test.keytab");         // TODO         // 访问HDFS     }   }

　　设置krb5.conf和core-site.xml、hdfs-site.xml相关大数据平台的配置文件，用来初始化相关配置信息。使用krb5.conf登陆没有问题，不管设置多个kdc server还是单个，jdk中的相关模块都会解析出来，jdk源码位置为openjdk\jdk\src\share\classes\sun\security\krb5，其中openjdk的源码地址为http://hg.openjdk.java.net/jdk7u/jdk7u60/jdk/file/33c1eee28403/src/share/classes，其他版本的类似，解析类为sun.security.krb5.Config，部分代码如下：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

/**  * Private constructor - can not be instantiated externally.  */ private Config() throws KrbException {     /*      * If either one system property is specified, we throw exception.      */     String tmp = getProperty("java.security.krb5.kdc");     if (tmp != null) {         // The user can specify a list of kdc hosts separated by ":"         defaultKDC = tmp.replace(':', ' ');     } else {         defaultKDC = null;     }     defaultRealm = getProperty("java.security.krb5.realm");     if ((defaultKDC == null && defaultRealm != null) ||         (defaultRealm == null && defaultKDC != null)) {         throw new KrbException             ("System property java.security.krb5.kdc and " +              "java.security.krb5.realm both must be set or " +              "neither must be set.");     }       // Always read the Kerberos configuration file     try {         Vector<String> configFile;         String fileName = getJavaFileName();         if (fileName != null) {             configFile = loadConfigFile(fileName);             stanzaTable = parseStanzaTable(configFile);             if (DEBUG) {                 System.out.println("Loaded from Java config");             }         } else {             boolean found = false;             if (isMacosLionOrBetter()) {                 try {                     stanzaTable = SCDynamicStoreConfig.getConfig();                     if (DEBUG) {                         System.out.println("Loaded from SCDynamicStoreConfig");                     }                     found = true;                 } catch (IOException ioe) {                     // OK. Will go on with file                 }             }             if (!found) {                 fileName = getNativeFileName();                 configFile = loadConfigFile(fileName);                 stanzaTable = parseStanzaTable(configFile);                 if (DEBUG) {                     System.out.println("Loaded from native config");                 }             }         }     } catch (IOException ioe) {         // No krb5.conf, no problem. We'll use DNS or system property etc.     } }

　　

　　从代码可以看出，先从环境变量中获取java.security.krb5.kdc和java.security.krb5.realm，然后在读取配置文件java.security.krb5.conf。如果同时设置java.security.krb5.kdc、java.security.krb5.realm和java.security.krb5.conf，在登陆的时候，会使用java.security.krb5.kdc，除非登陆的realm与defaultRealm不一致，从代码中可以看出来。

　　那么，我要说的问题来了，在某些场景下，没有设置java.security.krb5.conf变量，而使用java.security.krb5.kdc、java.security.krb5.realm来登陆。在1个或多个kdc server情况下，这样的设置没有问题。我们知道kdc server的默认端口为88，使用的时UDP协议，当然可以为TCP协议，服务端口也可以修改，这时候大致为：

?

1	kdc=kdc.server.com:88

　　或者

?

1	kdc=kdc.server.com:1088

　　krb5.conf文件配置

?

1 2 3 4

[realms]     HADOOP.COM = {     kdc = kdc.server.com:1088 }

　　此时，设置环境变量：

?

1	System.setProperty("java.security.krb5.kdc", "kdc.server.com:1088");

　　那么，真正的问题就来了，执行登陆的话，会出现如下报错：

Caused by: GSSException: No valid credentials provided (Mechanism level: ICMP Port Unreachable)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
… 76 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
at java.net.DatagramSocket.receive(DatagramSocket.java:812)
at sun.security.krb5.internal.UDPClient.receive(NetClient.java:206)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:411)
at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:364)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.krb5.KdcComm.send(KdcComm.java:348)
at sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253)
at sun.security.krb5.KdcComm.send(KdcComm.java:229)
at sun.security.krb5.KdcComm.send(KdcComm.java:200)
at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:254)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:269)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

 

　　为什么会报错呢？因为在解析环境变量时，解析出问题了，现在，再来看看解析代码：

    /**
     * Private constructor - can not be instantiated externally.
     */
    private Config() throws KrbException {
        /*
         * If either one system property is specified, we throw exception.
         */
        String tmp = getProperty("java.security.krb5.kdc");
        if (tmp != null) {
            // The user can specify a list of kdc hosts separated by ":"
            defaultKDC = tmp.replace(':', ' ');
        } else {
            defaultKDC = null;
        }
        defaultRealm = getProperty("java.security.krb5.realm");
        if ((defaultKDC == null && defaultRealm != null) ||
            (defaultRealm == null && defaultKDC != null)) {
            throw new KrbException
                ("System property java.security.krb5.kdc and " +
                 "java.security.krb5.realm both must be set or " +
                 "neither must be set.");
        }

　　请看红色黄底的代码 defaultKDC = tmp.replace(':', ' ') ，对，就是这句代码的问题，他将kdc.server.com:1088分割为kdc.server.com和1088了，认为时两个kdc server。我的心崩溃呀，为什么要用冒号来分割多个server的配置，如果在使用默认端口的话，这样也没问题，但是，如果kdc修改了端口，这种通过环境变量设置kdc server的方式就没法用了。

　　到最后，原来时jdk源码的问题，正常的途径怕是设置不了了。但是，不是不能修改了，我们可以用反射来修改Config的defaultKDC的值，虽然说defaultKDC为final String ，到那时它是private final String defaultKDC;，所以还是可以修改的。

　　以上就是我分析的问题，在实际项目中是真实遇到过，因为涉及JDK底层代码，所以请大家来参详一下。

 

 

附：

1.kerberos配置文件设置：https://www.ibm.com/support/knowledgecenter/zh/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/tsec_kerb_create_conf.html