hive的java.lang.NoSuchMethodError: org.apache.hadoop.util.Time.monotonicNowNanos()问题解决

在Hive的cli中执行show databases;报一下错误

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Time.monotonicNowNanos()J
 at org.apache.hadoop.util.Timer.monotonicNowNanos(Timer.java:59)
 at org.apache.hadoop.util.StopWatch.start(StopWatch.java:63)
 at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:321)
 at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
 at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
 at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
 at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
 at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
 at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

错误分析

自己在HADOOP的配置目录下的hadoop-env.sh中添加了

# Hive
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*

其中HBase的lib下有hadoop-common-2.5.1.jar,我的HADOOP版本是2.8.5.
估计是Hive语句执行时调用的jar文件是hadoop-common-2.5.1.jar中的org.apache.hadoop.util.Time类,不是hadoop2.8.5的org.apache.hadoop.util.Time类.
把hadoop2.8.5 的hadoop-common的jar文件与HBase的hadoop-common-2.5.1.jar分别解压后,在idea中反编译.结果分别如下:

• hadoop-common-2.5.1.jar

//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by Fernflower decompiler)
//
package org.apache.hadoop.util;
import org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@LimitedPrivate({"HDFS", "MapReduce"})
@Unstable
public final class Time {
    public Time() {
    }
    public static long now() {
        return System.currentTimeMillis();
    }
    public static long monotonicNow() {
        long NANOSECONDS_PER_MILLISECOND = 1000000L;
        return System.nanoTime() / 1000000L;
    }
}

• Hadoop2.8.5的hadoop-common jar文件

//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by Fernflower decompiler)
//
package org.apache.hadoop.util;
import java.text.SimpleDateFormat;
import org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@LimitedPrivate({"HDFS", "MapReduce"})
@Unstable
public final class Time {
    private static final long NANOSECONDS_PER_MILLISECOND = 1000000L;
    private static final ThreadLocal<SimpleDateFormat> DATE_FORMAT = new ThreadLocal<SimpleDateFormat>() {
        protected SimpleDateFormat initialValue() {
            return new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSSZ");
        }
    };
    public Time() {
    }
    public static long now() {
        return System.currentTimeMillis();
    }
    public static long monotonicNow() {
        return System.nanoTime() / 1000000L;
    }
    public static long monotonicNowNanos() {
        return System.nanoTime();
    }
    public static String formatTime(long millis) {
        return ((SimpleDateFormat)DATE_FORMAT.get()).format(millis);
    }
}

对比发现hadoop-common-2.5.1.jar的Time中没有monotonicNowNanos方法

解决方法

把自己的hadoop配置目录下的hadoop-env.sh的

# Hive
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*

替换为

# Hive
# export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
# export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*

总结

不要在Classpath里边放有版本冲突的jar包,但是大数据框架之间的依赖关系注定这是不太可能的.只好自己注意这类的问题了.