JDBC注册原理与自定义类加载器解决com.cloudera.hive.jdbc41.HS2Driver的加载【重点】
问题:maven systemPath的jar不被maven assembly插件所打包,导致运行期找不到相应的driver类;而且必须打成一个jar包,不能用maven-jar-plugin + lib形式
解决方案:
将driver HiveJdbc41.jar放于resource,作为普通资源,运行期使用自定义类加载器加载 (使用resource中的jar包资源作为UrlClassloader(二))
为了方便,本文使用mysql的jdbc做测试
0
https://yanbin.blog/custom-classload-dynamic-load-jdbc-driver/#more-8187 http://www.kfu.com/~nsayer/Java/dyn-jdbc.html https://blog.csdn.net/d6619309/article/details/53149384 https://blog.csdn.net/qq_35385196/article/details/81750639 http://www.hackerav.com/?post=44723
上面5个链接试图解决动态使用自定义类加载器加载mysql driver,但我尝试都失败了,我自己想办法
1 通过 https://blog.csdn.net/yangcheng33/article/details/52631940 研究jdbc的类加载器check机制
@CallerSensitive
public static Connection getConnection(String url,
String user, String password) throws SQLException {
java.util.Properties info = new java.util.Properties();
if (user != null) {
info.put("user", user);
}
if (password != null) {
info.put("password", password);
}
return (getConnection(url, info, Reflection.getCallerClass()));
}
private static Connection getConnection(
String url, java.util.Properties info, Class<?> caller) throws SQLException {
/*
* When callerCl is null, we should check the application's
* (which is invoking this class indirectly)
* classloader, so that the JDBC driver class outside rt.jar
* can be loaded from here.
*/
ClassLoader callerCL = caller != null ? caller.getClassLoader() : null;
synchronized(DriverManager.class) {
// synchronize loading of the correct classloader.
if (callerCL == null) {
callerCL = Thread.currentThread().getContextClassLoader();
}
}
if(url == null) {
throw new SQLException("The url cannot be null", "08001");
}
println("DriverManager.getConnection(\"" + url + "\")");
// Walk through the loaded registeredDrivers attempting to make a connection.
// Remember the first exception that gets raised so we can reraise it.
SQLException reason = null;
for(DriverInfo aDriver : registeredDrivers) {
// If the caller does not have permission to load the driver then
// skip it.
if(isDriverAllowed(aDriver.driver, callerCL)) {
try {
println(" trying " + aDriver.driver.getClass().getName());
Connection con = aDriver.driver.connect(url, info);
if (con != null) {
// Success!
println("getConnection returning " + aDriver.driver.getClass().getName());
return (con);
}
} catch (SQLException ex) {
if (reason == null) {
reason = ex;
}
}
} else {
println(" skipping: " + aDriver.getClass().getName());
}
}
// if we got here nobody could connect.
if (reason != null) {
println("getConnection failed: " + reason);
throw reason;
}
println("getConnection: no suitable driver found for "+ url);
throw new SQLException("No suitable driver found for "+ url, "08001");
}
private static boolean isDriverAllowed(Driver driver, ClassLoader classLoader) {
boolean result = false;
if(driver != null) {
Class<?> aClass = null;
try {
aClass = Class.forName(driver.getClass().getName(), true, classLoader); driver由自定义类加载器加载,而传入的是系统类加载器
} catch (Exception ex) {
result = false;
}
result = ( aClass == driver.getClass() ) ? true : false;
}
return result;
}
2 根据1中的分析,问题的关键在于这个:
mysql driver所在类加载器(自定义动态类加载器)与getconnection调用方类的类加载器必须是同一个,故需要一个代理人,突破DriverManger中Reflection.getCallerClass()的防线
这个代理类必须是系统类加载器不可见,而且这个代理类要由加载driver的自定义类加载器来加载,
具体有n种方式:
2.1 使用手动编译代理,放入resource;
缺点:部署麻烦,要先编译代理人,然后将代理人放入resource,然后再编译主项目,两次编译不符合公司要求
2.2 java文件作为文本资源,使用运行期编译;
缺点是运行期编译在各生产环境不可控;https://www.cnblogs.com/chenyf/p/10246154.html
2.3 先手写一个类proxyA,运行期获取proxyA的字节码,
2.3.1 使用asm或javassist copy一个proxyB,并重命名,搞在自定义类加载器的findclass中;
2.3.2 或不重命名,主程序强行findClass绕过系统类加载器中的同名proxyA
2.3.3 loadclass/findclass一个特定的类名,findclass中碰到这个就去copy A,变相偷着实现2.3.1——最终采纳
这样系统类加载器加载A,自定义加载器加载B,反射调用B去getconnection,drivermanager检查的时候仍然会发现driver加载器和调用类加载器都是自定义类加载器包bingo
2.4 使用asm或其他方式(java agent 加载器织入——java.lang.instrument包 AOP,使用javassist ,jdk动态代理源码底层(jdk生成字节码及5种字节码生产方式))造一个类;
太流氓,成本大,风险大,需要对字节码结构很了解
最终采用2.3.3
3 代码
src/main/java/lc3
jars
JdbcProxy.java
mysql-connector-java-=5.1.39.jar
Query
MyJdbcLoader
3.1 解压式(tomcat)
package lc3; import lc3.jars.Query; import java.io.*; import java.lang.reflect.Method; import java.net.URL; import java.net.URLClassLoader; import java.sql.*; import static lc3.jars.Query.URL; import static lc3.jars.Query.PASSWORD; import static lc3.jars.Query.USER; /** * https://www.cnblogs.com/silyvin/articles/12166228.html * 2.3.3 3.1 * Created by joyce on 2020/1/9. */ public class MyJdbcLoader_3_1 { public static void main(String [] f) throws Exception { try { /** * 这个在打包后没用,URLClassLoader无法加载jar中jar * ide可以是因为会将jar释放到target目录 */ // URL url = MyJdbcLoader.class.getResource("jars/mysql-connector-java-5.1.39.jar"); InputStream inputStream = MyJdbcLoader_3_1.class.getResourceAsStream("jars/mysql-connector-java-5.1.39.jar"); URL url = copyJar(inputStream, "tmp-mysql-connector-java-5.1.39.jar"); MyClassLoader classLoader = new MyClassLoader(new URL[]{url}); String driverClass = "com.mysql.jdbc.Driver"; Class.forName(driverClass, true, classLoader); /* loaclass 即可,不需要强制findclass,因为父系统类加载器只有lc3.jars.JdbcProxy */ Class proxy = classLoader.loadClass("MyJdbcProxy"); // Class proxy = classLoader.findClass("JdbcProxy"); Method method = proxy.getMethod("getConnection", String.class, String.class, String.class); Connection connection = (Connection)method.invoke(null, URL, USER, PASSWORD); Query.query(connection); } catch (Exception e) { e.printStackTrace(); } } private static class MyClassLoader extends URLClassLoader { public MyClassLoader(java.net.URL[] urls) { super(urls); } @Override protected Class<?> findClass(String name) throws ClassNotFoundException { if(!name.equals("MyJdbcProxy")) return super.findClass(name); try { InputStream inputStream = this.getParent().getResourceAsStream("lc3/jars/JdbcProxy.class"); byte [] tmp = new byte[1024]; ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); int len = -1; int lenTotal = 0; while ((len = inputStream.read(tmp)) != -1) { lenTotal += len; byteArrayOutputStream.write(tmp, 0, len); } byte [] bytes = byteArrayOutputStream.toByteArray(); if(bytes.length != lenTotal) throw new RuntimeException("copy JdbcProxy error"); return defineClass(bytes, 0, bytes.length); } catch (Exception e) { e.printStackTrace(); } return super.findClass(name); } } private static URL copyJar(InputStream inputStream, String name) throws IOException { File exist = new File(name); if(exist.exists()) return new URL("file:" + name); int len = -1; byte [] bytes = new byte[1024]; ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); while ((len = inputStream.read(bytes)) != -1) { byteArrayOutputStream.write(bytes, 0, len); } inputStream.close(); File file = new File(name); OutputStream outputStream = new FileOutputStream(file); outputStream.write(byteArrayOutputStream.toByteArray()); outputStream.close(); URL url = new URL("file:" + name); return url; } }
3.2
package lc3; import lc3.jars.Query; import java.io.*; import java.lang.reflect.Method; import java.net.URL; import java.net.URLClassLoader; import java.sql.Connection; import java.sql.Driver; import java.sql.DriverManager; import java.util.HashMap; import java.util.jar.JarEntry; import java.util.jar.JarInputStream; import java.util.zip.ZipEntry; import static lc3.jars.Query.*; /** * https://www.cnblogs.com/silyvin/articles/12166228.html * 2.3.3 3.1 * Created by joyce on 2020/1/9. */ public class MyJdbcLoader_3_2 { public static void main(String [] f) throws Exception { try { /** * 5.1.39不行 */ // InputStream inputStream = MyJdbcLoader_3_2.class.getResourceAsStream("jars/mysql-connector-java-5.1.39.jar"); // InputStream inputStream = MyJdbcLoader_3_2.class.getResourceAsStream("jars/mysql-connector-java-5.1.39-bin.jar"); InputStream inputStream = MyJdbcLoader_3_2.class.getResourceAsStream("jars/mysql-connector-java-5.1.0-bin.jar"); MyClassLoader classLoader = new MyClassLoader(new JarInputStream[]{new JarInputStream(inputStream)}); String driverClass = "com.mysql.jdbc.Driver"; Class cl = Class.forName(driverClass, true, classLoader); Class proxy = classLoader.loadClass("MyJdbcProxy"); Method method = proxy.getMethod("getConnection", String.class, String.class, String.class); Connection connection = (Connection)method.invoke(null, URL, USER, PASSWORD); Query.query(connection); } catch (Exception e) { e.printStackTrace(); } } private static class MyClassLoader extends ClassLoader { JarInputStream [] list = null; private HashMap<String, byte[]> classes = new HashMap<>(); public MyClassLoader(JarInputStream [] jarInputStream) { this.list = jarInputStream; for(JarInputStream jar : list) { JarEntry entry; try { while ((entry = jar.getNextJarEntry()) != null) { String name = entry.getName(); ByteArrayOutputStream out = new ByteArrayOutputStream(); int len = -1; byte [] tmp = new byte[1024]; while ((len = jar.read(tmp)) != -1) { out.write(tmp, 0, len); } byte[] bytes = out.toByteArray(); classes.put(name, bytes); } } catch (Exception e) { e.printStackTrace(); } } System.out.println("total classes - " + classes.size()); } @Override protected Class<?> findClass(String name) throws ClassNotFoundException { if(!name.equals("MyJdbcProxy")) try { InputStream in = getResourceAsStream(name.replace('.', '/') + ".class"); ByteArrayOutputStream out = new ByteArrayOutputStream(); int len = -1; byte [] tmp = new byte[1024]; while ((len = in.read(tmp)) != -1) { out.write(tmp, 0, len); } byte[] bytes = out.toByteArray(); return defineClass(name, bytes, 0, bytes.length); } catch (Exception e) { e.printStackTrace(); } try { InputStream inputStream = this.getParent().getResourceAsStream("lc3/jars/JdbcProxy.class"); byte [] tmp = new byte[1024]; ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); int len = -1; int lenTotal = 0; while ((len = inputStream.read(tmp)) != -1) { lenTotal += len; byteArrayOutputStream.write(tmp, 0, len); } byte [] bytes = byteArrayOutputStream.toByteArray(); if(bytes.length != lenTotal) throw new RuntimeException("copy JdbcProxy error"); return defineClass(bytes, 0, bytes.length); } catch (Exception e) { e.printStackTrace(); } return super.findClass(name); } @Override public InputStream getResourceAsStream(String name) { System.out.println("getResourceAsStream - " + name); if(classes.containsKey(name)) { return new ByteArrayInputStream(classes.get(name)); } System.out.println("getResourceAsStream - error - " + name); return super.getResourceAsStream(name); } } }
5.1.39 mysql、mysql-bin不成功,5.1.0-bin成功,此法不稳定可见一斑
期间报错:
total classes - 207 getResourceAsStream - com/mysql/jdbc/Driver.class getResourceAsStream - com/mysql/jdbc/NonRegisteringDriver.class driver:com.mysql.jdbc.Driver@2b193f2d:lc3.MyJdbcLoader_3_2$MyClassLoader@266474c2 getResourceAsStream - com/mysql/jdbc/StringUtils.class getResourceAsStream - com/mysql/jdbc/Connection.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties.class getResourceAsStream - com/mysql/jdbc/NotImplemented.class getResourceAsStream - com/mysql/jdbc/PreparedStatement.class getResourceAsStream - com/mysql/jdbc/Statement.class getResourceAsStream - com/mysql/jdbc/ServerPreparedStatement.class getResourceAsStream - com/mysql/jdbc/ResultSet.class getResourceAsStream - com/mysql/jdbc/util/LRUCache.class getResourceAsStream - com/mysql/jdbc/Connection$1.class getResourceAsStream - com/mysql/jdbc/log/Log.class getResourceAsStream - com/mysql/jdbc/log/StandardLogger.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties$BooleanConnectionProperty.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties$ConnectionProperty.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties$MemorySizeConnectionProperty.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties$IntegerConnectionProperty.class getResourceAsStream - com/mysql/jdbc/ConnectionProperties$StringConnectionProperty.class getResourceAsStream - com/mysql/jdbc/log/NullLogger.class getResourceAsStream - com/mysql/jdbc/Constants.class getResourceAsStream - com/mysql/jdbc/Util.class getResourceAsStream - com/mysql/jdbc/JDBC4Connection.class getResourceAsStream - com/mysql/jdbc/exceptions/NotYetImplementedException.class getResourceAsStream - com/mysql/jdbc/JDBC4Connection$1.class getResourceAsStream - com/mysql/jdbc/StandardSocketFactory.class getResourceAsStream - com/mysql/jdbc/SocketFactory.class getResourceAsStream - com/mysql/jdbc/CharsetMapping.class getResourceAsStream - com/mysql/jdbc/VersionedStringProperty.class getResourceAsStream - com/mysql/jdbc/log/LogFactory.class getResourceAsStream - com/mysql/jdbc/MysqlIO.class getResourceAsStream - com/mysql/jdbc/RowData.class getResourceAsStream - com/mysql/jdbc/ConnectionFeatureNotAvailableException.class getResourceAsStream - com/mysql/jdbc/CommunicationsException.class getResourceAsStream - com/mysql/jdbc/StreamingNotifiable.class getResourceAsStream - com/mysql/jdbc/PacketTooBigException.class getResourceAsStream - com/mysql/jdbc/Buffer.class getResourceAsStream - com/mysql/jdbc/MysqlDataTruncation.class getResourceAsStream - com/mysql/jdbc/CompressedInputStream.class getResourceAsStream - com/mysql/jdbc/util/ReadAheadInputStream.class getResourceAsStream - com/mysql/jdbc/Security.class getResourceAsStream - com/mysql/jdbc/Statement$CancelTask.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLTimeoutException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLTransientException.class getResourceAsStream - com/mysql/jdbc/Field.class getResourceAsStream - com/mysql/jdbc/MysqlDefs.class getResourceAsStream - com/mysql/jdbc/RowDataStatic.class getResourceAsStream - com/mysql/jdbc/NotUpdatable.class getResourceAsStream - com/mysql/jdbc/UpdatableResultSet.class getResourceAsStream - com/mysql/jdbc/JDBC4ResultSet.class getResourceAsStream - com/mysql/jdbc/JDBC4UpdatableResultSet.class getResourceAsStream - com/mysql/jdbc/SingleByteCharsetConverter.class getResourceAsStream - com/mysql/jdbc/EscapeProcessor.class getResourceAsStream - com/mysql/jdbc/LicenseConfiguration.class getResourceAsStream - com/mysql/jdbc/DatabaseMetaData.class getResourceAsStream - com/mysql/jdbc/DatabaseMetaData$SingleStringIterator.class getResourceAsStream - com/mysql/jdbc/DatabaseMetaData$IteratorWithCleanup.class getResourceAsStream - com/mysql/jdbc/DatabaseMetaData$ResultSetIterator.class getResourceAsStream - com/mysql/jdbc/DatabaseMetaDataUsingInfoSchema.class getResourceAsStream - com/mysql/jdbc/JDBC4DatabaseMetaData.class getResourceAsStream - com/mysql/jdbc/JDBC4DatabaseMetaDataUsingInfoSchema.class getResourceAsStream - com/mysql/jdbc/JDBC4PreparedStatement.class getResourceAsStream - com/mysql/jdbc/PreparedStatement$ParseInfo.class getResourceAsStream - com/mysql/jdbc/SQLError.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLTransientConnectionException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLNonTransientConnectionException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLNonTransientException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLDataException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLIntegrityConstraintViolationException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLSyntaxErrorException.class getResourceAsStream - com/mysql/jdbc/exceptions/MySQLTransactionRollbackException.class getResourceAsStream - com/mysql/jdbc/exceptions/jdbc4/CommunicationsException.class getResourceAsStream - com/mysql/jdbc/Messages.class getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages.class getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages.class java.lang.NullPointerException at lc3.MyJdbcLoader_3_2$MyClassLoader.findClass(MyJdbcLoader_3_2.java:86) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361) at java.util.ResourceBundle.getBundle(ResourceBundle.java:1082) at com.mysql.jdbc.Messages.<clinit>(Messages.java:54) at com.mysql.jdbc.SQLError.<clinit>(SQLError.java:178) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2918) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1601) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1710) at com.mysql.jdbc.Connection.execSQL(Connection.java:2436) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1402) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1556) at lc3.jars.Query.query(Query.java:26) at lc3.MyJdbcLoader_3_2.main(MyJdbcLoader_3_2.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages.properties java.lang.NullPointerException at lc3.MyJdbcLoader_3_2$MyClassLoader.findClass(MyJdbcLoader_3_2.java:86) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361) at java.util.ResourceBundle.getBundle(ResourceBundle.java:1082) at com.mysql.jdbc.Messages.<clinit>(Messages.java:54) at com.mysql.jdbc.SQLError.<clinit>(SQLError.java:178) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2918) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1601) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1710) at com.mysql.jdbc.Connection.execSQL(Connection.java:2436) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1402) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1556) at lc3.jars.Query.query(Query.java:26) at lc3.MyJdbcLoader_3_2.main(MyJdbcLoader_3_2.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh.class getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh.class getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh.properties getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh.properties getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh_CN.class getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh_CN.class java.lang.NullPointerException at lc3.MyJdbcLoader_3_2$MyClassLoader.findClass(MyJdbcLoader_3_2.java:86) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361) at java.util.ResourceBundle.getBundle(ResourceBundle.java:1082) at com.mysql.jdbc.Messages.<clinit>(Messages.java:54) at com.mysql.jdbc.SQLError.<clinit>(SQLError.java:178) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2918) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1601) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1710) at com.mysql.jdbc.Connection.execSQL(Connection.java:2436) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1402) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1556) at lc3.jars.Query.query(Query.java:26) at lc3.MyJdbcLoader_3_2.main(MyJdbcLoader_3_2.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) java.lang.NullPointerException at lc3.MyJdbcLoader_3_2$MyClassLoader.findClass(MyJdbcLoader_3_2.java:86) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2640) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361) at java.util.ResourceBundle.getBundle(ResourceBundle.java:1082) at com.mysql.jdbc.Messages.<clinit>(Messages.java:54) at com.mysql.jdbc.SQLError.<clinit>(SQLError.java:178) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2918) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1601) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1710) at com.mysql.jdbc.Connection.execSQL(Connection.java:2436) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1402) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1556) at lc3.jars.Query.query(Query.java:26) at lc3.MyJdbcLoader_3_2.main(MyJdbcLoader_3_2.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) java.lang.NullPointerException at lc3.MyJdbcLoader_3_2$MyClassLoader.findClass(MyJdbcLoader_3_2.java:86) getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh_CN.properties getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh_CN.properties getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans.class getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans.class getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans.properties getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans.properties getResourceAsStream - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans_CN.class getResourceAsStream - error - com/mysql/jdbc/LocalizedErrorMessages_zh_Hans_CN.class
3.3
package lc3; import lc3.jars.Query; import java.io.*; import java.lang.reflect.Method; import java.net.*; import java.sql.Connection; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import static lc3.jars.Query.*; /** * https://www.cnblogs.com/silyvin/articles/12166228.html * 2.3.3 3.3 * Created by joyce on 2020/1/9. */ public class MyJdbcLoader_3_3 { public static void main(String [] f) throws Exception { try { List<URL> urls = MyClassLoader.init(new String []{"jars/mysql-connector-java-5.1.39.jar"}); MyClassLoader classLoader = new MyClassLoader(urls.toArray(new java.net.URL [urls.size()])); String driverClass = "com.mysql.jdbc.Driver"; Class.forName(driverClass, true, classLoader); Class proxy = classLoader.loadClass("MyJdbcProxy"); Method method = proxy.getMethod("getConnection", String.class, String.class, String.class); Connection connection = (Connection)method.invoke(null, URL, USER, PASSWORD); Query.query(connection); } catch (Exception e) { e.printStackTrace(); } } private static class MyClassLoader extends URLClassLoader { public static List<URL> init(String [] resourceJars) throws Exception { List<java.net.URL> urls = new ArrayList<>(); Map<String, ByteArrayOutputStream> map = new ConcurrentHashMap<>(); java.net.URL.setURLStreamHandlerFactory(new URLStreamHandlerFactory() { public URLStreamHandler createURLStreamHandler(String urlProtocol) { System.out.println("Someone asked for protocol: " + urlProtocol); if ("myjarprotocol".equalsIgnoreCase(urlProtocol)) { return new URLStreamHandler() { @Override protected URLConnection openConnection(URL url) throws IOException { String key = url.toString().split(":")[1]; return new URLConnection(url) { public void connect() throws IOException {} public InputStream getInputStream() throws IOException { return new ByteArrayInputStream(map.get(key).toByteArray()); } }; } }; } return null; } }); for(String resourceJar : resourceJars) { InputStream in = MyClassLoader.class.getResourceAsStream(resourceJar); int len = -1; byte [] bytes = new byte[1024]; ByteArrayOutputStream jarBytes = new ByteArrayOutputStream(); while ((len = in.read(bytes)) != -1) { jarBytes.write(bytes, 0, len); } map.put(resourceJar, jarBytes); urls.add(new URL("myjarprotocol:" + resourceJar)); } return urls; } public MyClassLoader(java.net.URL[] urls) { super(urls); } @Override protected Class<?> findClass(String name) throws ClassNotFoundException { if(!name.equals("MyJdbcProxy")) return super.findClass(name); try { InputStream inputStream = this.getParent().getResourceAsStream("lc3/jars/JdbcProxy.class"); byte [] tmp = new byte[1024]; ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); int len = -1; int lenTotal = 0; while ((len = inputStream.read(tmp)) != -1) { lenTotal += len; byteArrayOutputStream.write(tmp, 0, len); } byte [] bytes = byteArrayOutputStream.toByteArray(); if(bytes.length != lenTotal) throw new RuntimeException("copy JdbcProxy error"); return defineClass(bytes, 0, bytes.length); } catch (Exception e) { e.printStackTrace(); } return super.findClass(name); } } }
3.4 其实系统类加载器也是可以添加URL的,不需要自定义类加载器,这样的话,2中的问题就不存在了
当然,这样的话,mysql-connector-5.1.39.jar这个包进入系统类加载器,就有可能导致冲突
package lc3; import lc3.jars.JdbcProxy; import lc3.jars.Query; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.lang.reflect.Method; import java.net.*; import java.sql.Connection; import java.sql.DriverManager; import java.util.ArrayList; import java.util.List; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import static lc3.jars.Query.*; /** * https://www.cnblogs.com/silyvin/articles/12166228.html * 3.4 * Created by joyce on 2020/1/9. */ public class MyJdbcLoader_3_4 { public static void main(String [] f) throws Exception { try { List<URL> list = init(new String [] {"jars/mysql-connector-java-5.1.39.jar"}); URLClassLoader systemClassloader = (URLClassLoader) ClassLoader.getSystemClassLoader(); Method systemClassloaderMethod = URLClassLoader.class.getDeclaredMethod("addURL", URL.class); systemClassloaderMethod.setAccessible(true); for(URL url : list) { systemClassloaderMethod.invoke(systemClassloader, url); } /** * 此处直接用系统类加载器的JdbcProxy */ Connection connection = (Connection) JdbcProxy.getConnection(URL, USER, PASSWORD); Query.query(connection); } catch (Exception e) { e.printStackTrace(); } } public static List<URL> init(String [] resourceJars) throws Exception { List<java.net.URL> urls = new ArrayList<>(); Map<String, ByteArrayOutputStream> map = new ConcurrentHashMap<>(); java.net.URL.setURLStreamHandlerFactory(new URLStreamHandlerFactory() { public URLStreamHandler createURLStreamHandler(String urlProtocol) { System.out.println("Someone asked for protocol: " + urlProtocol); if ("myjarprotocol".equalsIgnoreCase(urlProtocol)) { return new URLStreamHandler() { @Override protected URLConnection openConnection(URL url) throws IOException { String key = url.toString().split(":")[1]; return new URLConnection(url) { public void connect() throws IOException {} public InputStream getInputStream() throws IOException { return new ByteArrayInputStream(map.get(key).toByteArray()); } }; } }; } return null; } }); for(String resourceJar : resourceJars) { InputStream in = MyJdbcLoader_3_4.class.getResourceAsStream(resourceJar); int len = -1; byte [] bytes = new byte[1024]; ByteArrayOutputStream jarBytes = new ByteArrayOutputStream(); while ((len = in.read(bytes)) != -1) { jarBytes.write(bytes, 0, len); } map.put(resourceJar, jarBytes); urls.add(new URL("myjarprotocol:" + resourceJar)); } return urls; } }
4 最终采用3.3+2.3.3的方式
原因:
4.1 类隔离,防止冲突,侵入性低,所以不选用3.4方式
4.2 通过伪造URL的方式,仍然可以直接使用JDK的URLClassLoader,够问题,比自己实现一个ClassLoader稳定,比如本文的3.2,就表现出了不稳定
4.3 3.1的方式要将jar写入磁盘,东西都进内存了,还写进磁盘,太多此一举,太low
2021.4.30
第4种打整包插件,urlfactory already set 补充了方式3.3的劣势,以及我们在tomcat项目里面最终没有使用的原因
5 出现了defineClass重复定义错误,我们以3.3为例(hive+javaserver时使用3.3,spring+sybase时切到3.2,因为URL注册与spring boot冲突):
public static void main(String [] f) throws Exception {
try {
List<URL> urls = MyClassLoader.init(new String []{"jars/mysql-connector-java-5.1.39.jar"});
MyClassLoader classLoader = new MyClassLoader(urls.toArray(new java.net.URL [urls.size()]));
String driverClass = "com.mysql.jdbc.Driver";
Class.forName(driverClass, true, classLoader);
Class proxy = classLoader.loadClass("MyJdbcProxy");
classLoader.loadClass("MyJdbcProxy"); 增加
Method method = proxy.getMethod("getConnection", String.class, String.class, String.class);
Connection connection = (Connection)method.invoke(null, URL, USER, PASSWORD);
Query.query(connection);
} catch (Exception e) {
e.printStackTrace();
}
}
报错:
Exception in thread "main" java.lang.LinkageError: loader (instance of lc3/MyJdbcLoader_3_3$MyClassLoader): attempted duplicate class definition for name: "lc3/jars/JdbcProxy"
我们试着跟踪一下代码:
第一次loadclass MyJdbcProxy
MyClassLoader查看缓存(Class<?> c = findLoadedClass(name);)有无名为MyJdbcProxy的类——没有
MyClassLoader交给父加载器,因为MyJdbcProxy父加载器也没有,因为我们改名了lc3.jars.JdbcProxy---->MyJdbcProxy——也没有
MyClassLoader调用findClass,然后defineClass,native方法将类缓存入MyClassLoader
MyClassLoader第二次loadClass MyJdbcProxy,再次跑到了defineClass,导致异常
这里就有一个问题:
为什么第二次load在缓存中没找到?
问题的关键出在defineClass,看错误日志,重复定义的是lc3.jar.JdbcProxy,不是MyJdbcProxy,那么defineClass中定义的字节数组,没有经过ASM、javassist、或jdk修改,定义进去得仍然是lc3.jar.JdbcProxy,往上面第5行加粗的缓存进MyClassLoder的也是这个类名而不是MyJdbcProxy,这就解释了为什么第二次load没有找到缓存的字节码,而重新又去findClass、defineClass
那么我们能不能defineClass时指定MyJdbcProxy为类名?答案是不行的,因为字节码没修改过
这个情况与 使用resource中的jar包资源作为UrlClassloader(二) 中 2 不同,该文中,findClass只会调用一次(loadClass的name和defineClass字节码中的类名相同),而在这个案例中,对MyJdbcProxy的loadClass将每次都调用findClass,我们需要自己做缓存
问题找到了,解决方案为,第二次findClass开始,不重新defineClass,将第一次defineClass的返回Class对象缓存起来,第二次直接返回:
package lc3; public class MyJdbcLoader_3_3 { public static void main(String [] f) throws Exception { try { List<URL> urls = MyClassLoader.init(new String []{"jars/mysql-connector-java-5.1.39.jar"}); MyClassLoader classLoader = new MyClassLoader(urls.toArray(new java.net.URL [urls.size()])); String driverClass = "com.mysql.jdbc.Driver"; Class.forName(driverClass, true, classLoader); Class proxy = classLoader.loadClass("MyJdbcProxy"); /** * 此处原先将导致attempted duplicate class definition for name: "lc3/jars/JdbcProxy" * 出于偷懒,只有3.3修改了代码,增加了缓存 */ classLoader.loadClass("MyJdbcProxy"); Method method = proxy.getMethod("getConnection", String.class, String.class, String.class); Connection connection = (Connection)method.invoke(null, URL, USER, PASSWORD); Query.query(connection); } catch (Exception e) { e.printStackTrace(); } } private static class MyClassLoader extends URLClassLoader { private volatile Class aClass = null; .......... @Override protected Class<?> findClass(String name) throws ClassNotFoundException { if(!name.equals("MyJdbcProxy")) return super.findClass(name); if(name.equals("MyJdbcProxy") && aClass != null) return aClass; synchronized (MyClassLoader.class) { if(aClass != null) return aClass; try { InputStream inputStream = this.getParent().getResourceAsStream("lc3/jars/JdbcProxy.class"); ......... aClass = defineClass(bytes, 0, bytes.length); return aClass; } catch (Exception e) { e.printStackTrace(); } } return super.findClass(name); } } }
借用单例模式的volatile+doublecheck搞定多线程环境下的loadClass,虽然可能不必要
在此,我们回顾下:自定义类加载器 与 热加载中的情况,
public static void main(String[] args) throws MalformedURLException, ClassNotFoundException, IllegalAccessException, InstantiationException {
// System.out.println(System.getProperty("java.class.path"));
// System.out.println(System.getProperty("user.dir"));
String dir = "file:/Users/sunyuming/Documents/tool/jars//MySub-1.0.0-jar-with-dependencies.jar";
URL url = new URL(dir);
URL[] urls2 = {url};
// 若不指定parent参数,则默认由系统类加载器担任自定义类加载器的父加载器,输出parent:sun.misc.Launcher$AppClassLoader@3764951d
MyUrlClassLoader myUrlClassLoader = new MyUrlClassLoader(urls2);
System.out.println("parent:--" + myUrlClassLoader.getParent());
// 由于A在父加载器(系统类加载器)classpath下,根据双亲委派,优先由父加载器加载,输出A:sun.misc.Launcher$AppClassLoader@3764951d
Class CA = myUrlClassLoader.loadClass("lc.A");
System.out.println("A:--" + CA.getClassLoader());
CA.newInstance();
// 打破双亲委派机制,直接使用findclass绕开从父加载器寻找并由父加载器加载这一步,输出A:lc.Main$MyUrlClassLoader@5acf9800
// 当类加载请求到来时,先从缓存中查找该类对象,如果存在直接返回,如果不存在则交给该类加载去的父加载器去加载,倘若没有父加载则交给顶级启动类加载器去加载,最后倘若仍没有找到,则使用findClass()方法去加载
CA = myUrlClassLoader.findClass("lc.A");
/**
* https://www.cnblogs.com/silyvin/articles/12166228.html
* 此处会导致attempted duplicate class definition for name: "lc/A"
*/
// CA = myUrlClassLoader.findClass("lc.A");
System.out.println("A:--" + CA.getClassLoader());
// 实例化A,此时触发A的static代码块和B的加载,输出A:lc.Main$MyUrlClassLoader@5acf9800
// 由于B在父加载器classpath下,优先由系统类加载器加载,输出B:sun.misc.Launcher$AppClassLoader@3764951d
CA.newInstance();
}
6 遵照使用resource中的jar包资源作为UrlClassloader(二)对3.2、3.3、3.4的方式堆中的map在defineClass及return URL后remove,节约堆内存