ETL工具之kettle的使用(一) MongoDB->Mysql
2019-06-11 15:18 握瑾怀瑜 阅读(13426) 评论(0) 编辑 收藏 举报kettle简介:
Kettle是一款国外开源的ETL工具,纯java编写,可以在Window、Linux、Unix上运行,数据抽取高效稳定。
Kettle 中文名称叫水壶,该项目的主程序员MATT 希望把各种数据放到一个壶里,然后以一种指定的格式流出。
Kettle这个ETL工具集,它允许你管理来自不同数据库的数据,通过提供一个图形化的用户环境来描述你想做什么,而不是你想怎么做。
Kettle中有两种脚本文件,transformation和job,transformation完成针对数据的基础转换,job则完成整个工作流的控制
1:Kettle的下载与安装
下载地址 : https://community.hds.com/docs/DOC-1009855
1) 下载下来是压缩版 解压就可以使用 解压后双击 Spoon.bat 等几秒 即出现图形界面工具
2:从MongoDB数据库的otherData文档 中 读取数据到 Mysql数据库t_other_data表 中
1)在kettle中,mongodb input组件。
2) 编辑组件
3) 输入连接信息
4) 写查询语句
点击预览 可以预览成功说明配置无误
5) MongoDB->Mysql中MongoDB的配置就ok了 接下来配置表输出 即 mysql的配置
双击 表输出
6) 填写数据库连接信息
7) 点击测试 看mysql配置的是否无误
看到successfully说明 mysql配置无误 (这里最初我配置的时候报错 报错信息后面单拎出来记录)
8) 配置Mysql
9) 点击运行
10) 查看Mysql中t_other_data表中数据
3:第一次在配置好Mysql后 点击测试 Mysql是报错的
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
org.pentaho.di.core.exception.KettleDatabaseException: Error occurred while trying to connect to the database Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed. org.gjt.mm.mysql.Driver at org.pentaho.di.core.database.Database.normalConnect(Database.java:472) at org.pentaho.di.core.database.Database.connect(Database.java:370) at org.pentaho.di.core.database.Database.connect(Database.java:341) at org.pentaho.di.core.database.Database.connect(Database.java:331) at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.createDatabaseNodes(XulDatabaseExplorerController.java:380) at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.init(XulDatabaseExplorerController.java:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313) at org.pentaho.ui.xul.impl.AbstractXulDomContainer.initialize(AbstractXulDomContainer.java:141) at org.pentaho.ui.xul.swt.SwtXulRunner$1.run(SwtXulRunner.java:67) at org.eclipse.swt.widgets.Synchronizer.syncExec(Unknown Source) at org.eclipse.swt.widgets.Display.syncExec(Unknown Source) at org.pentaho.ui.xul.swt.SwtXulRunner.initialize(SwtXulRunner.java:64) at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerDialog.open(XulDatabaseExplorerDialog.java:92) at org.pentaho.di.ui.core.database.dialog.DataOverrideHandler.explore(DataOverrideHandler.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157) at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141) at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43) at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.eclipse.jface.window.Window.runEventLoop(Window.java:820) at org.eclipse.jface.window.Window.open(Window.java:796) at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80) at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47) at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:116) at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:61) at org.pentaho.di.ui.trans.step.BaseStepDialog.showDbDialogUnlessCancelledOrValid(BaseStepDialog.java:808) at org.pentaho.di.ui.trans.step.BaseStepDialog$EditConnectionListener.widgetSelected(BaseStepDialog.java:1515) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.trans.steps.tableoutput.TableOutputDialog.open(TableOutputDialog.java:888) at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120) at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8662) at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3293) at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785) at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source) at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source) at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source) at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source) at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1381) at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7817) at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9179) at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:707) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92) Caused by: org.pentaho.di.core.exception.KettleDatabaseException: Driver class 'org.gjt.mm.mysql.Driver' could not be found, make sure the 'MySQL' driver (jar file) is installed. org.gjt.mm.mysql.Driver at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:515) at org.pentaho.di.core.database.Database.normalConnect(Database.java:456) ... 66 more Caused by: java.lang.ClassNotFoundException: org.gjt.mm.mysql.Driver at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:490) ... 67 more
错误信息显示是jar包未安装
下载mysql-connector-java-5.0.7 jar包并放到lib目录下 关闭图形界面重新打开Spoon.bat即可
(有个小坑 : 我第一次下载的版本是8依然报错,随后版本换成5好用了 )
下载jar包地址: http://mvnrepository.com/artifact/mysql/mysql-connector-java
思考:MongoDB的数据读取到Mysql中是手动处理的 这在生产环境合理吗?如何自动实时的读取数据呢?下节总结
加油的人生不解释