kettle从入门到精通 第三十六课 kettle carte 集群
1、carte服务可以单体运行也可以集群方式运行,今天我们一起来学习下carte的集群模式部署和使用。本次示例用一个master和两个slave从节点演示。
carte-config-master-8080.xml 配置文件:
1 <slave_config> 2 <!-- 3 Document description... 4 5 - masters: You can list the slave servers to which this slave has to report back to. 6 If this is a master, we will contact the other masters to get a list of all the slaves in the cluster. 7 8 - report_to_masters : send a message to the defined masters to let them know we exist (Y/N) 9 10 - slaveserver : specify the slave server details of this carte instance. 11 IMPORTANT : the username and password specified here are used by the master instances to connect to this slave. 12 13 --> 14 15 <slaveserver> 16 <name>master1</name> 17 <hostname>xxx.xxx.6.89</hostname> 18 <port>8080</port> 19 <master>Y</master> 20 </slaveserver> 21 <repository> 22 <id>KettleDatabaseRepository</id> 23 <name>kettleDb</name> 24 <description>Database repository</description> 25 <is_default>true</is_default> 26 <connection>kettleDb</connection> 27 </repository> 28 29 </slave_config>
carte-config-8081.xml 配置文件:
<slave_config> <!-- Document description... - masters: You can list the slave servers to which this slave has to report back to. If this is a master, we will contact the other masters to get a list of all the slaves in the cluster. - report_to_masters : send a message to the defined masters to let them know we exist (Y/N) - slaveserver : specify the slave server details of this carte instance. IMPORTANT : the username and password specified here are used by the master instances to connect to this slave. --> <masters> <slaveserver> <name>master1</name> <hostname>xxx.xxx.6.89</hostname> <port>8080</port> <username>cluster</username> <password>cluster</password> <master>Y</master> </slaveserver> </masters> <report_to_masters>Y</report_to_masters> <slaveserver> <name>slave1-8081</name> <hostname>xxx.xxx.6.89</hostname> <port>8081</port> <username>cluster</username> <password>cluster</password> <master>N</master> </slaveserver> </slave_config>
carte-config-8082.xml 配置文件同carte-config-8081.xml 配置文件类似,这里省略。以上配置文件在pwd目录下,如下图所示:
2、启动主carte服务
windows:Carte.bat pwd/carte-config-master-8080.xml
linux:sh carte.sh pwd/carte-config-master-8080.xml
3、启动从carte服务
windows:
Carte.bat pwd/carte-config-8081.xml
Carte.bat pwd/carte-config-8082.xml
linux:
sh carte.sh pwd/carte-config-8081.xml
sh carte.sh pwd/carte-config-8082.xml
4、运行Spoon.bat或者spoon.sh 客户端,编辑转换文件,新建子服务器,将三个carte服务添加进来如下所示:
默认用户名和密码都是cluster,也可以自行修改。
5、基于子服务器列表创建kettle集群schemas,选择设置好的子服务器,这里演示采用静态cluster,Dynamic cluster模式支持动态添加子服务器。
6、设置转换文件中步骤的集群模式,右键生成记录步骤,选择自己创建的carte集群即可。
7、集群模式运行转换文件。通过集群提交时,作业或者转换是通过集群Master提交作业,Master和多个Slave采用动态抢占式(排它锁)机制抢占任务,实施任务的运行操作。