MySQL数据源接入DBus
MySQL数据源接入DBus
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.cannal概述
1>.cannal简介
canal [kə'næl]，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费 早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。 基于日志增量订阅和消费的业务包括 数据库镜像 数据库实时备份 索引构建和实时维护(拆分异构索引、倒排索引等) 业务 cache 刷新 带业务逻辑的增量数据处理 当前的canal支持源端MySQL版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x 重要版本更新说明 canal 1.1.x 版本（release_note）,性能与功能层面有较大的突破,重要提升包括: 整体性能测试&优化,提升了150%. 原生支持prometheus监控 原生支持kafka消息投递 原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析) 原生支持docker镜像 canal 1.1.4版本，迎来最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal动态管理能力，支持配置、任务、日志等在线白屏运维能力。 GitHub地址: https://github.com/alibaba/canal
2>.MySQL主备复制原理概述
如下图所示:
MySQL master 将数据变更写入二进制日志( binary log, 其中记录叫做二进制日志事件binary log events，可以通过 show binlog events 进行查看)
MySQL slave 将 master 的 binary log events 拷贝到它的中继日志(relay log)
MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据
3>.canal 工作原理
canal模拟MySQL slave的交互协议，伪装自己为MySQL slave，向MySQL master发送dump协议 MySQL master收到dump请求，开始推送binary log给MySQL slave(即canal) canal解析binary log对象(原始为 byte 流) 博主推荐阅读: https://github.com/alibaba/canal/blob/master/README.md
二.数据库源端创建心跳库
数据库源端配置只在第一次配置环境时需要，在以后的加表流程中不需要再配置。
此步骤中需要在mysql数据源的mysql_instance实例上创建一个名字为dbus的库，并在此库下创建表db_heartbeat_monitor和db_full_pull_requests两张表，用于心跳检测和全量拉取。
在数据源端新建的dbus库，可以实现无侵入方式接入多种数据源，业务系统无需任何修改，以无侵入性读取数据库系统的日志获得增量数据实时变化。
1>.自行配置MySQL主从复制
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/11816066.html
2>.创建dbus库和dbus用户及相应权限
[root@hdp103.yinzhengjie.org.cn ~]# mysql -uroot -p -S /yinzhengjie/softwares/mysql/data/mysql.sock Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 26 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> CREATE DATABASE dbus; Query OK, 1 row affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> CREATE USER 'dbus'@'%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> GRANT ALL ON dbus.* TO dbus@'%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected, 1 warning (0.01 sec) MySQL [(none)]> MySQL [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]#
[root@hdp101.yinzhengjie.org.cn ~]# mysql -udbus -pyinzhengjie -h hdp103.yinzhengjie.org.cn mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 39 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | dbus | | yinzhengjie_bigdata | +---------------------+ 3 rows in set (0.01 sec) mysql> mysql> QUIT Bye [root@hdp101.yinzhengjie.org.cn ~]# [root@hdp101.yinzhengjie.org.cn ~]#
3>.创建dbus库中需要包含的2张表
use dbus; -- ---------------------------- -- Table structure for db_full_pull_requests -- ---------------------------- DROP TABLE IF EXISTS `db_full_pull_requests`; CREATE TABLE `db_full_pull_requests` ( `seqno` bigint(19) NOT NULL AUTO_INCREMENT, `schema_name` varchar(64) DEFAULT NULL, `table_name` varchar(64) NOT NULL, `physical_tables` varchar(10240) DEFAULT NULL, `scn_no` int(11) DEFAULT NULL, `split_col` varchar(50) DEFAULT NULL, `split_bounding_query` varchar(512) DEFAULT NULL, `pull_target_cols` varchar(512) DEFAULT NULL, `pull_req_create_time` timestamp(6) NOT NULL, `pull_start_time` timestamp(6) NULL DEFAULT NULL, `pull_end_time` timestamp(6) NULL DEFAULT NULL, `pull_status` varchar(16) DEFAULT NULL, `pull_remark` varchar(1024) DEFAULT NULL, PRIMARY KEY (`seqno`) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; SET FOREIGN_KEY_CHECKS=0; -- ---------------------------- -- Table structure for db_heartbeat_monitor -- ---------------------------- DROP TABLE IF EXISTS `db_heartbeat_monitor`; CREATE TABLE `db_heartbeat_monitor` ( `ID` bigint(19) NOT NULL AUTO_INCREMENT COMMENT '心跳表主键', `DS_NAME` varchar(64) NOT NULL COMMENT '数据源名称', `SCHEMA_NAME` varchar(64) NOT NULL COMMENT '用户名', `TABLE_NAME` varchar(64) NOT NULL COMMENT '表名', `PACKET` varchar(256) NOT NULL COMMENT '数据包', `CREATE_TIME` datetime NOT NULL COMMENT '创建时间', `UPDATE_TIME` datetime NOT NULL COMMENT '修改时间', PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
MySQL [dbus]> SHOW TABLES; +-----------------------+ | Tables_in_dbus | +-----------------------+ | db_full_pull_requests | | db_heartbeat_monitor | +-----------------------+ 2 rows in set (0.00 sec) MySQL [dbus]> MySQL [dbus]> DESC db_full_pull_requests; +----------------------+----------------+------+-----+----------------------+--------------------------------+ | Field | Type | Null | Key | Default | Extra | +----------------------+----------------+------+-----+----------------------+--------------------------------+ | seqno | bigint(19) | NO | PRI | NULL | auto_increment | | schema_name | varchar(64) | YES | | NULL | | | table_name | varchar(64) | NO | | NULL | | | physical_tables | varchar(10240) | YES | | NULL | | | scn_no | int(11) | YES | | NULL | | | split_col | varchar(50) | YES | | NULL | | | split_bounding_query | varchar(512) | YES | | NULL | | | pull_target_cols | varchar(512) | YES | | NULL | | | pull_req_create_time | timestamp(6) | NO | | CURRENT_TIMESTAMP(6) | on update CURRENT_TIMESTAMP(6) | | pull_start_time | timestamp(6) | YES | | NULL | | | pull_end_time | timestamp(6) | YES | | NULL | | | pull_status | varchar(16) | YES | | NULL | | | pull_remark | varchar(1024) | YES | | NULL | | +----------------------+----------------+------+-----+----------------------+--------------------------------+ 13 rows in set (0.00 sec) MySQL [dbus]>
MySQL [dbus]> DESC db_heartbeat_monitor; +-------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+----------------+ | ID | bigint(19) | NO | PRI | NULL | auto_increment | | DS_NAME | varchar(64) | NO | | NULL | | | SCHEMA_NAME | varchar(64) | NO | | NULL | | | TABLE_NAME | varchar(64) | NO | | NULL | | | PACKET | varchar(256) | NO | | NULL | | | CREATE_TIME | datetime | NO | | NULL | | | UPDATE_TIME | datetime | NO | | NULL | | +-------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec) MySQL [dbus]>
三.准备业务库表
1>.创建业务库和表(这一步你可以自行定义测试的库和表)
MySQL [(none)]> CREATE DATABASE yinzhengjie_bigdata; Query OK, 1 row affected (0.00 sec) MySQL [(none)]> MySQL [(none)]> USE yinzhengjie_bigdata Database changed MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> CREATE TABLE student(id int,name varchar(50)); Query OK, 0 rows affected (0.06 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> SHOW TABLES; +-------------------------------+ | Tables_in_yinzhengjie_bigdata | +-------------------------------+ | student | +-------------------------------+ 1 row in set (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> DESC student; +-------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+-------------+------+-----+---------+-------+ | id | int(11) | YES | | NULL | | | name | varchar(50) | YES | | NULL | | +-------+-------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> CREATE USER 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> GRANT ALL ON yinzhengjie_bigdata.* TO 'jason'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected, 1 warning (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
[root@hdp103.yinzhengjie.org.cn ~]# mysql -ujason -pyinzhengjie -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 38 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | yinzhengjie_bigdata | +---------------------+ 2 rows in set (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
2>.为dbus用户分配咱们上一步骤创建的业务数据库权限
MySQL [yinzhengjie_bigdata]> GRANT SELECT ON yinzhengjie_bigdata.* TO 'dbus'@'%'; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]>
四.部署cannal
1>.在数据库中创建canal用户
MySQL [yinzhengjie_bigdata]> CREATE USER 'canal'@'172.200.1.%' IDENTIFIED BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> GRANT SELECT,REPLICATION SLAVE,REPLICATION CLIENT ON *.* TO 'canal'@'172.200.1.%'; Query OK, 0 rows affected (0.01 sec) MySQL [yinzhengjie_bigdata]> MySQL [yinzhengjie_bigdata]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MySQL [yinzhengjie_bigdata]>
[root@hdp103.yinzhengjie.org.cn ~]# mysql -ucanal -pyinzhengjie -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 32 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +---------------------+ | Database | +---------------------+ | information_schema | | dbus | | dbusmgr | | mysql | | performance_schema | | sys | | yinzhengjie_bigdata | +---------------------+ 7 rows in set (0.01 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]#
2>.cannel自动化部署
在DBus-keeper页面添加数据线时可以选择自动部署cannal，当然我们也可以一键部署，只需要把dbus官方提供的依赖canal的安装包放到指定目录并解压即可，具体操作如下所示。 [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll total 417572 -rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz drwxr-xr-x 5 root root 222 Mar 13 22:06 dbuskeeper_web -rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# tar -zxf dbus-canal-auto-0.5.0.tar.gz [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll total 417572 drwxr-xr-x 5 500 500 95 Feb 20 2019 dbus-canal-auto-0.5.0 -rw-r--r-- 1 root root 24808996 Mar 4 23:51 dbus-canal-auto-0.5.0.tar.gz drwxr-xr-x 5 root root 222 Mar 13 22:06 dbuskeeper_web -rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0 total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Feb 20 2019 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]#
3>.如上图所示，咱们需要修改conf目录下的canal-auto.properties文件
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0 total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Feb 20 2019 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbus-canal-auto-0.5.0/conf/ total 4 -rw-r--r-- 1 500 500 331 Oct 19 2018 canal-auto.properties [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# vim dbus-canal-auto-0.5.0/conf/canal-auto.properties [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# cat dbus-canal-auto-0.5.0/conf/canal-auto.properties #数据源名称，需要与dbus keeper中添加的一致 dsname=yinzhengjie_bigdata #zk地址,替换成自己的信息 zk.path=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 #canal 用户连接地址。即：要canal去同步的源端库的备库的地址 canal.address=hdp103.yinzhengjie.org.cn:3306 #canal用户名 canal.user=canal #canal密码，替换成自己配置的 canal.pwd=yinzhengjie [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]#
4>.执行deploy.sh脚本
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Mar 14 10:02 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# cat deploy.sh #!/bin/sh #获得当前shell所在路径 basepath=$(cd `dirname $0`; pwd) #echo $basepath echo "****************************************** starting *****************************************" #jvm启动参数 #GC_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/data/dbus-canal-auto/logs/gc/gc.log" LOG_CONF="-Dlogs.base.path=$basepath -Duser.dir=$basepath" OOM_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$basepath/logs/oom" JVM_OPTS="-server -Xmx4096m -Xms100m -XX:NewRatio=1" CLASS_PATH="" MAIN="com.creditease.dbus.canal.auto.AutoDeployStart" if [ "x$1" = "xcheck" ] then MAIN="com.creditease.dbus.canal.auto.AutoCheckStart" fi #导入jar和config进入classpath for i in $basepath/lib/*.jar; do CLASS_PATH=$i:"$CLASS_PATH"; done export CLASS_PATH=.:$CLASS_PATH java $JVM_OPTS $LOG_CONF $OOM_OPTS -classpath $CLASS_PATH $MAIN sleep 1 cd reports filename=`ls -ltr |grep canal_| tail -n 1 | awk '{print $9}'` if [ "x$filename" != "x" ]; then cat $filename echo "report文件： $filename" sleep 1 fi [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# ll total 16 -rwxr-xr-x 1 500 500 651 Feb 20 2019 addLine.sh drwxr-xr-x 6 500 500 69 Feb 20 2019 canal drwxr-xr-x 2 500 500 35 Mar 14 10:02 conf -rwxr-xr-x 1 500 500 654 Feb 20 2019 delLine.sh -rwxr-xr-x 1 500 500 1103 Feb 20 2019 deploy.sh drwxr-xr-x 2 500 500 4096 Feb 20 2019 lib [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh ****************************************** starting ***************************************** ************************************ CANAL CONFIG CHECK BEGIN!******************************* 数据源名称: yinzhengjie_bigdata zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 备库地址: hdp103.yinzhengjie.org.cn:3306 canal 用户名: canal canal 密码: yinzhengjie *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 数据库连接成功... 检查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 2020-03-14 10:38:14,778 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting 2020-03-14 10:38:14,808 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_201 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log 4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux 2020-03-14 10:38:14,810 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_64 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root 2020-03-14 10:38:14,811 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0 2020-03-14 10:38:14,825 INFO - [org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c urator.ConnectionState@25359ed82020-03-14 10:38:14,840 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp102.yinzhengjie.org.cn/172.200.1.102:2181. Will not attempt to authenticate using SASL (unknown error) 2020-03-14 10:38:14,900 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp102.yinzhengjie.org.cn/172.200.1.102:2181, initiating session 2020-03-14 10:38:14,911 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp102.yinzhengjie.org.cn/172.200.1.102:2181, sessionid = 0x270d619e962000a, negotiated timeout = 60000 2020-03-14 10:38:14,915 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal 2020-03-14 10:38:14,977 INFO - [com.creditease.dbus.commons.ZkService.createNode(ZkService.java:159)] 节点创建成功, Path: /DBus/Canal/canal-yinzhengjie_bigdata create zk node: /DBus/Canal/canal-yinzhengjie_bigdata node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** 2020-03-14 10:38:15,007 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x270d619e962000a closed ************************************ CANAL CONFIG CHECK SUCCESS!***************************** **************************************** CANAL DEPLOY BEGIN!********************************* **************************************** COPY CANAL BEGIN *********************************** copy canal files: cp -r canal/. canal-yinzhengjie_bigdata 2020-03-14 10:38:15,013 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x270d619e962000a copy instance files: cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata **************************************** COPY CANAL SUCCESS ********************************* ************************************ EDIT CANAL.PROPERTIES BEGIN **************************** props: canal.port = 10000 props: canal.zkServers = hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata props: canal.destinations = yinzhengjie_bigdata props: canal.auto.scan = false props: canal.instance.filter.query.dcl = true props: canal.instance.filter.query.dml = true props: canal.instance.binlog.format = ROW props: canal.instance.binlog.image = FULL props: #canal.instance.global.spring.xml = classpath:spring/file-instance.xml props: canal.instance.global.spring.xml = classpath:spring/default-instance.xml ********************************** EDIT CANAL.PROPERTIES SUCCESS **************************** ****************************** UPDATE INSTANCE.PROPERTIES BEGIN ***************************** instance file path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties props: canal.instance.master.address = hdp103.yinzhengjie.org.cn:3306 props: canal.instance.dbUsername = canal props: canal.instance.dbPassword = yinzhengjie props: canal.instance.connectionCharset = UTF-8 ***************************** UPDATE INSTANCE.PROPERTIES SUCCESS **************************** starting canal..... exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.sh exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.sh exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log exec: ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}' canal 进程启动成功， pid 80463 请手动检查当前目录下canal.log，和yinzhengjie_bigdata.log中有无错误信息。 ********************************* CANAL DEPLOY SCCESS! ************************************** ************************************ CANAL CONFIG CHECK BEGIN!******************************* 数据源名称: yinzhengjie_bigdata zk地址: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 备库地址: hdp103.yinzhengjie.org.cn:3306 canal 用户名: canal canal 密码: yinzhengjie *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 数据库连接成功... 检查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal create zk node: /DBus/Canal/canal-yinzhengjie_bigdata node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** ************************************ CANAL CONFIG CHECK SUCCESS!***************************** **************************************** CANAL DEPLOY BEGIN!********************************* **************************************** COPY CANAL BEGIN *********************************** copy canal files: cp -r canal/. canal-yinzhengjie_bigdata copy instance files: cp -r /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/example/. /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata **************************************** COPY CANAL SUCCESS ********************************* ************************************ EDIT CANAL.PROPERTIES BEGIN **************************** props: canal.port = 10000 props: canal.zkServers = hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181/DBus/Canal/canal-yinzhengjie_bigdata props: canal.destinations = yinzhengjie_bigdata props: canal.auto.scan = false props: canal.instance.filter.query.dcl = true props: canal.instance.filter.query.dml = true props: canal.instance.binlog.format = ROW props: canal.instance.binlog.image = FULL props: #canal.instance.global.spring.xml = classpath:spring/file-instance.xml props: canal.instance.global.spring.xml = classpath:spring/default-instance.xml ********************************** EDIT CANAL.PROPERTIES SUCCESS **************************** ****************************** UPDATE INSTANCE.PROPERTIES BEGIN ***************************** instance file path /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/conf/yinzhengjie_bigdata/instance.properties props: canal.instance.master.address = hdp103.yinzhengjie.org.cn:3306 props: canal.instance.dbUsername = canal props: canal.instance.dbPassword = yinzhengjie props: canal.instance.connectionCharset = UTF-8 ***************************** UPDATE INSTANCE.PROPERTIES SUCCESS **************************** starting canal..... exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/stop.sh exec: sh /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/startup.sh exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log exec: ps aux | grep "/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin" | grep -v "grep" | awk '{print $2}' canal 进程启动成功， pid 80463 请手动检查当前目录下canal.log，和yinzhengjie_bigdata.log中有无错误信息。 ********************************* CANAL DEPLOY SCCESS! ************************************** report文件： canal_deploy_yinzhengjie_bigdata_20200314103813.txt [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
5>.检查部署是否正常
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]# bash deploy.sh check ****************************************** starting ***************************************** ************************************* CANAL CHECK BEGIN! ************************************ *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 数据库连接成功... 检查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 2020-03-14 10:41:30,246 INFO - [org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:224)] Starting 2020-03-14 10:41:30,261 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2020-03-14 10:41:30,263 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:host.name=hdp103.yinzhengjie.org.cn 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.version=1.8.0_201 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.vendor=Oracle Corporation 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre 2020-03-14 10:41:30,264 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.class.path=.:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/zookeeper-3.4.8.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-log 4j12-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/slf4j-api-1.7.12.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/reflections-0.9.11.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/ojdbc14-10.2.0.2.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/netty-3.7.0.Final.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/mysql-connector-java-5.1.35.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/log4j-1.2.17.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-smart-2.3.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/json-path-2.3.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/jline-0.9.94.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/javassist-3.21.0-GA.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/guava-20.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/fastjson-1.2.29.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/druid-1.0.19.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-commons-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/dbus-canal-auto-0.5.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/db2jcc4-4.23.42.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-recipes-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-framework-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/curator-client-2.8.0.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-lang3-3.6.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/commons-cli-1.3.1.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/asm-5.0.4.jar:/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/lib/accessors-smart-1.2.jar:2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.io.tmpdir=/tmp 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:java.compiler=<NA> 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.name=Linux 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.arch=amd64 2020-03-14 10:41:30,266 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:os.version=3.10.0-957.el7.x86_64 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.name=root 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.home=/root 2020-03-14 10:41:30,267 INFO - [org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client environment:user.dir=/yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0 2020-03-14 10:41:30,270 INFO - [org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:438)] Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.c urator.ConnectionState@25359ed82020-03-14 10:41:30,292 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032)] Opening socket connection to server hdp101.yinzhengjie.org.cn/172.200.1.101:2181. Will not attempt to authenticate using SASL (unknown error) 2020-03-14 10:41:30,373 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:876)] Socket connection established to hdp101.yinzhengjie.org.cn/172.200.1.101:2181, initiating session 2020-03-14 10:41:30,385 INFO - [org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1299)] Session establishment complete on server hdp101.yinzhengjie.org.cn/172.200.1.101:2181, sessionid = 0x170d61a5413000e, negotiated timeout = 60000 2020-03-14 10:41:30,393 INFO - [org.apache.curator.framework.state.ConnectionStateManager.postState(ConnectionStateManager.java:228)] State change: CONNECTED node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** 2020-03-14 10:41:30,452 INFO - [org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)] Session: 0x170d61a5413000e closed 2020-03-14 10:41:30,453 INFO - [org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:519)] EventThread shut down for session: 0x170d61a5413000e exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log **************************************** CANAL CHECK SUCCESS ******************************** ************************************* CANAL CHECK BEGIN! ************************************ *************************** CHECK DATABASE CANAL ACCOUNT BEGIN ***************************** canal user: canal canal pwd: yinzhengjie slave url: jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbus?characterEncoding=utf-8 数据库连接成功... 检查blog format: show variables like '%bin%' binlog_format : ROW ****************************** CHECK DATABASE CANAL ACCOUNT SUCCESS ************************* ********************************** CHECK CANAL ZK NODE BEGIN ******************************** zk str: hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 node exit ,skip zk node: /DBus node exit ,skip zk node: /DBus/Canal node exit ,skip zk node: /DBus/Canal/canal-yinzhengjie_bigdata ******************************** CHECK CANAL ZK NODE SUCCESS ******************************** exec: rm -f canal.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/canal/canal.log canal.log exec: rm -f yinzhengjie_bigdata.log exec: ln -s /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/logs/yinzhengjie_bigdata/yinzhengjie_bigdata.log yinzhengjie_bigdata.log **************************************** CANAL CHECK SUCCESS ******************************** report文件： canal_check_yinzhengjie_bigdata_20200314104129.txt [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0]#
6>.管理canal服务
[root@hdp103.yinzhengjie.org.cn ~]# ll /yinzhengjie/softwares/dbus/dbus-canal-auto-0.5.0/canal-yinzhengjie_bigdata/bin/ total 16 -rw-r--r-- 1 root root 5 Mar 14 16:11 canal.pid -rwxr-xr-x 1 root root 1145 Mar 14 10:38 startup.bat -rwxr-xr-x 1 root root 2956 Mar 14 10:38 startup.sh -rwxr-xr-x 1 root root 1356 Mar 14 10:38 stop.sh [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
五.DBus一键加线
1>.数据线的概念
所谓数据线是一种形象的说法，DBus要想采集某个数据库(数据源)的数据，就得拉一根线连接数据库和DBus，加数据线就是DBus-Keeper页面配置一条数据线，即指定从哪个数据源的哪个库采集哪些表的数据。
Dbus对每个DataSource数据源配置一条数据线，当要添加新的datasource时，需要新添加一条数据线。下面对通过dbus keeper页面添加新数据线的步骤进行介绍
2>.删除自动部署canal的配置
我们采用手工方式部署canal，所以把相关配置删掉，避免加数据线时出现问题，具体操作如下图所示。 依次点击"配置中心","zk管理","DBus","Commons","auto-deploy-canal.conf"(需要鼠标右击),"删除节点"
删除成功后，会有以下的提示信息，建议大家都删除掉哟，否则你尽管下面再不勾选自动部署canal依旧会部署失败，这算是官方的一个bug啦~
3>.管理员身份进入dbus keeper页面，并依次点击"数据源管理","新建数据线"
4>.填写数据源基本信息 （master和slave jdbc连接串信息）
jdbc:MySQL://hdp103.yinzhengjie.org.cn:3306/dbus?useUnicode=true&characterEncoding=utf8&autoReconnect=true&failOverReadOnly=false&noAccessToProcedureBodies=true&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false
5>.下一步添加的schema，勾选要添加的表。Keeper支持一次添加多个schema下的多个table
6>.启动Topology
我们从上往下依次点击启动按钮，点击按钮后当前页面就不可用被选择啦，如下图所示。
第一个dsName启动成功后，它的状态会变成running状态哟~如下图所示。
启动成功后，我们通过Ambari的storm服务菜单可以找到打开storm WebUI的界面，可以看到的确是有Topology启动。
接下来的工作就简单了，按照上述的操作，依次启动三个Topology，启动成功后，状态均变为running状态，如下图所示。
再次查看Storm WebUI，可以看到相应的状态信息。你可以点击每一个任务查看详细哟~
7>.上一步执行成功后，再次回到"数据源管理"，我们看到数据线就加好了，如下图所示。
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。 欢迎加入基础架构自动化运维：598432640，大数据SRE进阶之路：959042252，DevOps进阶之路：526991186
