宜信数据采集平台DBus生产环境部署实战案例
宜信数据采集平台DBus生产环境部署实战案例
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.准备工作
1>.MySQL数据库安装
DBus的数据库我们可以和Ambari Server公用同一个MySQL数据库。因此关于MySQL的安装部署我这里就不罗嗦啦，毕竟以前的笔记已经整理过。 博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12219108.html
2>.创建dbusmgr库
[root@hdp101.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/my.cnf [mysqld] default_authentication_plugin=mysql_native_password validate_password.policy=LOW datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [root@hdp101.yinzhengjie.org.cn ~]# [root@hdp101.yinzhengjie.org.cn ~]#
mysql> CREATE DATABASE dbusmgr DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected, 2 warnings (0.01 sec) mysql>
3>.创建dbusmgr库的授权用户
mysql> CREATE USER 'dbusmgr'@'%' IDENTIFIED by 'dbusmgr%123'; Query OK, 0 rows affected (0.00 sec) mysql> mysql> GRANT ALL PRIVILEGES ON dbusmgr.* to 'dbusmgr'@'%'; Query OK, 0 rows affected (0.01 sec) mysql>
[root@hdp101.yinzhengjie.org.cn ~]# mysql -udbusmgr -pdbusmgr%123 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 26 Server version: 8.0.19 MySQL Community Server - GPL Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | dbusmgr | | information_schema | +--------------------+ 2 rows in set (0.00 sec) mysql> mysql>
[root@hdp103.yinzhengjie.org.cn ~]# mysql -udbusmgr -pdbusmgr%123 -h hdp101.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 24 Server version: 8.0.19 MySQL Community Server - GPL Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | dbusmgr | | information_schema | +--------------------+ 2 rows in set (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
4>.不推荐使用MySQL 8.x系列数据库，我使用后出现在后期初始化时报错，如下图所示。
出现上述错误后，我在"/usr/share/java/"这个目录换了不知道多少版本呀，最后咋试都不好使。甚至我怀疑DBus不在"/usr/share/java/"目录下找驱动。
于是查看DBus下查看其依赖包，我们使用"jar -tf 包名.jar"可以看到它内部嵌套了驱动，如下图所示。难怪我在"/usr/share/java"怎么换驱动都不好使，原来它们在程序中写死了。
找到问题之后，自然出现了解决方案:
方案一:
使用IDE工具重新下载编译DBus的MySQL启动包，将默认的mysql 5.1.46版本换为支持mysql 8.x服务端的驱动包。
方案二：
降低MySQL版本，本篇博客选择第二种方案。
5>.重新安装MySQL 5.7系列的重新指向上述的授权语句即可，如下图所示
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/10371899.html
[root@hdp103.yinzhengjie.org.cn ~]# mysql -udbusmgr -pdbusmgr%123 -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 14 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | dbusmgr | +--------------------+ 2 rows in set (0.00 sec) MySQL [(none)]> MySQL [(none)]> QUIT Bye [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
二.Dbus基础依赖软件安装
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12267370.html
三.DBus安装配置
1>.hdp103.yinzhengjie.org.cn节点到其它2个节点免密登录
此步骤我在搭建HDP时已经部署过啦，这里就不罗嗦啦。如果没有配置免密登录可以自行参考我之前的笔记哟~ 博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12219108.html
2>.下载DBus-Keeper并解压到指定目录
[root@hdp103.yinzhengjie.org.cn ~]# mkdir /yinzhengjie/softwares/dbus && cd /yinzhengjie/softwares/dbus [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# mv ~/dbuskeeper_web.zip ./ [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll total 393344 -rw-r--r-- 1 root root 402780844 Mar 4 23:56 dbuskeeper_web.zip #该文件是参考官方文档编译的包已经整理了一系列已经配置好的配置文件 [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# unzip dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll -h total 385M drwxr-xr-x 4 root root 210 Nov 9 14:41 dbuskeeper_web -rw-r--r-- 1 root root 385M Mar 4 23:56 dbuskeeper_web.zip [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# ll dbuskeeper_web #该目录有很多脚本和测试文件 total 128644 -rw-r--r-- 1 root root 107258754 Nov 9 14:41 base_jars.zip -rw-r--r-- 1 root root 1709366 Mar 11 2019 build.zip drwxr-xr-x 7 root root 85 Nov 5 18:31 conf -rw-r--r-- 1 root root 1247 Sep 10 2018 config.properties -rw-r--r-- 1 root root 22731246 Nov 9 14:36 dbus-heartbeat-0.5.0.zip -rw-r--r-- 1 root root 139 Aug 15 2018 init.sh drwxr-xr-x 2 root root 187 Nov 5 18:31 lib -rw-r--r-- 1 root root 1449 Nov 6 16:46 nginx.conf -rw-r--r-- 1 root root 1169 Aug 7 2018 nginx.conf.bak -rw-r--r-- 1 root root 524 Aug 16 2018 start.sh -rw-r--r-- 1 root root 138 Aug 15 2018 stop.sh [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]#
3>.配置nginx
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# \cp dbuskeeper_web/nginx.conf /etc/nginx/ #强制覆盖Nginx的默认配置文件，注意"\cp"前面的反斜线表示取消系统定义的别名。 [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# \cp dbuskeeper_web/build.zip /usr/share/nginx/html/ [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus]# cd /usr/share/nginx/html/ [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# ll total 1680 -rw-r--r-- 1 root root 494 Aug 13 2019 50x.html -rw-r--r-- 1 root root 1709366 Mar 6 21:52 build.zip -rw-r--r-- 1 root root 612 Aug 13 2019 index.html [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# unzip build.zip [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]#
[root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# ll total 1680 -rw-r--r-- 1 root root 494 Aug 13 2019 50x.html -rw-r--r-- 1 root root 1709366 Mar 6 21:52 build.zip -rw-r--r-- 1 root root 612 Aug 13 2019 index.html [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# unzip build.zip Archive: build.zip creating: build/ inflating: build/.htaccess inflating: build/0.b9dfddf96ba25b5c1910.chunk.js inflating: build/0cb39992db243479f5383d10b75482ea.svg extracting: build/0da0ea7c18ef6b517bad8cd4caec14c4.png inflating: build/1.f1400da9009eb812032d.chunk.js inflating: build/15b6a1a4b2195bc97473d4e17dc6fe18.svg inflating: build/2.143ce019527409ddb59b.chunk.js extracting: build/3a1271766ae5a70e545c8421ed69cf55.png inflating: build/524781dfef675a4af82dd14fd6f9e12f.woff extracting: build/62023286590235f8307be7e7209940bd.png inflating: build/69e87d0a5254aa657040912d4be3c163.svg inflating: build/74e0ad871ad3fa7b0d5d83722510e462.jpg inflating: build/8d5a92dc5024ff8c3565845e334f5551.jpg inflating: build/a40b6b285526376dbeb09b6cb766b932.ttf inflating: build/a65d90ff4afaab0d78f7819c054e101f.png extracting: build/b12c984ac1d1557ae5624953da3c0275.png extracting: build/b9afc4b8c86520d2f4731e5fc6801299.png inflating: build/c622d3cb392c5551f8b2b8dbba62e2eb.jpg inflating: build/c94f2d118c440cb2d31eb1d408b71a8b.jpg inflating: build/ce2084a5f5fe7b5a429b38735adb5af0.jpg inflating: build/da59b3e69b878c9490fb7d7bc65761b0.eot extracting: build/e7106ab41fdbe98a89b66adf7c13df9b.png extracting: build/ec0e387d0657e14ae0e2349e6742455b.png inflating: build/ef91aeb50371e9237933e921e5b8bc98.jpg inflating: build/favicon.ico extracting: build/fd176838e002c471fe649f7f768a8133.png inflating: build/index.html inflating: build/main.774c2bb35937c89f4d50.js inflating: build/manifest.json inflating: build/sw.js [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# ll total 1684 -rw-r--r-- 1 root root 494 Aug 13 2019 50x.html drwxr-xr-x 2 root root 4096 Mar 11 2019 build -rw-r--r-- 1 root root 1709366 Mar 6 21:52 build.zip -rw-r--r-- 1 root root 612 Aug 13 2019 index.html [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]# ll build total 5308 -rw-r--r-- 1 root root 41795 Mar 11 2019 0.b9dfddf96ba25b5c1910.chunk.js -rw-r--r-- 1 root root 561 Mar 11 2019 0cb39992db243479f5383d10b75482ea.svg -rw-r--r-- 1 root root 294 Mar 11 2019 0da0ea7c18ef6b517bad8cd4caec14c4.png -rw-r--r-- 1 root root 143734 Mar 11 2019 15b6a1a4b2195bc97473d4e17dc6fe18.svg -rw-r--r-- 1 root root 2320 Mar 11 2019 1.f1400da9009eb812032d.chunk.js -rw-r--r-- 1 root root 26022 Mar 11 2019 2.143ce019527409ddb59b.chunk.js -rw-r--r-- 1 root root 25527 Mar 11 2019 3a1271766ae5a70e545c8421ed69cf55.png -rw-r--r-- 1 root root 46596 Mar 11 2019 524781dfef675a4af82dd14fd6f9e12f.woff -rw-r--r-- 1 root root 17232 Mar 11 2019 62023286590235f8307be7e7209940bd.png -rw-r--r-- 1 root root 526 Mar 11 2019 69e87d0a5254aa657040912d4be3c163.svg -rw-r--r-- 1 root root 130467 Mar 11 2019 74e0ad871ad3fa7b0d5d83722510e462.jpg -rw-r--r-- 1 root root 98553 Mar 11 2019 8d5a92dc5024ff8c3565845e334f5551.jpg -rw-r--r-- 1 root root 86624 Mar 11 2019 a40b6b285526376dbeb09b6cb766b932.ttf -rw-r--r-- 1 root root 4560 Mar 11 2019 a65d90ff4afaab0d78f7819c054e101f.png -rw-r--r-- 1 root root 328 Mar 11 2019 b12c984ac1d1557ae5624953da3c0275.png -rw-r--r-- 1 root root 42844 Mar 11 2019 b9afc4b8c86520d2f4731e5fc6801299.png -rw-r--r-- 1 root root 121644 Mar 11 2019 c622d3cb392c5551f8b2b8dbba62e2eb.jpg -rw-r--r-- 1 root root 34026 Mar 11 2019 c94f2d118c440cb2d31eb1d408b71a8b.jpg -rw-r--r-- 1 root root 147990 Mar 11 2019 ce2084a5f5fe7b5a429b38735adb5af0.jpg -rw-r--r-- 1 root root 86902 Mar 11 2019 da59b3e69b878c9490fb7d7bc65761b0.eot -rw-r--r-- 1 root root 30089 Mar 11 2019 e7106ab41fdbe98a89b66adf7c13df9b.png -rw-r--r-- 1 root root 326 Mar 11 2019 ec0e387d0657e14ae0e2349e6742455b.png -rw-r--r-- 1 root root 19443 Mar 11 2019 ef91aeb50371e9237933e921e5b8bc98.jpg -rw-r--r-- 1 root root 11056 Mar 11 2019 favicon.ico -rw-r--r-- 1 root root 9392 Mar 11 2019 fd176838e002c471fe649f7f768a8133.png -rw-r--r-- 1 root root 1823 Mar 11 2019 index.html -rw-r--r-- 1 root root 4211160 Mar 11 2019 main.774c2bb35937c89f4d50.js -rw-r--r-- 1 root root 619 Mar 11 2019 manifest.json -rw-r--r-- 1 root root 10201 Mar 11 2019 sw.js [root@hdp103.yinzhengjie.org.cn /usr/share/nginx/html]#
[root@hdp103.yinzhengjie.org.cn ~]# ps -ef | grep nginx root 14986 1 0 21:56 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 14988 14986 0 21:56 ? 00:00:00 nginx: worker process root 15259 5495 0 21:57 pts/0 00:00:00 grep --color=auto nginx [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]# systemctl restart nginx.service [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]# ps -ef | grep nginx root 15271 1 0 21:57 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 15272 15271 0 21:57 ? 00:00:00 nginx: worker process root 15276 5495 0 21:57 pts/0 00:00:00 grep --color=auto nginx [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
4>.修改DBus-Keeper的启动配置(下面标记红色的注释部分你需要根据自己的主机情况进行修改哟~)
[root@hdp103.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/dbus/dbuskeeper_web/config.properties # register端口，可以不改 register.server.port=9090 # gateway端口，可以不改 gateway.server.port=5090 # mgr端口，可以不改 mgr.server.port=8901 # service端口，可以不改 service.server.port=10001 # ssh密钥路径 pubKeyPath=~/.ssh/id_rsa # eureka地址，可以不改，需要注意端口号和上面register一直 eureka.client.serviceUrl.defaultZone=http://localhost:9090/eureka/ # eureka预警邮箱 eureka.instance.metadataMap.alarmEmail=example@example.com # ZK地址 zk.str=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 # mysql管理库相关配置，其中mysql驱动可以不改 spring.datasource.driver-class-name=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://hdp103.yinzhengjie.org.cn:3306/dbusmgr?characterEncoding=utf-8 spring.datasource.username=dbusmgr spring.datasource.password=Dbusmgr!@#123 #kafka地址,注意哈，咱们的kafka版本默认端口6667哟~ bootstrap.servers=hdp101.yinzhengjie.org.cn:6667,hdp102.yinzhengjie.org.cn:6667,hdp103.yinzhengjie.org.cn:6667 bootstrap.servers.version=2.0.0 #influxdb地址:influxdb_url_web(外网地址),influxdb_url_dbus(内网地址),不区分内外网相同即可 influxdb_url_web=http://hdp103.yinzhengjie.org.cn:8086 influxdb_url_dbus=http://hdp103.yinzhengjie.org.cn:8086 #nginx地址,注意端口一定要修改为8080哟，这是咱们自定义的配置文件指定的DBus的WebUI端口 nginx.ip=hdp103.yinzhengjie.org.cn nginx.port=8080 [root@hdp103.yinzhengjie.org.cn ~]# [root@hdp103.yinzhengjie.org.cn ~]#
5>.初始化配置
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# chmod u+x init.sh start.sh stop.sh [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# ll total 128644 -rw-r--r-- 1 root root 107258754 Nov 9 14:41 base_jars.zip -rw-r--r-- 1 root root 1709366 Mar 11 2019 build.zip drwxr-xr-x 7 root root 85 Nov 5 18:31 conf -rw-r--r-- 1 root root 1435 Mar 6 22:07 config.properties -rw-r--r-- 1 root root 22731246 Nov 9 14:36 dbus-heartbeat-0.5.0.zip -rwxr--r-- 1 root root 139 Aug 15 2018 init.sh drwxr-xr-x 2 root root 187 Nov 5 18:31 lib -rw-r--r-- 1 root root 1449 Nov 6 16:46 nginx.conf -rw-r--r-- 1 root root 1169 Aug 7 2018 nginx.conf.bak -rwxr--r-- 1 root root 524 Aug 16 2018 start.sh -rwxr--r-- 1 root root 138 Aug 15 2018 stop.sh [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# ./init.sh #初始化完不报错在启动,如有报错需要先解决报错后在启动Dbus-Keeper
6>.启动DBus-Keeper(需要等待30秒左右)
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# ll total 128696 -rw-r--r-- 1 root root 107258754 Nov 9 14:41 base_jars.zip -rw-r--r-- 1 root root 1709366 Mar 11 2019 build.zip drwxr-xr-x 7 root root 85 Nov 5 18:31 conf -rw-r--r-- 1 root root 1433 Mar 13 22:06 config.properties -rw-r--r-- 1 root root 22785814 Mar 13 22:06 dbus-heartbeat-0.5.0.zip -rwxr--r-- 1 root root 139 Aug 15 2018 init.sh drwxr-xr-x 2 root root 226 Mar 13 22:06 lib drwxr-xr-x 2 root root 6 Mar 13 22:06 logs -rw-r--r-- 1 root root 1449 Nov 6 16:46 nginx.conf -rw-r--r-- 1 root root 1169 Aug 7 2018 nginx.conf.bak -rwxr--r-- 1 root root 524 Aug 16 2018 start.sh -rwxr--r-- 1 root root 138 Aug 15 2018 stop.sh [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]# ./start.sh 启动register-server... 启动gateway... 启动keeper-mgr... 启动keeper-service... 验证是否启动成功... 加载config文件... [main] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=hdp103.yinzhengjie.org.cn [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_201 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/yinzhengjie/softwares/jdk1.8.0_201/jre [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=lib/dbus-keeper-auto-deploy-0.5.0-jar-with-dependencies.jar [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-957.el7.x86_64 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/yinzhengjie/softwares/dbus/dbuskeeper_web [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=hdp101.yinzhengjie.org.cn:2181,hdp102.yinzhengjie.org.cn:2181,hdp103.yinzhengjie.org.cn:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2280cdac [main-SendThread(hdp103.yinzhengjie.org.cn:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server hdp103.yinzhengjie.org.cn/172.200.1.103:2181. Will not attempt to authenticate using SASL (unknown error) [main-SendThread(hdp103.yinzhengjie.org.cn:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to hdp103.yinzhengjie.org.cn/172.200.1.103:2181, initiating session [main-SendThread(hdp103.yinzhengjie.org.cn:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server hdp103.yinzhengjie.org.cn/172.200.1.103:2181, sessionid = 0x370d420830b000e, negotiated timeout = 60000 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED DbusKeeper已经初始化 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x370d420830b000e closed [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x370d420830b000e 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动中,请耐心等待... 启动成功！！ [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web]#
7>.访问浏览器的初始化页面
浏览器输入: http://hdp103.yinzhengjie.org.cn:8080/init
8>.查看我们的初始化配置及自定义Grafana信息
9>.配置Storm信息及其它信息后点击确定
9>.初始化成功后自动跳转到登录页面(使用默认的用户名(admin)和密码(12345678)进行登录)
10>.登录成功
11>.检查心跳
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]# ll total 756 -rw-r--r-- 1 root root 125212 Mar 14 07:01 gateway.log -rw-r--r-- 1 root root 334924 Mar 14 07:02 mgr.log -rw-r--r-- 1 root root 63577 Mar 14 07:04 register.log -rw-r--r-- 1 root root 242205 Mar 14 07:02 service.log [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]# [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]# jps 8932 register-server-0.5.0.jar #该程序是注册中心 8933 gateway-0.5.0.jar #该程序是API网关 8934 keeper-mgr-0.5.0.jar #该程序就是Keeper的webUI，如下图所示，如果发现webUI有报错信息，就可以查看其对应的mgr.log日志哟~ 8935 keeper-service-0.5.0.jar #该程序是服务 ...... [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]#
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]# tail mgr.log 2020-03-14 06:59:24.606 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : user:root,host:hdp101.yinzhengjie.org.cn,port:22,keyPath:~/.ssh/id_rsa,command:ps -ef | grep 'com.creditease.dbus.heartbeat.start.Start' | grep -v grep | awk '{print $2}' 2020-03-14 06:59:25.716 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : inputMsg: 2020-03-14 06:59:25.716 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : errorMsg: 2020-03-14 06:59:25.716 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : user:root,host:hdp102.yinzhengjie.org.cn,port:22,keyPath:~/.ssh/id_rsa,command:ps -ef | grep 'com.creditease.dbus.heartbeat.start.Start' | grep -v grep | awk '{print $2}' 2020-03-14 06:59:26.823 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : inputMsg: 2020-03-14 06:59:26.823 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : errorMsg: 2020-03-14 06:59:26.823 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : user:root,host:hdp103.yinzhengjie.org.cn,port:22,keyPath:~/.ssh/id_rsa,command:ps -ef | grep 'com.creditease.dbus.heartbeat.start.Start' | grep -v grep | awk '{print $2}' 2020-03-14 06:59:27.914 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : inputMsg: 2020-03-14 06:59:27.914 INFO 8934 --- [nio-8901-exec-5] com.creditease.dbus.utils.SSHUtils : errorMsg: 2020-03-14 07:02:00.643 INFO 8934 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/logs]#
如上图所示，其它服务检查都正常就唯独主机检查出错啦，这是因为心跳进程脚本没有启动，需要我们手动启动，如下图所示。 [root@hdp103.yinzhengjie.org.cn ~]# cd /app/dbus/heartbeat/dbus-heartbeat-0.5.0/ [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# ll total 8 drwxr-xr-x 2 root root 151 Mar 13 22:06 conf drwxr-xr-x 2 root root 151 Nov 5 22:06 conf_opensource -rwxr-xr-x 1 root root 1153 Nov 7 14:02 heartbeat.sh drwxr-xr-x 2 root root 4096 Mar 13 22:06 lib [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# nohup ./heartbeat.sh > /dev/null 2>&1 & [1] 45368 [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]# jps | grep Start 45368 Start [root@hdp103.yinzhengjie.org.cn /app/dbus/heartbeat/dbus-heartbeat-0.5.0]#
在各个节点启动脚本后，再一次检查集群，如下图所示。所有的服务都是正常的啦~
12>.登录数据库查看已经创建的表
[root@hdp103.yinzhengjie.org.cn ~]# mysql -udbusmgr -pdbusmgr%123 -h hdp103.yinzhengjie.org.cn Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 15 Server version: 5.7.25-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> MySQL [(none)]> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | dbusmgr | +--------------------+ 2 rows in set (0.00 sec) MySQL [(none)]> MySQL [(none)]> USE dbusmgr Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MySQL [dbusmgr]> MySQL [dbusmgr]> MySQL [dbusmgr]> SHOW TABLES; +--------------------------------------------+ | Tables_in_dbusmgr | +--------------------------------------------+ | t_avro_schema | | t_data_schema | | t_dba_encode_columns | | t_dbus_datasource | | t_ddl_event | | t_encode_plugins | | t_fullpull_history | | t_meta_version | | t_plain_log_rule_group | | t_plain_log_rule_group_version | | t_plain_log_rule_type | | t_plain_log_rules | | t_plain_log_rules_version | | t_project | | t_project_encode_hint | | t_project_resource | | t_project_sink | | t_project_topo | | t_project_topo_table | | t_project_topo_table_encode_output_columns | | t_project_topo_table_meta_version | | t_project_user | | t_query_rule_group | | t_sink | | t_table_action | | t_table_meta | | t_user | +--------------------------------------------+ 27 rows in set (0.01 sec) MySQL [dbusmgr]>
13>.手动创建表
MySQL [dbusmgr]> CREATE TABLE `t_data_tables`( -> `id` int(11) unsigned NOT NULL AUTO_INCREMENT, -> `ds_id` int(11) unsigned NOT NULL COMMENT 't_dbus_datasource 表ID', -> `schema_id` int(11) unsigned NOT NULL COMMENT 't_tab_schema 表ID', -> `schema_name` varchar(64) NOT NULL DEFAULT '' COMMENT '表名', -> `table_name` varchar(64) NOT NULL DEFAULT '' COMMENT '表名', -> `table_name_alias` varchar(64) NOT NULL DEFAULT '' COMMENT '别名', -> `physical_table_regex` varchar(96) DEFAULT NULL, -> `output_topic` varchar(96) DEFAULT '' COMMENT 'kafka_topic', -> `ver_id` int(11) unsigned DEFAULT NULL COMMENT '当前使用的meta版本ID', -> `status` varchar(32) NOT NULL DEFAULT 'abort' COMMENT 'ok,abort,inactive,waiting\r\nok:正常使用;abort:需要抛弃该表的数据;waiting:等待拉全量;inactive:不可用 ', -> `create_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '创建时间', -> `meta_change_flg` int(1) DEFAULT '0' COMMENT 'meta变更标识，初始值为：0，表示代表没有发生变更，1：代表meta发生变更。该字段目前mysql appender模块使用。', -> `batch_id` int(11) DEFAULT '0' COMMENT '批次ID，用来标记拉全量的批次，每次拉全量会++，增量只使用该字段并不修改', -> `ver_change_history` varchar(128) DEFAULT NULL, -> `ver_change_notice_flg` int(1) NOT NULL DEFAULT '0', -> `output_before_update_flg` int(1) NOT NULL DEFAULT '0', -> `description` varchar(128) DEFAULT NULL, -> `fullpull_col` varchar(255) DEFAULT '' COMMENT '全量分片列:配置column名称', -> `fullpull_split_shard_size` varchar(255) DEFAULT '' COMMENT '全量分片大小配置:配置-1代表不分片', -> `fullpull_split_style` varchar(255) DEFAULT '' COMMENT '全量分片类型:MD5', -> `is_open` int(1) DEFAULT '0' COMMENT 'mongo是否展开节点,0不展开,1一级展开', -> `is_auto_complete` tinyint(4) DEFAULT '0' COMMENT 'mongoDB的表是否补全数据；如果开启，增量中更新操作会回查并补全数据', -> PRIMARY KEY (`id`), -> UNIQUE KEY `idx_sid_tabname` (`schema_id`,`table_name`) USING BTREE -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected (0.02 sec) MySQL [dbusmgr]>
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/conf/init]# cat dbus_mgr.sql -- ---------------------------- -- Table structure for t_avro_schema -- ---------------------------- DROP TABLE IF EXISTS `t_avro_schema`; CREATE TABLE `t_avro_schema` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `ds_id` int(11) NOT NULL COMMENT 't_dbus_datasource表ID', `namespace` varchar(64) NOT NULL DEFAULT '' COMMENT 'schema名字', `schema_name` varchar(64) NOT NULL DEFAULT '' COMMENT 'schema名字', `full_name` varchar(128) NOT NULL DEFAULT '' COMMENT 'schema名字', `schema_hash` int(11) NOT NULL, `schema_text` text NOT NULL COMMENT 'schema字符串，json', `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), UNIQUE KEY `idx_schema_name` (`full_name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='解析消息的avro schema表'; -- ---------------------------- -- Table structure for t_data_schema -- ---------------------------- DROP TABLE IF EXISTS `t_data_schema`; CREATE TABLE `t_data_schema` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `ds_id` int(11) unsigned NOT NULL COMMENT 't_dbus_datasource 表ID', `schema_name` varchar(64) NOT NULL DEFAULT '' COMMENT 'schema', `status` varchar(32) NOT NULL DEFAULT '' COMMENT '状态 active/inactive', `src_topic` varchar(64) NOT NULL DEFAULT '' COMMENT '源topic', `target_topic` varchar(64) NOT NULL DEFAULT '' COMMENT '目表topic', `create_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '创建时间', `description` varchar(128) DEFAULT NULL COMMENT 'schema描述信息', PRIMARY KEY (`id`), UNIQUE KEY `idx_dsid_sname` (`ds_id`,`schema_name`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_data_tables -- ---------------------------- DROP TABLE IF EXISTS `t_data_tables`; CREATE TABLE `t_data_tables` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `ds_id` int(11) unsigned NOT NULL COMMENT 't_dbus_datasource 表ID', `schema_id` int(11) unsigned NOT NULL COMMENT 't_tab_schema 表ID', `schema_name` varchar(64) DEFAULT NULL, `table_name` varchar(64) NOT NULL DEFAULT '' COMMENT '表名', `table_name_alias` varchar(64) NOT NULL DEFAULT '' COMMENT '别名', `physical_table_regex` varchar(96) DEFAULT NULL, `output_topic` varchar(96) DEFAULT '' COMMENT 'kafka_topic', `ver_id` int(11) unsigned DEFAULT NULL COMMENT '当前使用的meta版本ID', `status` varchar(32) NOT NULL DEFAULT 'abort' COMMENT 'ok,abort,inactive,waiting\r\nok:正常使用;abort:需要抛弃该表的数据;waiting:等待拉全量;inactive:不可用 ', `create_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '创建时间', `meta_change_flg` int(1) DEFAULT '0' COMMENT 'meta变更标识，初始值为：0，表示代表没有发生变更，1：代表meta发生变更。该字段目前mysql appender模块使用。', `batch_id` int(11) DEFAULT '0' COMMENT '批次ID，用来标记拉全量的批次，每次拉全量会++，增量只使用该字段并不修改', `ver_change_history` varchar(128) DEFAULT NULL, `ver_change_notice_flg` int(1) NOT NULL DEFAULT '0', `output_before_update_flg` int(1) NOT NULL DEFAULT '0', `description` varchar(128) DEFAULT NULL, `fullpull_col` varchar(255) DEFAULT '' COMMENT '全量分片列:配置column名称', `fullpull_split_shard_size` varchar(255) DEFAULT '' COMMENT '全量分片大小配置:配置-1代表不分片', `fullpull_split_style` varchar(255) DEFAULT '' COMMENT '全量分片类型:MD5', `is_open` int(1) DEFAULT '0' COMMENT 'mongo是否展开节点,0不展开,1一级展开', `is_auto_complete` tinyint(4) DEFAULT '0' COMMENT 'mongoDB的表是否补全数据；如果开启，增量中更新操作会回查并补全数据', PRIMARY KEY (`id`), UNIQUE KEY `idx_sid_tabname` (`schema_id`,`table_name`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_dba_encode_columns -- ---------------------------- DROP TABLE IF EXISTS `t_dba_encode_columns`; CREATE TABLE `t_dba_encode_columns` ( `id` int(11) NOT NULL AUTO_INCREMENT, `table_id` int(11) DEFAULT NULL COMMENT '表ID', `field_name` varchar(64) DEFAULT NULL COMMENT '字段名称', `plugin_id` int(11) DEFAULT NULL COMMENT '脱敏插件ID', `encode_type` varchar(64) DEFAULT NULL COMMENT '脱敏方式', `encode_param` varchar(4096) DEFAULT NULL COMMENT '脱敏使用的参数', `desc_` varchar(64) DEFAULT NULL COMMENT '描述', `truncate` int(1) DEFAULT '0' COMMENT '1:当字符串类型字段值脱敏后超出源表字段长度时按照源表字段长度截取 0:不做截取操作', `override` tinyint(4) DEFAULT NULL COMMENT '是否覆盖当前脱敏配置(0:不覆盖,1:覆盖)', `update_time` datetime DEFAULT NULL COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_dbus_datasource -- ---------------------------- DROP TABLE IF EXISTS `t_dbus_datasource`; CREATE TABLE `t_dbus_datasource` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `ds_name` varchar(64) NOT NULL DEFAULT '' COMMENT '数据源名字', `ds_type` varchar(32) NOT NULL DEFAULT '' COMMENT '数据源类型oracle/mysql', `instance_name` varchar(32) DEFAULT NULL, `status` varchar(32) NOT NULL DEFAULT '' COMMENT '状态：active/inactive', `ds_desc` varchar(64) NOT NULL COMMENT '数据源描述', `topic` varchar(64) NOT NULL DEFAULT '', `ctrl_topic` varchar(64) NOT NULL DEFAULT '', `schema_topic` varchar(64) NOT NULL, `split_topic` varchar(64) NOT NULL, `master_url` varchar(4000) NOT NULL DEFAULT '' COMMENT '主库jdbc连接串', `slave_url` varchar(4000) NOT NULL DEFAULT '' COMMENT '备库jdbc连接串', `dbus_user` varchar(64) NOT NULL, `dbus_pwd` varchar(64) NOT NULL, `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `ds_partition` varchar(512) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `uidx_ds_name` (`ds_name`), UNIQUE KEY `ds_name` (`ds_name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='dbus数据源配置表'; -- ---------------------------- -- Table structure for t_ddl_event -- ---------------------------- DROP TABLE IF EXISTS `t_ddl_event`; CREATE TABLE `t_ddl_event` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `event_id` int(11) unsigned DEFAULT NULL COMMENT 'oracle源端event表的serno', `ds_id` int(11) NOT NULL, `schema_name` varchar(64) NOT NULL, `table_name` varchar(64) NOT NULL, `column_name` varchar(64) DEFAULT NULL, `ver_id` int(11) unsigned NOT NULL COMMENT '当前表版本id', `trigger_ver` int(11) unsigned DEFAULT NULL COMMENT 'oracle源端event表中version', `ddl_type` varchar(64) NOT NULL, `ddl` varchar(3000) NOT NULL, `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_encode_plugins -- ---------------------------- DROP TABLE IF EXISTS `t_encode_plugins`; CREATE TABLE `t_encode_plugins` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(64) DEFAULT NULL COMMENT '名称', `project_id` int(11) DEFAULT NULL COMMENT '项目ID', `path` varchar(256) DEFAULT NULL COMMENT '存储路径', `encoders` varchar(1024) DEFAULT NULL, `status` varchar(32) DEFAULT NULL COMMENT '状态', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_fullpull_history -- ---------------------------- DROP TABLE IF EXISTS `t_fullpull_history`; CREATE TABLE `t_fullpull_history` ( `id` bigint(20) NOT NULL, `type` varchar(20) NOT NULL COMMENT 'global, indepent, normal', `dsName` varchar(64) NOT NULL, `schemaName` varchar(64) NOT NULL, `tableName` varchar(64) NOT NULL, `version` int(11) DEFAULT NULL, `batch_id` int(11) DEFAULT NULL, `state` varchar(20) DEFAULT NULL COMMENT 'init, splitting, pulling, ending, abort', `error_msg` varchar(8096) DEFAULT NULL, `init_time` datetime DEFAULT NULL, `start_split_time` datetime DEFAULT NULL, `start_pull_time` datetime DEFAULT NULL, `end_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `finished_partition_count` bigint(20) DEFAULT NULL, `total_partition_count` bigint(20) DEFAULT NULL, `finished_row_count` bigint(20) DEFAULT NULL, `total_row_count` bigint(20) DEFAULT NULL, `project_name` varchar(128) DEFAULT NULL, `topology_table_id` int(11) DEFAULT NULL, `target_sink_id` int(11) DEFAULT NULL, `target_sink_topic` varchar(256) DEFAULT NULL, `full_pull_req_msg_offset` bigint(20) DEFAULT NULL, `first_shard_msg_offset` bigint(20) DEFAULT NULL, `last_shard_msg_offset` bigint(20) DEFAULT NULL, `split_column` varchar(64) DEFAULT NULL, `fullpull_condition` varchar(512) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_meta_version -- ---------------------------- DROP TABLE IF EXISTS `t_meta_version`; CREATE TABLE `t_meta_version` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `table_id` int(11) NOT NULL, `ds_id` int(11) unsigned NOT NULL COMMENT 't_dbus_datasource表的ID', `db_name` varchar(64) NOT NULL DEFAULT '' COMMENT '数据库名', `schema_name` varchar(64) NOT NULL DEFAULT '' COMMENT 'schema名', `table_name` varchar(64) NOT NULL DEFAULT '' COMMENT '表名', `version` int(11) NOT NULL COMMENT '版本号', `inner_version` int(11) NOT NULL COMMENT '内部版本号', `event_offset` bigint(20) DEFAULT NULL COMMENT '触发version变更的消息在kafka中的offset值', `event_pos` bigint(20) DEFAULT NULL COMMENT '触发version变更的消息在trail文件中的pos值', `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新日期', `comments` varchar(128) DEFAULT NULL COMMENT '表注释', PRIMARY KEY (`id`), KEY `idx_event_offset` (`event_offset`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='t_table_meta的版本信息'; -- ---------------------------- -- Table structure for t_plain_log_rule_group -- ---------------------------- DROP TABLE IF EXISTS `t_plain_log_rule_group`; CREATE TABLE `t_plain_log_rule_group` ( `id` int(11) NOT NULL AUTO_INCREMENT, `table_id` int(11) NOT NULL, `group_name` varchar(64) NOT NULL, `status` varchar(16) NOT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `json_extract_rule_xml_pack` longtext, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_plain_log_rule_group_version -- ---------------------------- DROP TABLE IF EXISTS `t_plain_log_rule_group_version`; CREATE TABLE `t_plain_log_rule_group_version` ( `id` int(11) NOT NULL AUTO_INCREMENT, `table_id` int(11) NOT NULL, `group_name` varchar(64) NOT NULL, `status` varchar(16) NOT NULL, `ver_id` int(11) NOT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_plain_log_rule_type -- ---------------------------- DROP TABLE IF EXISTS `t_plain_log_rule_type`; CREATE TABLE `t_plain_log_rule_type` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `rule_type_name` varchar(16) NOT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_plain_log_rules -- ---------------------------- DROP TABLE IF EXISTS `t_plain_log_rules`; CREATE TABLE `t_plain_log_rules` ( `id` int(11) NOT NULL AUTO_INCREMENT, `group_id` int(11) NOT NULL, `order_id` int(4) NOT NULL, `rule_type_name` varchar(16) NOT NULL, `rule_grammar` mediumtext NOT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_plain_log_rules_version -- ---------------------------- DROP TABLE IF EXISTS `t_plain_log_rules_version`; CREATE TABLE `t_plain_log_rules_version` ( `id` int(11) NOT NULL AUTO_INCREMENT, `group_id` int(11) NOT NULL, `order_id` int(4) NOT NULL, `rule_type_name` varchar(16) NOT NULL, `rule_grammar` varchar(20480) NOT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project -- ---------------------------- DROP TABLE IF EXISTS `t_project`; CREATE TABLE `t_project` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `project_name` varchar(128) DEFAULT NULL COMMENT '项目名称', `project_display_name` varchar(32) NOT NULL DEFAULT '' COMMENT '可以用中文。长度不超过32个中文字。 要求唯一。仅作显示用', `project_owner` varchar(32) DEFAULT NULL, `project_icon` varchar(128) DEFAULT NULL, `project_desc` varchar(256) DEFAULT NULL COMMENT '项目描述', `project_expire` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '项目有效期', `topology_num` tinyint(4) DEFAULT NULL, `allow_admin_manage` tinyint(4) DEFAULT NULL, `schema_change_notify_flag` tinyint(4) DEFAULT '1' COMMENT '备同步报警通知开关', `schema_change_notify_emails` varchar(1024) DEFAULT NULL, `slave_sync_delay_notify_flag` tinyint(4) DEFAULT '1' COMMENT '主备同步报警通知开关', `slave_sync_delay_notify_emails` varchar(1024) DEFAULT NULL, `fullpull_notify_flag` tinyint(4) DEFAULT '1' COMMENT '拉全量报警通知', `fullpull_notify_emails` varchar(1024) DEFAULT NULL, `data_delay_notify_flag` tinyint(4) DEFAULT NULL, `data_delay_notify_emails` varchar(1024) DEFAULT NULL, `status` varchar(16) DEFAULT 'active' COMMENT '项目状态 active，inactive', `storm_start_path` varchar(1024) DEFAULT NULL COMMENT '启动storm start地址', `storm_ssh_user` varchar(64) DEFAULT NULL COMMENT '启动storm nimbus jar包用户', `storm_api_url` varchar(1024) DEFAULT NULL COMMENT '调用storm api地址', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', `keytab_path` varchar(256) DEFAULT NULL COMMENT 'Kerberos秘钥存储path', `principal` varchar(256) DEFAULT NULL COMMENT 'Kerberos账户', PRIMARY KEY (`id`), UNIQUE KEY `project_display_name` (`project_display_name`) USING BTREE, UNIQUE KEY `project_name` (`project_name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='剩余时间，给前端判断颜色，同时标识是否发送了通知邮件'; -- ---------------------------- -- Table structure for t_project_encode_hint -- ---------------------------- DROP TABLE IF EXISTS `t_project_encode_hint`; CREATE TABLE `t_project_encode_hint` ( `id` int(11) NOT NULL AUTO_INCREMENT, `project_id` int(11) DEFAULT NULL, `table_id` int(11) DEFAULT NULL, `field_name` varchar(64) DEFAULT NULL COMMENT '字段名称', `encode_plugin_id` int(11) DEFAULT NULL COMMENT '脱敏插件ID', `encode_type` varchar(64) DEFAULT NULL COMMENT '脱敏方式', `encode_param` varchar(4096) DEFAULT NULL COMMENT '脱敏使用的参数', `desc_` varchar(64) DEFAULT NULL COMMENT '描述', `truncate` int(1) DEFAULT '0' COMMENT '1:当字符串类型字段值脱敏后超出源表字段长度时按照源表字段长度截取 0:不做截取操作', `update_time` datetime DEFAULT NULL COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_resource -- ---------------------------- DROP TABLE IF EXISTS `t_project_resource`; CREATE TABLE `t_project_resource` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `project_id` int(11) unsigned NOT NULL COMMENT 'project id', `table_id` int(11) unsigned NOT NULL COMMENT 'dbus mgr中的table id', `fullpull_enable_flag` tinyint(3) unsigned NOT NULL DEFAULT '1', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`), KEY `idx_projectid_tableid` (`project_id`,`table_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_sink -- ---------------------------- DROP TABLE IF EXISTS `t_project_sink`; CREATE TABLE `t_project_sink` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `project_id` int(11) NOT NULL, `sink_id` varchar(64) NOT NULL, `update_time` datetime DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_topo -- ---------------------------- DROP TABLE IF EXISTS `t_project_topo`; CREATE TABLE `t_project_topo` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `topo_name` varchar(64) NOT NULL COMMENT 'topo name', `topo_config` varchar(1024) DEFAULT NULL, `jar_version` varchar(32) DEFAULT NULL, `jar_file_path` varchar(256) DEFAULT NULL, `status` varchar(32) DEFAULT NULL COMMENT 'stopped,running,changed(修改了表的配置,比如脱敏)', `topo_comment` varchar(256) DEFAULT NULL, `project_id` int(11) unsigned DEFAULT NULL COMMENT 'project id', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`), KEY `idx_status_toponame` (`status`,`topo_name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_topo_table -- ---------------------------- DROP TABLE IF EXISTS `t_project_topo_table`; CREATE TABLE `t_project_topo_table` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `project_id` int(11) unsigned NOT NULL COMMENT 'project table id', `table_id` int(11) NOT NULL, `topo_id` int(11) unsigned NOT NULL COMMENT 'project id', `status` varchar(32) NOT NULL COMMENT 'new,changed,stopped, running', `output_topic` varchar(256) NOT NULL COMMENT 'output topic', `output_type` varchar(32) DEFAULT NULL COMMENT '输出数据格式: json/ums_1.*', `sink_id` int(11) DEFAULT NULL, `output_list_type` int(1) DEFAULT NULL COMMENT '输出列的列类型：0，贴源输出表的所有列（输出列随源端schema变动而变动；1，指定固定的输出列（任何时候只输出您此时此地选定的列）', `meta_ver` int(11) DEFAULT NULL COMMENT '为指定输出列时才有meta version', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', `schema_change_flag` tinyint(4) DEFAULT '0' COMMENT '表结构变更标志0:未变更,1:变更', `fullpull_col` varchar(255) DEFAULT NULL COMMENT '全量分片列:配置column名称', `fullpull_split_shard_size` varchar(255) DEFAULT NULL COMMENT '全量分片大小配置:配置-1代表不分片', `fullpull_split_style` varchar(255) DEFAULT NULL COMMENT '全量分片类型:MD5', `fullpull_condition` varchar(500) DEFAULT NULL COMMENT '全量拉取条件,例如id>100', PRIMARY KEY (`id`), KEY `idx_projectid_tableid_topoid` (`project_id`,`table_id`,`topo_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_topo_table_encode_output_columns -- ---------------------------- DROP TABLE IF EXISTS `t_project_topo_table_encode_output_columns`; CREATE TABLE `t_project_topo_table_encode_output_columns` ( `id` int(11) NOT NULL AUTO_INCREMENT, `project_topo_table_id` int(11) DEFAULT NULL, `field_name` varchar(64) DEFAULT NULL COMMENT '字段名称', `encode_plugin_id` int(11) DEFAULT NULL COMMENT '脱敏插件ID', `field_type` varchar(64) DEFAULT NULL COMMENT '字段类型', `data_length` bigint(20) DEFAULT NULL COMMENT '数据长度', `encode_type` varchar(64) DEFAULT NULL COMMENT '脱敏方式', `encode_param` varchar(4096) DEFAULT NULL COMMENT '脱敏使用的参数', `desc_` varchar(64) DEFAULT NULL COMMENT '描述', `truncate` int(1) DEFAULT '0' COMMENT '1:当字符串类型字段值脱敏后超出源表字段长度时按照源表字段长度截取 0:不做截取操作', `encode_source` int(1) DEFAULT NULL COMMENT '脱敏规则的来源：0，源端脱敏，最开始的dtables包含的脱敏信息；1，admin脱敏，新增project操作中，新增Resource中配置的脱敏信息；2，自定义脱敏，除前两种列外，用户添加的列的脱敏信息；3，无，表示没有脱敏信息。', `update_time` datetime DEFAULT NULL COMMENT '更新时间', `schema_change_flag` tinyint(4) DEFAULT '0' COMMENT '表结构变更标志0:未变更,1:变更', `data_scale` int(11) DEFAULT '0' COMMENT '小数部分长度', `data_precision` int(11) DEFAULT '0' COMMENT '数据精度', `schema_change_comment` varchar(1024) DEFAULT '' COMMENT '记录变更过程', `special_approve` tinyint(4) NOT NULL DEFAULT '0' COMMENT '是否经过特批(0:未特批,1:特批(特批过的不用脱敏))', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='存储topo表中输出列的信息，如果某些列配置了脱敏信息，则会显示脱敏信息；'; -- ---------------------------- -- Table structure for t_project_topo_table_meta_version -- ---------------------------- DROP TABLE IF EXISTS `t_project_topo_table_meta_version`; CREATE TABLE `t_project_topo_table_meta_version` ( `id` int(11) NOT NULL AUTO_INCREMENT, `project_id` int(11) NOT NULL, `topo_id` int(11) NOT NULL, `table_id` int(11) NOT NULL COMMENT '源table表的id，非t_project_topo_table的id', `version` int(11) DEFAULT NULL, `column_name` varchar(64) DEFAULT NULL, `data_type` varchar(64) DEFAULT NULL, `data_length` bigint(20) DEFAULT NULL COMMENT '数据长度', `update_time` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', `schema_change_flag` tinyint(4) DEFAULT '0' COMMENT '表结构变更标志0:未变更,1:变更', `data_precision` int(11) DEFAULT '0' COMMENT '数据精度', `data_scale` int(11) DEFAULT '0' COMMENT '小数部分长度', `schema_change_comment` varchar(1024) DEFAULT '' COMMENT '记录变更过程', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_project_user -- ---------------------------- DROP TABLE IF EXISTS `t_project_user`; CREATE TABLE `t_project_user` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `project_id` int(11) unsigned NOT NULL, `user_id` int(11) unsigned NOT NULL, `user_level` varchar(32) NOT NULL DEFAULT 'full' COMMENT '用户在项目中的权限：full 或者 readonly', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_query_rule_group -- ---------------------------- DROP TABLE IF EXISTS `t_query_rule_group`; CREATE TABLE `t_query_rule_group` ( `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '表的主键', `table_id` bigint(20) DEFAULT NULL COMMENT 't_dbus_datasource 表的 id', `ver_id` bigint(20) DEFAULT NULL, `status` varchar(255) DEFAULT NULL COMMENT '状态 active 有效 inactive 无效', `description` varchar(256) DEFAULT NULL, `query` varchar(4000) DEFAULT NULL COMMENT '获取数据的 sql 或者其他查询语句', `time_field` varchar(256) DEFAULT NULL COMMENT '生成batch的字段', `init_time` datetime DEFAULT NULL COMMENT '生成 batch 的初始时间', `time_span_ms` bigint(20) DEFAULT NULL COMMENT '生成 batch 的时间间隔', `correction_ms` bigint(20) DEFAULT NULL COMMENT '开始时间修正值', `reserve_time_ms` bigint(20) DEFAULT NULL COMMENT '保留时间毫秒数(系统当前时间 - end_time < reserve_time_ms 的数据不能获取)', `output_style` varchar(255) DEFAULT NULL COMMENT '输出json的格式，flatten：一级扁平展开输出，default：一级展开输出', `save_as_cols` varchar(10240) DEFAULT NULL, `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_query_rule_ver_id` (`ver_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT; -- ---------------------------- -- Table structure for t_sink -- ---------------------------- DROP TABLE IF EXISTS `t_sink`; CREATE TABLE `t_sink` ( `id` int(11) NOT NULL AUTO_INCREMENT, `sink_name` varchar(64) DEFAULT NULL, `sink_desc` varchar(255) DEFAULT NULL, `sink_type` varchar(32) DEFAULT NULL, `url` varchar(512) DEFAULT NULL, `update_time` datetime DEFAULT NULL, `is_global` tinyint(4) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_table_action -- ---------------------------- DROP TABLE IF EXISTS `t_table_action`; CREATE TABLE `t_table_action` ( `id` bigint(20) NOT NULL, `table_id` bigint(20) DEFAULT NULL, `order_id` int(11) DEFAULT NULL, `action_source` varchar(50) DEFAULT NULL, `action_type` varchar(60) DEFAULT NULL, `action_value` varchar(200) DEFAULT NULL, `commit` varchar(100) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_table_meta -- ---------------------------- DROP TABLE IF EXISTS `t_table_meta`; CREATE TABLE `t_table_meta` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `ver_id` int(11) NOT NULL COMMENT '版本id（t_meta_version）', `column_name` varchar(64) NOT NULL DEFAULT '' COMMENT '替换特殊字符后生成的列名，替换规则：replaceAll("[^A-Za-z0-9_]", "_")', `original_column_name` varchar(64) DEFAULT NULL COMMENT '数据库中原始的列名', `column_id` int(4) NOT NULL COMMENT '列ID', `internal_column_id` int(11) DEFAULT NULL, `hidden_column` varchar(8) DEFAULT NULL, `virtual_column` varchar(8) DEFAULT NULL, `original_ser` int(11) NOT NULL COMMENT '源表变更序号', `data_type` varchar(128) NOT NULL DEFAULT '' COMMENT '数据类型', `data_length` bigint(20) NOT NULL COMMENT '数据长度', `data_precision` int(11) DEFAULT NULL COMMENT '数据精度', `data_scale` int(11) DEFAULT NULL COMMENT '小数部分长度', `char_length` int(11) DEFAULT NULL, `char_used` varchar(1) DEFAULT NULL, `nullable` varchar(1) NOT NULL DEFAULT '' COMMENT '是否可空Y/N', `is_pk` varchar(1) NOT NULL DEFAULT '' COMMENT '是否为主键Y/N', `pk_position` int(2) DEFAULT NULL COMMENT '主键的顺序', `alter_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '创建/修改的时间', `comments` varchar(512) DEFAULT NULL COMMENT '列注释', `default_value` varchar(256) DEFAULT NULL, PRIMARY KEY (`id`), KEY `idx_table_meta` (`ver_id`,`column_name`,`original_ser`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='抓取的源表数据meta信息'; -- ---------------------------- -- Table structure for t_user -- ---------------------------- DROP TABLE IF EXISTS `t_user`; CREATE TABLE `t_user` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `role_type` varchar(16) NOT NULL DEFAULT 'user' COMMENT '用户类型', `status` varchar(16) NOT NULL DEFAULT 'active' COMMENT '用户状态', `user_name` varchar(128) NOT NULL COMMENT '用户名', `password` varchar(256) DEFAULT NULL COMMENT '密码', `email` varchar(128) NOT NULL DEFAULT '' COMMENT 'email', `phone_num` varchar(32) DEFAULT '' COMMENT '电话号码', `update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`), UNIQUE KEY `idx_user_email` (`email`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- ---------------------------- -- Table structure for t_name_alias_mapping -- ---------------------------- DROP TABLE IF EXISTS `t_name_alias_mapping`; CREATE TABLE `t_name_alias_mapping` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键ID', `type` int(2) NOT NULL COMMENT '别名类型1,router拓扑别名;2,增量拓扑别名', `name` varchar(64) NOT NULL COMMENT '名称', `name_id` int(11) NOT NULL COMMENT '名称对应ID', `alias` varchar(64) NOT NULL COMMENT '别名', `update_time` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='别名表'; [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/conf/init]#
[root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/conf/init]# cat dbus_mgr.sql | grep "CREATE TABLE" | wc -l #理论上来说，应该创建29张表~ 29 [root@hdp103.yinzhengjie.org.cn /yinzhengjie/softwares/dbus/dbuskeeper_web/conf/init]#
14>.DBus的用户手册
博主推荐阅读: https://bridata.github.io/DBus/tutorial.html
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。 欢迎加入基础架构自动化运维：598432640，大数据SRE进阶之路：959042252，DevOps进阶之路：526991186
