doris集群环境部署
环境:
OS:Centos 7
DB:doris 2.1.1
192.168.1.101 fe be broker
192.168.1.103 fe be broker
192.168.1.105 fe be broker
说明:
1.doris2.0开始fe、be、brokser都是使用同一个安装包,解压后分别有fe、be、broker目录
2.broker(用于访问hdfs)可以不安装,该项为可选项
1.获取安装介质
https://doris.apache.org/zh-CN/download/
我这里下载的是
apache-doris-2.1.0-bin-x64.tar.gz
根据机器是否支持avx2找相应的包进行下载
cat /proc/cpuinfo |grep avx2
[root@host01 soft]# cat /proc/cpuinfo |grep avx2
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq monitor ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase avx2 invpcid rdseed clflushopt md_clear flush_l1d arch_capabilities
我这里是支持的
2.安装java
每台机器都要进行安装
[root@host01 soft]# java -version
java version "1.8.0_351"
Java(TM) SE Runtime Environment (build 1.8.0_351-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.351-b10, mixed mode)
3.设置系统最大打开文件句柄数
每台机器都要配置
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
4.关闭交换分区
swapoff -a ##临时关闭
永久删除Swap挂载
vim /etc/fstab
# 注释 swap 行
重启系统
5.关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service
6.修改/etc/sysctl.conf
#设置系统中每个进程可以拥有的最大内存映射区域数量
vi /etc/sysctl.conf
vm.max_map_count=2000000
#加载并应用内核
sysctl -p
#################安装fe########################
1.解压
在第一个节点上解压
[root@host01 soft]# tar -xvf apache-doris-2.1.0-bin-x64.tar.gz
[root@host01 soft]# mv apache-doris-2.1.0-bin-x64 /opt/doris
2.修改fe配置参数
修改第一个节点的配置
mysql_service_nio_enabled = true
priority_networks = 192.168.1.0/24 #新增该字段，特别是多网卡服务器上，该参数必须添加
qe_query_timeout_second=3000 ##超时设置，最大只能设置259200秒,即72小时
enable_batch_delete_by_default=true ##启用批量删除
3.启动第一个节点
[root@host01 ~]#/opt/doris/fe/bin/start_fe.sh --daemon
[root@host01 ~]#/opt/doris/fe/bin/stop_fe.sh ##停止命令
4.查看是否启动
[root@host01 doris]# jps
16986 DorisFE
17210 Jps
查看相应端口情况
[root@host01 fe]# ss -nlp|grep 9030
[root@host01 fe]# ss -nlp|grep 8030
[root@host01 fe]# ss -nlp|grep 9020
[root@host01 fe]# ss -nlp|grep 9010
5.mysql客户端登录
需要在节点上安装mysql客户端,或是直接解压mysql安装包即可使用
[root@host01 bin]# /opt/mysql57/bin/mysql -uroot -P 9030 -h 127.0.0.1 #直接回车，默认没有密码
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.7.99
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SHOW PROC '/frontends' \G;
*************************** 1. row ***************************
Name: fe_3708e8f3_f6f4_4446_a950_87d8bae2075d
Host: 192.168.1.188
EditLogPort: 9010
HttpPort: 8030
QueryPort: 9030
RpcPort: 9020
ArrowFlightSqlPort: -1
Role: FOLLOWER
IsMaster: true
ClusterId: 591245542
Join: true
Alive: true
ReplayedJournalId: 67
LastStartTime: 2024-03-13 13:59:02
LastHeartbeat: 2024-03-13 14:03:53
IsHelper: true
ErrMsg:
Version: doris-2.1.0-rc11-91efb6a43d
CurrentConnected: Yes
1 row in set (0.13 sec)
ERROR:
No query specified
发现这里ip是192.168.1.188 这个是该机器上的vip地址,真实地址是:192.168.1.101
这里不使用vip,需要将vip删掉掉,删除掉发现fe无法启动,后面我是重新安装才解决问题的.
6.另外两个节点也同样进行解压
节点2
[root@host02 soft]# tar -xvf apache-doris-2.1.0-bin-x64.tar.gz
[root@host02 soft]# mv apache-doris-2.1.0-bin-x64 /opt/doris
节点3
[root@host03 soft]# tar -xvf apache-doris-2.1.0-bin-x64.tar.gz
[root@host03 soft]# mv apache-doris-2.1.0-bin-x64 /opt/doris
7.将节点1上的配置文件拷贝到另外两个节点
在节点1上操作
scp /opt/doris/fe/conf/fe.conf root@192.168.1.103:/opt/doris/fe/conf/
scp /opt/doris/fe/conf/fe.conf root@192.168.1.105:/opt/doris/fe/conf/
8.启动另外两个节点
这里确保节点1已经正在运行了,启动剩下的2个节点
节点2:
[root@host02 soft]#/opt/doris/fe/bin/start_fe.sh --helper 192.168.1.101:9010 --daemon
节点3:
[root@host03 soft]#/opt/doris/fe/bin/start_fe.sh --helper 192.168.1.101:9010 --daemon
9.fe添加新节点
这里在节点1上操作
/opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101
mysql> ALTER SYSTEM ADD FOLLOWER "192.168.1.103:9010";
Query OK, 0 rows affected (0.08 sec)
mysql> ALTER SYSTEM ADD FOLLOWER "192.168.1.105:9010";
Query OK, 0 rows affected (0.05 sec)
10.再次查看fe情况
我这里是登录节点1上查看,登录其他节点查看也是可以的
/opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101
SHOW PROC '/frontends' \G; #确保相关节点 join和active值为true
mysql> SHOW PROC '/frontends' \G;
*************************** 1. row ***************************
Name: fe_32c6dcbb_138c_4379_94de_d5561b856474
Host: 192.168.1.103
EditLogPort: 9010
HttpPort: 8030
QueryPort: 9030
RpcPort: 9020
ArrowFlightSqlPort: -1
Role: FOLLOWER
IsMaster: false
ClusterId: 424480426
Join: true
Alive: true
ReplayedJournalId: 195
LastStartTime: 2024-03-13 15:05:38
LastHeartbeat: 2024-03-13 15:11:54
IsHelper: true
ErrMsg:
Version: doris-2.1.0-rc11-91efb6a43d
CurrentConnected: No
*************************** 2. row ***************************
Name: fe_74db1be3_65d7_49bb_a4ac_ef0f6dd72197
Host: 192.168.1.105
EditLogPort: 9010
HttpPort: 8030
QueryPort: 9030
RpcPort: 9020
ArrowFlightSqlPort: -1
Role: FOLLOWER
IsMaster: false
ClusterId: 424480426
Join: true
Alive: true
ReplayedJournalId: 195
LastStartTime: 2024-03-13 15:05:52
LastHeartbeat: 2024-03-13 15:11:54
IsHelper: true
ErrMsg:
Version: doris-2.1.0-rc11-91efb6a43d
CurrentConnected: No
*************************** 3. row ***************************
Name: fe_5e401a1b_6e76_4819_90e8_afb211bab1ad
Host: 192.168.1.101
EditLogPort: 9010
HttpPort: 8030
QueryPort: 9030
RpcPort: 9020
ArrowFlightSqlPort: -1
Role: FOLLOWER
IsMaster: true
ClusterId: 424480426
Join: true
Alive: true
ReplayedJournalId: 197
LastStartTime: 2024-03-13 14:56:21
LastHeartbeat: 2024-03-13 15:11:54
IsHelper: true
ErrMsg:
Version: doris-2.1.0-rc11-91efb6a43d
CurrentConnected: Yes
3 rows in set (0.19 sec)
ERROR:
No query specified
10.这个时候可以登录fe界面
http://192.168.1.101:8030
http://192.168.1.103:8030
http://192.168.1.105:8030
账号是root,密码为空
查看fe情况
http://192.168.1.101:8030/System?path=/frontends
##########################部署be##################################
1.修改配置文件
vi /opt/doris/be/conf/be.conf
brpc_port = 8060
storage_root_path = ${DORIS_HOME}/storage ##数据存储路径,默认是注释掉的,开启即可,路径不存在的话需要创建
##我这里的路径是:/opt/doris/be/storage
2.启动BE和Broker
#三个节点都执行以下操作
[root@host01 storage]# /opt/doris/be/bin/start_be.sh --daemon
Please set vm.max_map_count to be 2000000 under root using 'sysctl -w vm.max_map_count=2000000'.
[root@host01 storage]# /opt/doris/be/bin/start_be.sh --daemon
Please disable swap memory before installation.
[root@host01 storage]#/opt/doris/extensions/apache_hdfs_broker/bin/start_broker.sh --daemon
3.查看BE和Broker进程
[root@host01 conf]# jps
5252 DorisFE
15671 Jps
15624 BrokerBootstrap
13134 DorisBE
[root@host01 conf]# ss -nlp|grep 8060 ##BE端口 brpc_port
[root@host01 conf]# ss -nlp|grep 9050 ##BE端口 heartbeat_service_port
[root@host01 conf]# ss -nlp|grep 9060 ##BE端口 be_port
[root@host01 conf]# ss -nlp|grep 8040 ##BE端口 webserver_port
[root@localhost conf]# ss -nlp|grep 8000 ##Broker端口
5.数据库添加节点
登录fe,我这里从节点1上登录
#添加BE节点
SQL>ALTER SYSTEM ADD BACKEND "192.168.1.101:9050";
SQL>ALTER SYSTEM ADD BACKEND "192.168.1.103:9050";
SQL>ALTER SYSTEM ADD BACKEND "192.168.1.105:9050";
SQL>SHOW BACKENDS \G;
#添加Broker节点
SQL>ALTER SYSTEM ADD BROKER my_broker "192.168.1.101:8000";
SQL>ALTER SYSTEM ADD BROKER my_broker "192.168.1.103:8000";
SQL>ALTER SYSTEM ADD BROKER my_broker "192.168.1.105:8000";
SQL>SHOW BROKER \G;
6.修改root密码
登录fe
/opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101
mysql> SET PASSWORD FOR 'root' = PASSWORD('mysql');
Query OK, 0 rows affected (0.03 sec)
这个时候登录和网页相关的登录都需要密码了.
[root@host01 ~]# /opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101 -pmysql
############################集群维护##########################
日常维护
1.启动顺序
先启动fe,再启动be和broker
1.1 启动fe
第一个节点
[root@host01 ~]# /opt/doris/fe/bin/start_fe.sh --daemon
其他节点:
[root@host02 soft]#/opt/doris/fe/bin/start_fe.sh --helper 192.168.1.101:9010 --daemon
[root@host03 soft]#/opt/doris/fe/bin/start_fe.sh --helper 192.168.1.101:9010 --daemon
第一次安装其他节点启动需要带上helper,之后可以不用带上即可启动其他的节点
/opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101
SHOW PROC '/frontends' \G; #确保相关节点 join和active值为true
1.2 启动be和broker
每个节点都按照如下方式启动(没有节点顺序):
/opt/doris/be/bin/start_be.sh --daemon
/opt/doris/extensions/apache_hdfs_broker/bin/start_broker.sh --daemon
2.停掉顺序
先停掉be和broker,再停掉fe
/opt/doris/fe/bin/stop_fe.sh
说明:3个fe,停掉了2个后就不能使用了
###########################开发相关############################
1.创建数据库和表
[root@host01 ~]# /opt/mysql57/bin/mysql -uroot -P 9030 -h 192.168.1.101 -pmysql
mysql> create database db_hxl;
Query OK, 0 rows affected (0.06 sec)
use db_hxl;
CREATE TABLE tb_test
(
id LARGEINT NOT NULL COMMENT "id",
name VARCHAR(20)
)
AGGREGATE KEY(id,name)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
insert into tb_test values(1,'name1');
insert into tb_test values(2,'name2');
insert into tb_test values(3,'name3');
insert into tb_test values(4,'name4');
insert into tb_test values(5,'name5');
insert into tb_test values(5,'name5'); ##再写入一条id=5的,系统不添加,相同的记录只会记录一条
insert into tb_test values(5,'name55'); ##该条会添加,name=name55前面的不存在
mysql> show tables;
+------------------+
| Tables_in_db_hxl |
+------------------+
| example_tbl |
+------------------+
1 row in set (0.04 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| __internal_schema |
| db_hxl |
| information_schema |
| mysql |
+--------------------+
4 rows in set (0.00 sec)
2.创建用户并授权(语法与mysql相同)
CREATE USER 'hxl' IDENTIFIED BY 'mysql';
GRANT ALL ON db_hxl.* TO 'hxl'@'%';
使用新创建的用户登录
[root@host01 ~]# /opt/mysql57/bin/mysql -uhxl -P 9030 -h 192.168.1.101 -pmysql
show roles;
SELECT CURRENT_USER();
mysql> SELECT CURRENT_USER();
+----------------+
| current_user() |
+----------------+
| 'hxl'@'%' |
+----------------+
1 row in set (0.01 sec)
mysql> show grants for 'hxl'@'%';
查看所有的用户
mysql> show all grants;
################节点宕机验证####################
1.验证fe停掉
3个节点组成的fe集群,宕机2个就不可使用了的.
2.验证be停掉
3个节点组成的be,宕机2个就不能写入数据
mysql> insert into tb_test values(9,'name9');
ERROR 1105 (HY000): errCode = 3, detailMessage = tablet 13033 alive replica num 0 < load required replica num 1, alive backends: []
也不能查询
mysql> select * from tb_test;
ERROR 1105 (HY000): errCode = 2, detailMessage = tablet 13033 has no queryable replicas. err: replica 13034's backend 11584 does not exist or not alive
3.验证broker停掉
3个节点组成的broker集群,3个节点全部宕机,fe、be集群正常使用
