【GP集群部署】集群部署（2+4模式），Greenplum集群源码安装文档
-
一、环境准备
-
操作系统版本
$ cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
Python版本：Python 2.7.5
- Greenplum版本：
gpdb-6.17.5.tar.gz
下载地址： https://github.com/greenplum-db/gpdb/releases
- IP地址规划
10.172.140.8 vip
10.172.140.9 t3-zeze-bigdata-gp09
10.172.140.10 t3-zeze-bigdata-gp10
10.172.140.11 t3-zeze-bigdata-gp11
10.172.140.12 t3-zeze-bigdata-gp12
10.172.140.13 t3-zeze-bigdata-gp13
10.172.140.14 t3-zeze-bigdata-gp14
- 架构规划
部署2+4的架构，segment节点1个data实例1个mirror实例
计划将10.172.140.9作为master，10.172.140.10作为standby master, 10.172.140.11、10.172.140.12、10.172.140.13作为segment instance
-
1)修改host文件（所有节点）
$ vi /etc/hosts
# for greenplum
10.172.140.9 t3-zeze-bigdata-gp09
10.172.140.10 t3-zeze-bigdata-gp10
10.172.140.11 t3-zeze-bigdata-gp11
10.172.140.12 t3-zeze-bigdata-gp12
10.172.140.13 t3-zeze-bigdata-gp13
10.172.140.14 t3-zeze-bigdata-gp14
拼接字符串，修改host文件-
2)创建用户和用户组（所有节点）
groupadd -g 5432 gpadmin
useradd -g 5432 –u5432 -m -d /home/gpadmin -s /bin/bash gpadmin
echo gpadmin | passwd gpadmin --stdin
usermod -aG wheel gpadmin
-
3)修改系统内核（所有节点）
vi /etc/sysctl.conf
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.overcommit_memory = 2
sysctl -p
-
4)修改文件打开限制（所有节点）
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 13107
-
5)关闭防火墙（所有节点）
启动： systemctl start firewalld
关闭： systemctl stop firewalld
查看状态： systemctl status firewalld
开机禁用 ： systemctl disable firewalld
开机启用 ： systemctl enable firewalld
-
6)关闭SELINUX（所有节点）
vi /etc/selinux/config
SELINUX=disabled
-
7)设置时区（所有节点）
timedatectl set-timezone Asia/Shanghai
-
8)创建安装文件目录（所有节点）
su - gpadmin
mkdir /gpdb/{gp_home,gp_base} 存放软件安装目录及数据目录
chown -R gpadmin:gpadmin /home/gpadmin
mkdir /home/gpadmin/dba 存放安装软件的目录及临时文件
-
9)上传greenplum安装包到master节点，其他安装包所有节点都需要上传
$ cd dba
$ ls |grep tar
gpdb-6.17.5.tar.gz
pip-20.1.tar.gz
psutil-3.2.2.tar.gz
PyYAML-5.3.tar.gz
wheel-0.33.6.tar.gz
-
10)配置ssh免密登录（所有节点）
$ su - gpadmin
$ssh-keygen -t rsa
将其他节点公钥id_rsa.pub导入到其他节点的认证文件
$cat id_rsa.pub >> authorized_keys
或者采用：
cat >> authorized_keys << EOF
xxxxx
EOF
并修改文件权限
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
最后，进行测试
ssh t3-zeze-bigdata-gp09 date
ssh t3-zeze-bigdata-gp10 date
ssh t3-zeze-bigdata-gp11 date
ssh t3-zeze-bigdata-gp12 date
ssh t3-zeze-bigdata-gp13 date
ssh t3-zeze-bigdata-gp14 date
-
二、集群安装
-
1)安装依赖包及python环境
yum install -y apr-devel
yum install -y libevent-devel
yum install -y libcurl-devel.x86_64
yum install -y bzip2-devel
yum install -y gcc-c++
yum install -y bison.x86_64 bison-devel.x86_64
yum install -y openssl-devel
yum install -y flex
yum install -y python-pip
yum install -y python-devel
yum install -y gcc python27-devel libxml2 libxml2-devel libxslt libxslt-devel openssl libzstd
root执行
# cd /home/greenplum/dba/pip/
# python setup.py install
# cd /home/greenplum/dba/PyYAML/
# python setup.py install
# cd /home/greenplum/dba/wheel/
# python setup.py install
# cd psutil-3.2.2
# python setup.py install
-
2)初始化master数据库实例
./configure --prefix=/gpdb/gp_home --with-pgport=5432 --disable-gpcloud --disable-orca
make
make install
-
3)修改用户greenplum的环境变量.bash_profile
master节点、standby master节点
#for greenplum
source /gpdb/gp_home/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/gpdb/gp_base/gpmaster/gpseg-1
export PGPORT=5432
segment instance节点
#for greenplum
source /gpdb/gp_home/greenplum_path.sh
export PGPORT=5432
-
4)创建配置集群hostlist文件、seg_hosts
$ su - gpadmin
$ mkdir -p /gpdb/gp_home/conf/
$ vi /gpdb/gp_home/conf/hostlist
t3-zeze-bigdata-gp09
t3-zeze-bigdata-gp10
t3-zeze-bigdata-gp11
t3-zeze-bigdata-gp12
t3-zeze-bigdata-gp13
t3-zeze-bigdata-gp14
$ vi /gpdb/gp_home/conf/seg_hosts
t3-zeze-bigdata-gp11
t3-zeze-bigdata-gp12
t3-zeze-bigdata-gp13
t3-zeze-bigdata-gp14
-
5)将gphome分发到每个节点
$ source /gpdb/gp_home/greenplum_path.sh
$ gpssh-exkeys -f /gpdb/gp_home/conf/hostlist
$ gpssh -f /home/greenplum/gphome/conf/hostlist
=> pwd
[t3-zeze-bigdata-gp12] /gpdb/gp_home
[t3-zeze-bigdata-gp09] /gpdb/gp_home
[t3-zeze-bigdata-gp10] /gpdb/gp_home
[t3-zeze-bigdata-gp11] /gpdb/gp_home
[t3-zeze-bigdata-gp13] /gpdb/gp_home
[t3-zeze-bigdata-gp14] /gpdb/gp_home
=> tar -cf gphome.tar gphome
/gpdb/gp_home/bin/gpscp -f /gpdb/gp_home/conf/hostlist gp.tar =:/gpdb/gp_home/ （复制到每台机器命令）
创建pgbase目录
$ gpssh -f /gpdb/gp_home /conf/hostlist
=> mkdir gpdatap1 gpdatam1
=> exit
在master及standby master节点编辑gpinitsystem_config文件
vi /gpdb/gp_home/conf/gpinitsystem_config
ARRAY_NAME="Greenplum"
SEG_PREFIX=gpseg
PORT_BASE=33000
declare -a DATA_DIRECTORY=(/gpdb/gp_base/gpdatap1)
MASTER_HOSTNAME=t3-tkhibdp-bdp-gp01
MASTER_DIRECTORY=/gpdb/gp_base/gpmaster
MASTER_PORT=5432
TRUSTED_SHELL=/usr/bin/ssh
MIRROR_PORT_BASE=43000
REPLICATION_PORT_BASE=34000
MIRROR_REPLICATION_PORT_BASE=44000
declare -a MIRROR_DATA_DIRECTORY=(/gpdb/gp_base/gpdatam1)
MACHINE_LIST_FILE=/gpdb/gp_home/conf/seg_hosts
-
6)安装GP集群数据库
source ~/.bash_profile
gpinitsystem -c /gpdb/gp_home/conf/gpinitsystem_config -h /gpdb/gp_home /conf/seg_hosts -s t3-zeze-bigdata-gp10
-
7)绑定虚拟vip
挂载vip
ip a a 10.172.140.8/32 brd + dev bond0 && arping -q -c 3 -U -I bond0 10.172.140.8
卸载vip
# ip a d 10.172.140.8/32 brd + dev bond0
三、集群管理
检查standby master节点是否正常部署
$ gpstate -a
$ gpstate -a 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Starting gpstate with args: -a 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.0.0-beta.1 build dev' 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.26 (Greenplum Database 6.0.0-beta.1 build dev) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit compiled on Mar 27 2022 13:46:32' 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Obtaining Segment details from master... 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Gathering data from segments... 20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Greenplum instance status summary 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Master instance = Active 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Master standby = t3-zeze-bigdata-gp10 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Standby master state = Standby host passive 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total segment instance count from metadata = 8 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Primary Segment Status 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total primary segments = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total primary segment valid (at master) = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total primary segment failures (at master) = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid files found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of /tmp lock files found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number postmaster processes missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number postmaster processes found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Mirror Segment Status 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total mirror segments = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total mirror segment valid (at master) = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total mirror segment failures (at master) = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid files found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number of /tmp lock files found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number postmaster processes missing = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number postmaster processes found = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number mirror segments acting as primary segments = 0 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 6 20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:----------------------------------------------------- $
检查data与mirror数据同步关系
$ gpstate -c
$ gpstate -c 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Starting gpstate with args: -c 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.0.0-beta.1 build dev' 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.26 (Greenplum Database 6.0.0-beta.1 build dev) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit compiled on Mar 27 2022 13:46:32' 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Obtaining Segment details from master... 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-------------------------------------------------------------- 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--Current GPDB mirror list and status 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--Type = Spread 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-------------------------------------------------------------- 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Status Data State Primary Datadir Port Mirror Datadir Port 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized t3-zeze-bigdata-gp11 /gpdb/gp_base/gpdatap1/gpseg0 33000 t3-zeze-bigdata-gp12 /gpdb/gp_base/gpdatam1/gpseg0 43000 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized t3-zeze-bigdata-gp12 /gpdb/gp_base/gpdatap1/gpseg1 33000 t3-zeze-bigdata-gp13 /gpdb/gp_base/gpdatam1/gpseg1 43000 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized t3-zeze-bigdata-gp13 /gpdb/gp_base/gpdatap1/gpseg2 33000 t3-zeze-bigdata-gp14 /gpdb/gp_base/gpdatam1/gpseg2 43000 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized t3-zeze-bigdata-gp14 /gpdb/gp_base/gpdatap1/gpseg3 33000 t3-zeze-bigdata-gp09 /gpdb/gp_base/gpdatam1/gpseg3 43000 20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--------------------------------------------------------------
$gpstop -M fast
$gpstart -a
内存配置：
$ gpconfig -c shared_buffers -v 4096MB -m 4096MB
$ gpconfig -c gp_vmem_protect_limit -v 30720 -m 30720
$ gpconfig -c max_connections -v 2000 -m 400
$ gpconfig --skipvalidation -c wal_send_client_timeout -v 60s -m 60s
