【GP集群部署】集群部署（2+4模式），Greenplum集群源码安装文档

一、环境准备
操作系统版本

$ cat /etc/redhat-release

CentOS Linux release 7.8.2003 (Core)

Python版本：Python 2.7.5

Greenplum版本：

gpdb-6.17.5.tar.gz

下载地址： https://github.com/greenplum-db/gpdb/releases

IP地址规划

10.172.140.8 vip

10.172.140.9 t3-zeze-bigdata-gp09

10.172.140.10 t3-zeze-bigdata-gp10

10.172.140.11 t3-zeze-bigdata-gp11

10.172.140.12 t3-zeze-bigdata-gp12

10.172.140.13 t3-zeze-bigdata-gp13

10.172.140.14 t3-zeze-bigdata-gp14

架构规划

部署2+4的架构，segment节点1个data实例1个mirror实例

计划将10.172.140.9作为master，10.172.140.10作为standby master, 10.172.140.11、10.172.140.12、10.172.140.13作为segment instance

1)修改host文件（所有节点）

$ vi /etc/hosts

# for greenplum

10.172.140.9 t3-zeze-bigdata-gp09

10.172.140.10 t3-zeze-bigdata-gp10

10.172.140.11 t3-zeze-bigdata-gp11

10.172.140.12 t3-zeze-bigdata-gp12

10.172.140.13 t3-zeze-bigdata-gp13

10.172.140.14 t3-zeze-bigdata-gp14

拼接字符串，修改host文件

2)创建用户和用户组（所有节点）

groupadd -g 5432 gpadmin

useradd -g 5432 –u5432 -m -d /home/gpadmin -s /bin/bash gpadmin

echo gpadmin | passwd gpadmin --stdin

usermod -aG wheel gpadmin

3)修改系统内核（所有节点）

vi /etc/sysctl.conf

kernel.shmmax = 500000000

kernel.shmmni = 4096

kernel.shmall = 4000000000

kernel.sem = 250 512000 100 2048

kernel.sysrq = 1

kernel.core_uses_pid = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.msgmni = 2048

net.ipv4.tcp_syncookies = 1

net.ipv4.ip_forward = 0

net.ipv4.conf.default.accept_source_route = 0

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_max_syn_backlog = 4096

net.ipv4.conf.all.arp_filter = 1

net.ipv4.ip_local_port_range = 1025 65535

net.core.netdev_max_backlog = 10000

net.core.rmem_max = 2097152

net.core.wmem_max = 2097152

vm.overcommit_memory = 2

sysctl -p

4)修改文件打开限制（所有节点）

vi /etc/security/limits.conf

* soft nofile 65536

* hard nofile 65536

* soft nproc 131072

* hard nproc 13107

5)关闭防火墙（所有节点）

启动： systemctl start firewalld

关闭： systemctl stop firewalld

查看状态： systemctl status firewalld

开机禁用： systemctl disable firewalld

开机启用： systemctl enable firewalld

6)关闭SELINUX（所有节点）

vi /etc/selinux/config

SELINUX=disabled

7)设置时区（所有节点）

timedatectl set-timezone Asia/Shanghai

su - gpadmin

mkdir /gpdb/{gp_home,gp_base} 存放软件安装目录及数据目录

chown -R gpadmin:gpadmin /home/gpadmin

mkdir /home/gpadmin/dba 存放安装软件的目录及临时文件

9)上传greenplum安装包到master节点，其他安装包所有节点都需要上传

$ cd dba

$ ls |grep tar

gpdb-6.17.5.tar.gz

pip-20.1.tar.gz

psutil-3.2.2.tar.gz

PyYAML-5.3.tar.gz

wheel-0.33.6.tar.gz

10)配置ssh免密登录（所有节点）

$ su - gpadmin

$ssh-keygen -t rsa

将其他节点公钥id_rsa.pub导入到其他节点的认证文件

$cat id_rsa.pub >> authorized_keys

或者采用：

cat >> authorized_keys << EOF
xxxxx
EOF

并修改文件权限

chmod 700 ~/.ssh

chmod 600 ~/.ssh/authorized_keys

最后，进行测试

ssh t3-zeze-bigdata-gp09 date

ssh t3-zeze-bigdata-gp10 date

ssh t3-zeze-bigdata-gp11 date

ssh t3-zeze-bigdata-gp12 date

ssh t3-zeze-bigdata-gp13 date

ssh t3-zeze-bigdata-gp14 date

二、集群安装
1)安装依赖包及python环境

yum install -y apr-devel

yum install -y libevent-devel

yum install -y libcurl-devel.x86_64

yum install -y bzip2-devel

yum install -y gcc-c++

yum install -y bison.x86_64 bison-devel.x86_64

yum install -y openssl-devel

yum install -y flex

yum install -y python-pip

yum install -y python-devel

yum install -y gcc python27-devel libxml2 libxml2-devel libxslt libxslt-devel openssl libzstd

root执行

# cd /home/greenplum/dba/pip/

# python setup.py install

# cd /home/greenplum/dba/PyYAML/

# python setup.py install

# cd /home/greenplum/dba/wheel/

# python setup.py install

# cd psutil-3.2.2

# python setup.py install

2)初始化master数据库实例

./configure --prefix=/gpdb/gp_home --with-pgport=5432 --disable-gpcloud --disable-orca

make

make install

3)修改用户greenplum的环境变量.bash_profile

master节点、standby master节点

#for greenplum

source /gpdb/gp_home/greenplum_path.sh

export MASTER_DATA_DIRECTORY=/gpdb/gp_base/gpmaster/gpseg-1

export PGPORT=5432

segment instance节点

#for greenplum

source /gpdb/gp_home/greenplum_path.sh

export PGPORT=5432

4)创建配置集群hostlist文件、seg_hosts

$ su - gpadmin

$ mkdir -p /gpdb/gp_home/conf/

$ vi /gpdb/gp_home/conf/hostlist

t3-zeze-bigdata-gp09

t3-zeze-bigdata-gp10

t3-zeze-bigdata-gp11

t3-zeze-bigdata-gp12

t3-zeze-bigdata-gp13

t3-zeze-bigdata-gp14

$ vi /gpdb/gp_home/conf/seg_hosts

t3-zeze-bigdata-gp11

t3-zeze-bigdata-gp12

t3-zeze-bigdata-gp13

t3-zeze-bigdata-gp14

5)将gphome分发到每个节点

$ source /gpdb/gp_home/greenplum_path.sh

$ gpssh-exkeys -f /gpdb/gp_home/conf/hostlist

$ gpssh -f /home/greenplum/gphome/conf/hostlist

=> pwd

[t3-zeze-bigdata-gp12] /gpdb/gp_home

[t3-zeze-bigdata-gp09] /gpdb/gp_home

[t3-zeze-bigdata-gp10] /gpdb/gp_home

[t3-zeze-bigdata-gp11] /gpdb/gp_home

[t3-zeze-bigdata-gp13] /gpdb/gp_home

[t3-zeze-bigdata-gp14] /gpdb/gp_home

=> tar -cf gphome.tar gphome

/gpdb/gp_home/bin/gpscp -f /gpdb/gp_home/conf/hostlist gp.tar =:/gpdb/gp_home/ （复制到每台机器命令）

创建pgbase目录

$ gpssh -f /gpdb/gp_home /conf/hostlist

=> mkdir gpdatap1 gpdatam1

=> exit

在master及standby master节点编辑gpinitsystem_config文件

vi /gpdb/gp_home/conf/gpinitsystem_config

ARRAY_NAME="Greenplum"

SEG_PREFIX=gpseg

PORT_BASE=33000

declare -a DATA_DIRECTORY=(/gpdb/gp_base/gpdatap1)

MASTER_HOSTNAME=t3-tkhibdp-bdp-gp01

MASTER_DIRECTORY=/gpdb/gp_base/gpmaster

MASTER_PORT=5432

TRUSTED_SHELL=/usr/bin/ssh

MIRROR_PORT_BASE=43000

REPLICATION_PORT_BASE=34000

MIRROR_REPLICATION_PORT_BASE=44000

declare -a MIRROR_DATA_DIRECTORY=(/gpdb/gp_base/gpdatam1)

MACHINE_LIST_FILE=/gpdb/gp_home/conf/seg_hosts

6)安装GP集群数据库

source ~/.bash_profile

gpinitsystem -c /gpdb/gp_home/conf/gpinitsystem_config -h /gpdb/gp_home /conf/seg_hosts -s t3-zeze-bigdata-gp10

7)绑定虚拟vip

挂载vip

ip a a 10.172.140.8/32 brd + dev bond0 && arping -q -c 3 -U -I bond0 10.172.140.8

卸载vip

# ip a d 10.172.140.8/32 brd + dev bond0

三、集群管理

检查standby master节点是否正常部署
$ gpstate -a

$ gpstate -a
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Starting gpstate with args: -a
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.0.0-beta.1 build dev'
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.26 (Greenplum Database 6.0.0-beta.1 build dev) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit compiled on Mar 27 2022 13:46:32'
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Obtaining Segment details from master...
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Gathering data from segments...
20220406:13:26:11:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Greenplum instance status summary
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Master instance                                           = Active
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Master standby                                            = t3-zeze-bigdata-gp10
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total segment instance count from metadata                = 8
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Primary Segment Status
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total primary segments                                    = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total primary segment failures (at master)                = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid files missing              = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs missing               = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of /tmp lock files missing                   = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number postmaster processes missing                 = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number postmaster processes found                   = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Mirror Segment Status
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total mirror segments                                     = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total mirror segment failures (at master)                 = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid files missing              = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs missing               = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of /tmp lock files missing                   = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number postmaster processes missing                 = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number postmaster processes found                   = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number mirror segments acting as primary segments   = 0
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 6
20220406:13:26:12:095774 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-----------------------------------------------------
$

gpstate -a

检查data与mirror数据同步关系
$ gpstate -c

$ gpstate -c
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Starting gpstate with args: -c
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.0.0-beta.1 build dev'
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.26 (Greenplum Database 6.0.0-beta.1 build dev) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit compiled on Mar 27 2022 13:46:32'
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-Obtaining Segment details from master...
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--------------------------------------------------------------
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--Current GPDB mirror list and status
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--Type = Spread
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--------------------------------------------------------------
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Status                             Data State     Primary               Datadir                         Port    Mirror                Datadir                         Port
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   t3-zeze-bigdata-gp11   /gpdb/gp_base/gpdatap1/gpseg0   33000   t3-zeze-bigdata-gp12   /gpdb/gp_base/gpdatam1/gpseg0   43000
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   t3-zeze-bigdata-gp12   /gpdb/gp_base/gpdatap1/gpseg1   33000   t3-zeze-bigdata-gp13   /gpdb/gp_base/gpdatam1/gpseg1   43000
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   t3-zeze-bigdata-gp13   /gpdb/gp_base/gpdatap1/gpseg2   33000   t3-zeze-bigdata-gp14   /gpdb/gp_base/gpdatam1/gpseg2   43000
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   t3-zeze-bigdata-gp14   /gpdb/gp_base/gpdatap1/gpseg3   33000   t3-zeze-bigdata-gp09   /gpdb/gp_base/gpdatam1/gpseg3   43000
20220406:13:27:42:095990 gpstate:t3-zeze-bigdata-gp09:gpadmin-[INFO]:--------------------------------------------------------------

gpstate -c

$gpstop -M fast

$gpstart -a

内存配置：

$ gpconfig -c shared_buffers -v 4096MB -m 4096MB

$ gpconfig -c gp_vmem_protect_limit -v 30720 -m 30720

$ gpconfig -c max_connections -v 2000 -m 400

$ gpconfig --skipvalidation -c wal_send_client_timeout -v 60s -m 60s

posted on 2022-04-06 13:44 HelonTian 阅读(447) 评论(0) 编辑收藏举报

【GP集群部署】集群部署（2+4模式），Greenplum集群源码安装文档

一、环境准备

1)修改host文件（所有节点）

2)创建用户和用户组（所有节点）

3)修改系统内核（所有节点）

4)修改文件打开限制（所有节点）

5)关闭防火墙（所有节点）

6)关闭SELINUX（所有节点）

7)设置时区（所有节点）

8)创建安装文件目录（所有节点）

9)上传greenplum安装包到master节点，其他安装包所有节点都需要上传

10)配置ssh免密登录（所有节点）

二、集群安装

1)安装依赖包及python环境

2)初始化master数据库实例

3)修改用户greenplum的环境变量.bash_profile

4)创建配置集群hostlist文件、seg_hosts

5)将gphome分发到每个节点

6)安装GP集群数据库

7)绑定虚拟vip

三、集群管理