mongodb多实例安装及replica set配置

　　需求：为了容灾，需要部署两台服务器，便于主备切换，当主服务器挂掉之后，主辅服务器角色永远对换，即使之前主服务器恢复也不再重新恢复之前主辅角色，直到下次发生主备切换

       分析：mongodb两种数据复制方式：1、主备，当主服务器挂掉之后，辅服务器不能够提供写服务，故mongodb主备方案不合适。2、replica set，但只有两个节点时，主节点挂掉，辅节点也不能提供写服务，与主备效果一样，所以至少需要三个mongodb实例，但当前只有两台服务器，所以有一台服务器要安装有两个mongodb实例，一台机器上不需要两份数据拷贝，故第三个节点使用arbiter，只进行投票，不会被选为主节点，也不会拷贝数据。另外考虑到需求中主备逻辑，结合replica set特点，arbiter只能在服务器中运行，考虑到主备角色调换，故两台服务器每台需要安装两个mongodb实例，其中一个为arbiter

　　两台服务器：

　　192.168.103.5     （原始主服务器）

　　192.168.103.28   （原始辅服务器）

一、两台服务器分别安装mongodb

安装步骤可参考：https://www.cnblogs.com/qq931399960/p/10425803.html

二、两台服务器分别安装arbiter

1、新建mongodb日志和数据库存储路径

mkdir -p /home/mongo-arbiter/db
mkdir -p /home/mongo-arbiter/log

2、拷贝配置文件

cp /etc/mongod.conf /etc/mongod-arbiter.conf

3、修改配置文件

vim /etc/mongod-arbiter.conf

systemLog.path修改为/home/mongo-arbiter/log/mongod.log

Storage.dbPath修改为/home/mongo-arbiter/db

processManagement.pidFilePath修改为/var/run/mongodb/mongod-arbiter.pid

net.port修改为27018

4、修改权限

chown -R mongod:mongod /home/mongo-arbiter

5、设置arbiter开机启动

cp /usr/lib/systemd/system/mongod.service /usr/lib/systemd/system/mongod-arbiter.service

修改开机启动文件

vim /usr/lib/systemd/system/mongod-arbiter.service

Environment改为："OPTIONS=-f /home/mongo-arbiter/mongod.conf"

PIDFile改为：/var/run/mongodb/mongod-arbiter.pid

使配置生效

systemctl enable mongod-arbiter.service
systemctl daemon-reload

初始化用户（四个实例的用户要一致）

 cat ./mongdb/initUser.js | mongo --port 27018 --shell

6、登录检测

mongo  -uroot -pabc123 --authenticationDatabase "admin" --port 27018

正常登陆，ok

三、配置replica set

1、停止所有mongodb实例

2、生成keyfile文件

openssl rand -base64 756 > /home/replSetKey
chmod 400 /home/replSetKey
chown -R mongod:mongod /home/replSetKey

2、将上述文件拷贝到另外一台服务器/home下（路径可自定义，只要四个实例使用相同kefile文件即可）

3、添加/修改mongodb实例配置（如下配置每个实例相同，4个mongodb实例都需要修改，如下列出的只有新加和修改的，不变的未列出）

security:
  keyFile: /home/replSetKey      ##  新加
replication:
  replSetName: mongoSet         ##  新加
net:
  bindIp: 0.0.0.0   ## 修改

4、启动主服务器默认27017端口mongodb实例，启动辅服务器中两个mongodb实例

systemctl start mongod
systemctl start mongod-arbiter

5、查看防火墙是否关闭，或者27017和27018端口是否放开

未关闭防火墙或者以上端口不能访问，则初始化replica set失败

关闭防火墙命令

systemctl stop firewalld
systemctl disable firewalld

6、在主服务器（103.5）进入mongodb命令

mongo -u "root" -p "abc123" --authenticationDatabase "admin"

7、初始化两个memeber的replicat set

config = {_id:"mongoSet", members:[{_id:0,host:"192.168.103.5:27017",priority:2},
　　　　　　　　　　　　　　　　　　{_id:1,host:"192.168.103.28:27017",priority:1}]}
rs.initiate(config, true)

此时可通过rs.status()状态

8、添加arbiter仲裁节点

rs.addArb("192.168.103.28:27018")

rs.status()查看各节点状态，如果有节点stateStr显示not reachable/healthy，health显示为0，则重启该节点，再次通过rs.status()查看是否正常

到此，主备服务基本配置结束

java客户端连接地址为：

mongodb://root:abc123@192.168.103.5:27017,192.168.103.28:27017/?authSource=mydb&replicaSet=mongoSet

四、重新配置replica set

当主备服务器切换，java客户端连接由于配置了replicaSet，可以自动去新主节点进行业务处理，不会有问题，当主设备恢复为后，原主服务器mongodb再次成为了主节点，但其他服务已经在新主服务器上

A、mongodb和web服务不在台机器，中间有网络传输，影响性能

B、此时的replica set中，主节点在辅服务器上，主服务器上只有辅节点和arbiter，当主服务器挂掉，只有一个mongodb节点，不能在提供写服务

综上两点，尤其是第二点，需要重新配置replica set，登录到mongodb主节点

mongo  -uroot -pabc123 --authenticationDatabase "admin"

或者在主服务器远程登录恢复的mongodb主节点

mongo -u "root" -p "abc123" --authenticationDatabase "admin" --host 192.168.103.5

重新配置

reconfig = {_id:"mongoSet","protocolVersion" : 1, members:[{_id:0,host:"192.168.103.5:27017",priority:1},
　　　　　　　　　　　　　　　　　　{_id:1,host:"192.168.103.28:27017",priority:2}]}

rs.reconfig(reconfig, true)

添加arbiter节点

rs.addArb("192.168.103.5:27018")

使用rs.status()查看状态，如果有节点stateStr显示not reachable/healthy，health显示为0，则重启该节点，再次通过rs.status()查看是否正常

 

PS：

1、若有节点stateStr出现not reachable/healthy，health显示为0，则该节点可能是arbiter节点，官方不推荐将arbiter节点安装到存在primary和slave节点的服务器上，不过当前只有两台服务器，也只有这样了，可以正常运行。

2、重新配置时，需要保证之前已存在的host的id是相同的

3、重新配置需要添加protocolVersion

 

五、使用脚本简化操作

1、提前准备好配置文件和arbiter开机启动service文件，需要的字段都添加上,但replica set相关字段为注释状态，因为打开后，所有节点都将无法进行写操作，新建用户后打开，并重启mongodb

2、提前准备好kefile，两台服务器都使用该文件

3、安装mongodb (数据库和日志路径可以自定义)

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)
function initenv(){
  setenforce 0
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
}
########inst mongdb 4.0#############
function inst(){
  cp -f ./mongdb/mongod.conf /etc/
  rpm -ivh ./mongdb/*rpm
  cd /home
  mkdir -p mongo-db
  mkdir -p mongo-log
  chown -R mongod:mongod ./mongo-db
  chown -R mongod:mongod ./mongo-log

  systemctl enable mongod
  systemctl start mongod
}
function initdb(){
  cd $basedir
  cat ./mongdb/initUser.js | mongo --shell
  cat ./mongdb/initIndex.js | mongo -u "root" -p "abc123" --authenticationDatabase "admin"
}

function cpReplicaSetKey(){
    cd $basedir
    cp ./mongdb/replSetKey /home/
    chmod 400 /home/replSetKey
    chown -R mongod:mongod /home/replSetKey
}

function openReplicaSet(){
    sed -i 's/#keyFile/keyFile/g' /etc/mongod.conf
    sed -i 's/#replication/replication/g' /etc/mongod.conf
    sed -i 's/#replSetName/replSetName/g' /etc/mongod.conf
}

function stopMongo(){
    systemctl stop mongod
}
function startMongo(){
    systemctl start mongod
}

function main(){
  initenv
  inst
  initdb
  stopMongo
  cpReplicaSetKey
  openReplicaSet
  startMongo
}

main | tee -a ./log/inst_mongodb.log

4、安装mongodb arbiter节点（在rs.addArb()之前其实还不能说是arbiter节点，只是起了个名字叫arbiter节点）

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)
function initenv(){
  setenforce 0
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
}
########inst mongdb 4.0#############
function inst(){
  cp -f ./mongdb/mongod-arbiter.conf /etc/
  cp -f ./mongdb/mongod-arbiter.service /usr/lib/systemd/system/
  
  mkdir -p /home/mongodb-arbiter/log
  mkdir -p /home/mongodb-arbiter/db
  chown -R mongod:mongod /home/mongodb-arbiter

  systemctl enable mongod-arbiter
  systemctl start mongod-arbiter
}
function initdb(){
  cd $basedir
  cat ./mongdb/initUser.js | mongo --port 27018 --shell
  cat ./mongdb/initIndex.js | mongo --port 27018 -u "root" -p "abc123" --authenticationDatabase "admin"
}

function openReplicaSet(){
    sed -i 's/#keyFile/keyFile/g' /etc/mongod-arbiter.conf
    sed -i 's/#replication/replication/g' /etc/mongod-arbiter.conf
    sed -i 's/#replSetName/replSetName/g' /etc/mongod-arbiter.conf
}

function stopMongo(){
    systemctl stop mongod
}
function startMongo(){
    systemctl start mongod
}

function main(){
  initenv
  inst
  initdb
  stopMongo
  openReplicaSet
  startMongo
}

main | tee -a ./log/inst_mongodb_arbiter.log

5、初始replica set

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)

source ./app.properties

nowtime=`date --date='0 days ago' "+%Y%m%d%H%M%S"`
daytime=`date --date='0 days ago' "+%Y%m%d"`

logPath=./log/init_repliaset_$daytime.log
function log(){
    echo $1 | tee -a $logPath
}

function logfile(){
    echo $1 >> $logPath
}

function testConfig(){
    log "check relate config."
    if [ "$masterServerIp" == "masterIp" ];then
        log "ERROR: masterServerIp is not correct in the file app.properties"
        exit 1
    fi

    if [ "$slaveServerIp" == "slaveIp" ];then
        log "ERROR: slaveServerIp is not correct in the file app.properties"
        exit 1
    fi

    local grepIp=`ifconfig | grep $masterServerIp`

    ## it is must execute in the master server which you input in the app.properties file when first initiate ths replica set
    if [ "$grepIp" == "" ];then
        log "the current server is not the master server,  please login on to $masterServerIp and execute this script."
        exit 1
    fi

    ## test the connection
    echo "use admin" > test.js
    echo "rs.slaveOk()" >> test.js
    echo "show dbs" >> test.js

    cat test.js | mongo --host $slaveServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $slaveServerIp --port 27018 -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27018 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $masterServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    rm -rf test.js
    log "check relate config end."
}

function initTwoMemberReplicaSet(){
    log "init two member replica set."
    echo "use admin" > init.js
    echo "config = {_id:\"mongoSet\", members:[{_id:0,host:\"$masterServerIp:27017\",priority:2},{_id:1,host:\"$slaveServerIp:27017\",priority:1}]}" >> init.js
    echo "rs.initiate(config, {force:true})" >> init.js
    cat init.js | mongo -uroot -pabc123 --authenticationDatabase "admin" >> $logPath
    ## sleep 3 second for primary election.
    sleep 3s
    log "init two member replica set end."
}

function isElectPrimary(){
    log "check is the master node selected."
    local masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
    local count=0
    while [[ "$masterStr" == "" ]];do
            sleep 1s
            count=$((count+1))
            masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
            if [ "$masterStr" != "" ];then
                log "the master node is selected."
                break
            fi
            if [ "$count" == "30" ];then
                log "ERROR: arbiter configure timeout, please configure manually with 'rs.addArb(\"$slaveServerIp:27018\")' in $masterServerIp:27017"
                exit 1
            fi
    done
    log "check is the master node selected end."
}

function addArbiter(){
    log "add arbiter ..."
    mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.addArb(\"$slaveServerIp:27018\")"
    log "add arbiter end."
}

function chekArbiter(){
    log "check arbiter."
    local arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3| grep "stateStr"`
    if [ "$arbiterStr" != "ARBITER" ];then
        log "ERROR: abriter $slaveServerIp:27018: $arbiterStr, may be you need restart mongodb with 'systemctl restart mongod-arbiter' command in $slaveServerIp."
    fi
    log "check arbiter end."
}

logfile "**************************$nowtime**********************************"
testConfig
initTwoMemberReplicaSet
isElectPrimary
addArbiter
chekArbiter
log "init end..."

6、重新配置replica set

#!/bin/bash
basedir=$(cd "$(dirname "$0")";pwd)

source ./app.properties

nowtime=`date --date='0 days ago' "+%Y%m%d%H%M%S"`
daytime=`date --date='0 days ago' "+%Y%m%d"`

logPath=./log/init_repliaset_$daytime.log
function log(){
    echo $1 | tee -a $logPath
}

function logfile(){
    echo $1 >> $logPath
}

function testConfig(){
    log "check relate config."
    if [ "$masterServerIp" == "masterIp" ];then
        log "ERROR: masterServerIp is not correct in the file app.properties"
        exit 1
    fi

    if [ "$slaveServerIp" == "slaveIp" ];then
        log "ERROR: slaveServerIp is not correct in the file app.properties"
        exit 1
    fi

    ## get current replica set master node
    local currMasterNode=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | cut -d ":" -f 2 | sed 's/\"//g'`
    
    if [ "`ifconfig | grep $currMasterNode`" == "" ];then
        log "the current server has no mongodb master node, please login on to $currMasterNode execute this script."
        exit 1
    fi
    ## test the connection

    echo "use admin" > test.js
    echo "rs.slaveOk()" >> test.js
    echo "show dbs" >> test.js

    cat test.js | mongo --host $slaveServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $masterServerIp -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27017 in $slaveServerIp server can not connect."
            exit 1;
    fi

    cat test.js | mongo --host $slaveServerIp --port 27018 -uroot -pabc123 --authenticationDatabase "admin" >> $logPath

    if [ $? != 0 ];then
            log "mongodb instance which port is 27018 in $slaveServerIp server can not connect."
            ## 有时，当主从服务器都重启之后，有一个仲裁节点由于不在replica set中，但配置文件中却存在replica的配置，造成不能够正常启动，但reconfig replica set后，需要将该节点reconfig到replica set中，故此处不能因为不能连接就执行exit，但输出错误信息给出提示，即使服务未启动，
            ## 执行rs.addArb()也可以添加仲裁节点，但必须确定添加的仲裁节点会启动，否则即使通过rs.remove删除掉了该仲裁节点，但主节点日志中也会一直打印错误信息，
    fi
    
    rm -rf test.js
    log "check relate config end."
}

function initTwoMemberReplicaSet(){
    log "reconfig two member replica set."
    echo "use admin" > reconfig.js
    ## the _id must be the same as before
    ## find the master server id in previous replica set
    local masterId=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$masterServerIp:27017" -B 1|grep "_id" | cut -d ":" -f 2|sed 's/,//g'`
    ## find the slave server id in previous replica set
    local slaveId=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27017" -B 1|grep "_id" | cut -d ":" -f 2|sed 's/,//g'`
    echo "config = {_id:\"mongoSet\", "protocolVersion" : 1, members:[{_id:$masterId,host:\"$masterServerIp:27017\",priority:2},{_id:$slaveId,host:\"$slaveServerIp:27017\",priority:1}]}" >> reconfig.js
    echo "rs.reconfig(config, {force:true})" >> reconfig.js
    cat reconfig.js | mongo -uroot -pabc123 --authenticationDatabase "admin" >> $logPath
    ## sleep 3 second for primary election.
    sleep 3s
    log "reconfig two member replica set end."
}

function isElectPrimary(){
    log "check is the master node selected."
    local masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
    local count=0
    while [[ "$masterStr" == "" ]];do
            sleep 1s
            count=$((count+1))
            masterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "PRIMARY" -B 4|grep "name" | grep $masterServerIp`
            if [ "$masterStr" != "" ];then
                log "the master node is selected."
                break
            fi
            if [ "$count" == "30" ];then
                log "ERROR: arbiter configure timeout, please configure manually with 'rs.addArb(\"$slaveServerIp:27018\")' in $masterServerIp:27017"
                exit 1
            fi
    done
    log "check is the master node selected end."
}

function addArbiter(){
    log "remove arbiter in the master server."
    ## master server do not deploy arbiter, when reconfig, after the master node reselected you must specify the --post for delete or add
    local beforeArbiterIp=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "ARBITER" -B 4|grep "name" | cut -d ":" -f 2 | sed 's/\"//g'`
    if [ "`echo $masterServerIp | grep $beforeArbiterIp`" != "" ];then
        log "remove $masterServerIp:27018 in replica set."
        mongo -uroot -pabc123 --authenticationDatabase "admin" --host $masterServerIp --eval "rs.remove(\"$masterServerIp:27018\")"
    fi
    log "remove arbiter in the master server end."
    log "add arbiter ..."
    mongo  -uroot -pabc123 --authenticationDatabase "admin" --host $masterServerIp --eval "rs.addArb(\"$slaveServerIp:27018\")"
    log "add arbiter end."
}

function chekArbiter(){
    log "check arbiter."
    local arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3 | grep "ARBITER"`
    ## arbiter connect need time， set to 15 seconds
    local count=0
    while [[ "$arbiterStr" == "" ]];do
        sleep 1s
        count=$((count+1))
        arbiterStr=`mongo  -uroot -pabc123 --authenticationDatabase "admin" --eval "rs.status()" | grep "stateStr" -B 4|grep "$slaveServerIp:27018" -A 3 | grep "ARBITER"`
        if [ "$arbiterStr" != "" ];then
            log "the arbiter node is selected."
            break
        fi
        ## wait for 15 seconds
        if [ "$count" == "15" ];then
            log "ERROR: abriter $slaveServerIp:27018: $arbiterStr, may be you need restart mongodb with 'systemctl restart mongod-arbiter' command in $slaveServerIp."
            break
        fi
    done
    log "check arbiter end."
}

logfile "**************************$nowtime**********************************"
testConfig
initTwoMemberReplicaSet
isElectPrimary
addArbiter
chekArbiter


log "init end..."

脚本中通过rs.addArb()添加未启动或者不存在的arbiter节点，可以添加成功，但主节点日志中会一直打印该arbiter节点相关的错误信息，即使通过rs.remove()删除掉该arbiter节点，错误信息也会打印，该问题，重启主节点可以解决

7、配置文件app.properties

masterServerIp=masterIp
slaveServerIp=slaveIp

执行初始化和重新配置replica set之前需要确认app.properties中配置是正确的

8、无论是初始化replica set或者重新配置replica set，前几次都会出现not reachable/healthy，此时根据shell页面打印的提示，在对应的服务器上执行systemctl restart mongod-arbiter即可