elasticsearch和redis集群搭建文档(手动分配slave节点)-电商系统
电商系统部署文档
下周就彻底离职了，工作还没着落，悲催
写在前面
系统优化参数
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_max_tw_buckets = 60000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 500000
net.core.somaxconn = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_keepalive_probes=5
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.ip_local_port_range = 1024 65535
vm.swappiness = 0
vm.max_map_count=262144
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.core.somaxconn = 32767
vm.overcommit_memory=1
echo never > /sys/kernel/mm/transparent_hugepage/enabled，并把这句话加入到/etc/rc.local中
1、elasticsearch部署
1.1 下载所需文件（3台机器都需要）
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.5.0/elasticsearch-analysis-ik-5.5.0.zip
1.2 安装es和ik分词软件（3台机器都需要）
yum install elasticsearch-5.5.0.rpm -y
#安装完es后，把ik分词软件elasticsearch-analysis-ik-5.5.0.zip拷贝至/usr/share/elasticsearch/plugins目录，然后执行下面命令
cd /usr/share/elasticsearch/plugins
unzip elasticsearch-analysis-ik-5.5.0.zip -d ik && rm -f elasticsearch-analysis-ik-5.5.0.zip
1.3 配置es
如果是yum安装的java的话，就不需要设置，如果是自定义安装的java就需要设置，不然的话，es会报找不到java的错误
下面是原文件/etc/sysconfig/elasticsearch
################################
# Elasticsearch
################################
# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch
# Elasticsearch Java path
#修改此处
JAVA_HOME=/usr/local/java/jdk1.8.0_152
# Elasticsearch configuration directory
CONF_DIR=/etc/elasticsearch
# Elasticsearch data directory
#DATA_DIR=/var/lib/elasticsearch
# Elasticsearch logs directory
#LOG_DIR=/var/log/elasticsearch
# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch
# Additional Java OPTS
#ES_JAVA_OPTS=
# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true
################################
# Elasticsearch service
################################
# SysV init.d
#
# When executing the init script, this user will be used to run the elasticsearch service.
# The default value is 'elasticsearch' and is declared in the init.d file.
# Note that this setting is only used by the init script. If changed, make sure that
# the configured user can read and write into the data, work, plugins and log directories.
# For systemd service, the user is usually configured in file /usr/lib/systemd/system/elasticsearch.service
#ES_USER=elasticsearch
#ES_GROUP=elasticsearch
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5
################################
# System properties
################################
# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65536
# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using Systemd, the LimitMEMLOCK property must be set
# in /usr/lib/systemd/system/elasticsearch.service
#MAX_LOCKED_MEMORY=unlimited
# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144
jvm配置
/etc/elasticsearch/jvm.options，修改下面所示：因为这3台机器还装了redis集群，所以分配为16G内存
-Xms16g
-Xmx16g
elasticsearch.yml配置
在该文件下面增加下面的配置即可，另外2台服务器修改node.name和network.host这2个字段
其中需要提前创建/data/elasticsearch，/var/log/elasticsearch目录，并授予elasticsearch.elasticsearch权限 chown -R elasticsearch.elasticsearch /data/elasticsearch && chown -R elasticsearch.elasticsearch /var/log/elasticsearch
## 157增加
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: shop-system
node.name: ser5-167.tech-idc.net
node.master: true
node.data: true
network.host: 10.80.5.167
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.80.5.167", "10.80.5.168","10.80.5.169"]
discovery.zen.minimum_master_nodes: 2
## 158增加
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: shop-system
node.name: ser5-168.tech-idc.net
node.master: true
node.data: true
network.host: 10.80.5.168
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.80.5.167", "10.80.5.168","10.80.5.169"]
discovery.zen.minimum_master_nodes: 2
## 159增加
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: shop-system
node.name: ser5-169.tech-idc.net
node.master: true
node.data: true
network.host: 10.80.5.169
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.80.5.167", "10.80.5.168","10.80.5.169"]
discovery.zen.minimum_master_nodes: 2
1.4 重启es设置开启自启动
systemctl start elasticsearch
systemctl enable elasticsearch
2、redis-cluster集群部署
2.1 下载所需文件
wget http://download.redis.io/releases/redis-3.2.11.tar.gz（所有机器下载）
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz （其中一台机器即可）
2.2 安装redis(所有机器执行)
tar xvf redis-3.2.11.tar.gz -C /usr/local
cd redis-3.2.11 && make && make install
安装完后验证
[root@ser5-167 elasticsearch]# redis-cli --version
redis-cli 3.2.11
2.3 创建目录
端口选择设置为7379-7380
mkdir -p /data/redis/{7379,7380}/{conf,data} #在服务器10.80.5.157创建
mkdir -p /data/redis/{7379,7380}/{conf,data} #在服务器10.80.5.158创建
mkdir -p /data/redis/{7379,7380}/{conf,data} #在服务器10.80.5.159创建
mkdir -p /var/log/redis && mkdir -p /var/run/redis （3台服务器上执行）
2.4 配置redis.conf
#157 7379端口
daemonize yes
pidfile "/var/run/redis/redis-7379.pid"
dir "/data/redis/7379/data"
port 7379
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.167
loglevel notice
logfile "/var/log/redis/redis-7379.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7379.conf"
appendonly yes
appendfilename "appendonly-7379.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
===================================
#157 7380端口
daemonize yes
pidfile "/var/run/redis/redis-7380.pid"
dir "/data/redis/7380/data"
port 7380
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.167
loglevel notice
logfile "/var/log/redis/redis-7380.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7380.conf"
appendonly yes
appendfilename "appendonly-7380.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
================================
# 158 7379端口
daemonize yes
pidfile "/var/run/redis/redis-7379.pid"
dir "/data/redis/7379/data"
port 7379
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.168
loglevel notice
logfile "/var/log/redis/redis-7379.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7379.conf"
appendonly yes
appendfilename "appendonly-7379.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
======================================
# 158 7380端口
daemonize yes
pidfile "/var/run/redis/redis-7380.pid"
dir "/data/redis/7380/data"
port 7380
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.168
loglevel notice
logfile "/var/log/redis/redis-7380.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7380.conf"
appendonly yes
appendfilename "appendonly-7380.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
=========================================
# 159 7379端口
daemonize yes
pidfile "/var/run/redis/redis-7379.pid"
dir "/data/redis/7379/data"
port 7379
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.169
loglevel notice
logfile "/var/log/redis/redis-7379.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7379.conf"
appendonly yes
appendfilename "appendonly-7379.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
====================================
# 159 8086端口
daemonize yes
pidfile "/var/run/redis/redis-7380.pid"
dir "/data/redis/7380/data"
port 7380
tcp-backlog 511
tcp-keepalive 60
bind 10.80.5.169
loglevel notice
logfile "/var/log/redis/redis-7380.log"
databases 16
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "node-7380.conf"
appendonly yes
appendfilename "appendonly-7380.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 80-100
auto-aof-rewrite-min-size 64mb
2.5 确认redis是否都起来了
#157执行
/usr/local/bin/redis-server /data/redis/7379/conf/redis.conf
/usr/local/bin/redis-server /data/redis/7380/conf/redis.conf
#158执行
/usr/local/bin/redis-server /data/redis/7379/conf/redis.conf
/usr/local/bin/redis-server /data/redis/7380/conf/redis.conf
#159执行
/usr/local/bin/redis-server /data/redis/7379/conf/redis.conf
/usr/local/bin/redis-server /data/redis/7380/conf/redis.conf
ps -ef|grep redis
2.6 安装ruby
tar xvf ruby-2.3.1.tar.gz
cd ruby-2.3.1 && ./configure --prefix=/usr/local/ruby && make && make install
#编译成功后拷贝命令到/usr/local/bin/目录
cd /usr/local/ruby/
cp bin/ruby /usr/local/bin
cp bin/gem /usr/local/bin
#验证是否okay
ruby --version
#安装rubygem redis的依赖
gem install redis
#拷贝redis-trib.rb命令至/usr/local/bin
cp /usr/local/reds-3.2.11/src/redis-trib.rb /usr/local/bin
cp /usr/local/bin/gem /bin/
2.7 初始化集群
由于这是3台机器搭建的集群，需要手动配置slave节点，不然的话，可能master节点和slave节点在同一台机器上，要是那台机器宕机的话，集群失效，所以先不创建副本
redis-trib.rb create --replicas 0 10.80.5.167:7379 10.80.5.168:7379 10.80.5.169:7379
然后再进去其中一台机器redis-cli -h 192.168.5.167 -p 7379执行cluster nodes，找到三个主的master id，master slave分配关系如下
10.80.5.167:7379(master) 10.80.5.168:7380(slave)
10.80.5.168:7379(master) 10.80.5.169:7380(slave)
10.80.5.169:7379(master) 10.80.5.167:7380(slave)
然后依次执行以下三条命令
redis-trib.rb add-node --slave --master-id ea5a83279cffe4c6b7ee7532903978601ff56fd81 10.80.5.168:7380 10.80.5.167:7379
redis-trib.rb add-node --slave --master-id 16efffee642caa4c3e16c137e0bdbe3b637e120dc 10.80.5.169:7380 10.80.5.168:7379
redis-trib.rb add-node --slave --master-id 413eddde18e9d27dd62c69bc413c3c63eb05867b0 10.80.5.167:7380 10.80.5.169:7379
2.8 坑
至此，redis集群就搭建好了，遇到坑安装rubygem redis的依赖的时候，一开始安装的是3.3.0版本，无法初始化集群，后来直接安装最新版的就好了，gem install redis
3台机器搭建的redis集群，redis-trib.rb自动创建集群的时候有可能master节点和slave节点在同一台机器上，所以需要手动分配，不知道有没有自动分配也能分散在不同机器上的方法
3、redis集群密码设置
方式一：修改所有Redis集群中的redis.conf文件加入：
masterauth passwd123
requirepass passwd123
说明：这种方式需要重新启动各节点
方式二：进入各个实例进行设置：
./redis-cli -c -p 7379 -h 10.80.5.167
config set masterauth passwd123
config set requirepass passwd123
auth passwd123
config rewrite
之后分别给各节点设置上密码。
注意：各个节点密码都必须一致，否则Redirected就会失败， 推荐这种方式，这种方式会把密码写入到redis.conf里面去，且不用重启。
用方式二修改密码，./redis-trib.rb check 10.80.5.157:7379执行时可能会报[ERR] Sorry, can't connect to node 10.80.5.157:7379，因为7379的redis.conf没找到密码配置。
2、设置密码之后如果需要使用redis-trib.rb的各种命令
解决办法：vim /usr/local/ruby/lib/ruby/gems/2.3.0/gems/redis-4.1.3/lib/redis/client.rb，然后修改password
class Client
DEFAULTS = {
:url => lambda { ENV["REDIS_URL"] },
:scheme => "redis",
:host => "127.0.0.1",
:port => 6379,
:path => nil,
:timeout => 5.0,
:password => "passwd123",
:db => 0,
:driver => nil,
:id => nil,
:tcp_keepalive => 0,
:reconnect_attempts => 1,
:inherit_socket => false
}
注意：client.rb路径可以通过find命令查找：find / -name 'client.rb'
参考
https://www.cnblogs.com/linjiqin/p/7462822.html
https://blog.csdn.net/u010533511/article/details/89390387
天天向上，空杯心态。
