Rancher高可用部署

机器准备

        30.128  四层代理服务器   节点上需要安装rke kubectl helm nginx

                     128节点需要和其它三个节点建立ssh互信实现免密登录 rke是通过ssh登录来远程操作其它节点的

                     128主机名设置为cf.rancher.com

        30.110  k8s-node1

        30.112  k8s-node2

        30.113  k8s-node3

                  110，112，113三个节点通过rke部署一个k8s集群 这个集群只用来运行Rancher Server实现Rancher Server的高可用

rke安装k8s

           安装     rke up --config=./rancher-cluster.yml

           卸载    rke remove --config=./rancher-cluster.yml

nodes:
  - address: 192.168.30.110
    user: admin
    role: [controlplane,worker,etcd]
  - address: 192.168.30.129
    user: admin
    role: [controlplane,worker,etcd]
  - address: 192.168.30.133
    user: admin
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

ingress:
  provider: nginx
  options:
    use-forwarded-headers: "true"
rancher-cluster.yml

    

    mkdir ~/.kube
    cp kube_config_rancher-cluster.yml ~/.kube/config

   

    表示k8s集群安装成功

Helm安装rancher

       helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable

       kubectl create namespace cattle-system

       kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml

       kubectl create namespace cert-manager

       helm repo add jetstack https://charts.jetstack.io

       helm repo update

       helm install  cert-manager jetstack/cert-manager  --namespace cert-manager  --version v0.15.0

       

       helm install rancher rancher-stable/rancher  --namespace cattle-system  --set hostname=cf.rancher.com

       nginx中证书对应的域名需要与hostname选项匹配,否则ingress将无法代理访问Rancher

      

      

 Nginx配置反向代理访问RancherUI

     在128节点上配置

      [admin conf.d]$ nginx -v
              nginx version: nginx/1.18.0

      1.配置主机名

      

      2.生成自签名证书

#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
    echo  ' --ssl-size: ssl加密位数，默认2048；'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM

if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_TRUSTED_DOMAIN})
    dns+=(${SSL_DOMAIN})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt
create_self_signed_cert.sh

    ./create_self-signed-cert.sh --ssl-domain=cf.rancher.com --ssl-trusted-ip=192.168.30.128 --ssl-size=2048 --ssl-date=3650

     3. 配置nginx文件/etc/nginx/conf.d

    upstream rancher {
        server 192.168.30.110:80;
        server 192.168.30.129:80;
        server 192.168.30.133:80;
    }

    map $http_upgrade $connection_upgrade {
        default Upgrade;
        ''      close;
    }

    server {
        listen 443 ssl http2;
        server_name cf.rancher.com;
        ssl_certificate /home/admin/certs/tls.crt;
        ssl_certificate_key /home/admin/certs/tls.key;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://rancher;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            # 这里将允许您在 Rancher UI 中打开命令行窗口时，窗口可以保留最多15分钟。没有这个参数时，默认值为1分钟，一分钟后在Rancher>中的shell会自动关闭。
            proxy_read_timeout 900s;
            proxy_buffering off;
        }
    }

    server {
        listen 80;
        server_name cf.rancher.com;
        return 301 https://$server_name$request_uri;
    }
nginx.conf

     4.重新启动nginx

windows客户端配置

      修改 c:\windows\system32\drivers\etc\hosts文件

      

     打开浏览器访问(只能用域名访问,不能用nginx服务器的IP访问)

      

      

为Agent Pod添加主机别名(/etc/hosts)

        如果没有内部DNS服务器而是通过添加/etc/hosts主机别名的方式指定的Rancher Server域名,那么不管通过哪种方式(自定义、导入、Host驱动等)创建K8S集群,K8S集群运行起来之后,因为cattle-cluster-agent Pod和cattle-node-agent pod无法通过DNS记录找到Rancher Server URL,最终导致无法通信

  

      1.在local集群中执行以下命令为Rancher Server容器配置hosts

kubectl --kubeconfig=$kubeconfig -n cattle-system \
    patch deployments rancher --patch '{
        "spec": {
            "template": {
                "spec": {
                    "hostAliases": [
                        {
                            "hostnames":
                            [
                                "xxx.cnrancher.com"
                            ],
                                "ip": "192.168.1.100"
                        }
                    ]
                }
            }
        }
    }'
View Code

      2.cattle-cluster-agent Pod和cattle-node-agent pod需要在LOCAL集群初始化之后才会部署,所以先通过Rancher Server URL访问Rancher Web UI进行初始化

      3.cattle-cluster-agent pod（local集群和业务集群均执行）

kubectl --kubeconfig=$kubeconfig -n cattle-system \
patch deployments cattle-cluster-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "demo.cnrancher.com"
                        ],
                            "ip": "192.168.1.100"
                    }
                ]
            }
        }
    }
}'
View Code

      4.cattle-node-agent pod（local集群和业务集群均执行）

kubectl --kubeconfig=$kubeconfig -n cattle-system \
patch  daemonsets cattle-node-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "xxx.rancher.com"
                        ],
                            "ip": "192.168.1.100"
                    }
                ]
            }
        }
    }
}'
View Code

自签名证书

   

  1.将CA证书以pem格式复制到名为ca-additional.pem的文件中,并用kubectl在命名空间cattle-system中创建tls-ca-additional密文

      kubectl  -n cattle-system create secret generic tls-ca-additional --from-file=ca-additional.pem

  2.在Helm部署Rancher Server时,添加以下参数

      helm install rancher rancher-stable/rancher  --namespace cattle-system  --set hostname=cf.rancher.com  --set additionalTrustedCAs=true

使用自定义证书的完整部署步骤

        1.在负载均衡器上执行一键生成ssl自签名证书脚本将自动生成tls.crt、tls.key、cacerts.pem三个文件。文件名称不能修改。ca文件名称必须是cacerts.pem

        

       2.使用kubectl在命名空间cattle-system中创建tls-ca secret 

          kubectl create namespace cattle-system

          kubectl -n cattle-system create secret generic tls-ca  --from-file=/home/admin/certs/cacerts.pem

      3.安装Rancher server

         helm install rancher rancher-stable/rancher \
         --namespace cattle-system \
         --set hostname=cf.rancher.com \
        --set ingress.tls.source=secret \
        --set privateCA=true

       4.修改pod中的hosts解析

      

kubectl  -n cattle-system \
patch deployments rancher --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "cf.rancher.com"
                        ],
                            "ip": "192.168.30.128"
                    }
                ]
            }
        }
    }
}'
View Code

       5.最终部署结果

    

     所有的pod都能正常运行起来

posted @ 2020-07-20 11:21  不懂123  阅读(5956)  评论(0)  编辑  收藏  举报