k8s

K8S基本概念：
Pod/Pod控制器
Name/Namespace
Label/Label选择器
Service/Ingress

核心组件：
配置存储中心->ETCD服务
主控(master)节点
kube-apiserver服务
kube-controller-manager服务
kube-scheduler服务
运算(node)节点
kube-kubelet服务
kube-proxy服务

CLI客户端：
kubectl

核心附件：
CNI网络插件->flannel/calico
服务器发现用插件->coredns
服务器暴露插件->traefik
GUI管理插件->Dashboard

k8s安装部署方式
二进制部署
kubeadmin部署

二进制部署:
基础工具组件安装
yum install -y epel-*
yum install -y wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils vim less

安装bind(容器需要使用到域名解析到IP)
yum install -y bind
部署参见DNS笔记

*证书签发*
使用cfssl签发
下载git代码，go语言环境
make编译
可以先生成模板文件(再在模板文件上编辑)
cfssl print-defaults csr > ca-csr.json
cfssl print-defaults config > ca-config.json
创建ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "k8s",
"OU": "DevOps"
}
]
}
签发CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

查看CA证书

cfssl certinfo -cert ca.pem

部署harbor(需要先安装docker-compose)
下载安装包
harbor-offline-installer-v2.3.4.tgz
cd 解压目录
解压后编辑harbor.yml(根据自己需求修改，密码都在里面)
http:
port: 8080
安装:
sh install.sh
启动：
docker-compose up -d
DNS内增加域名解析
使用harbor仓库：
docker tag ba6acccedd29 harbor.colinshi.com:8080/public/ubuntu:v12
docker login -u admin -p admin1234 {harborIp}
登入故障：
Error response from daemon: Get "https://harbor.colinshi.com:8080/v2/": http: server gave HTTP response to HTTPS client
编辑docker启动文件 /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=harbor.colinshi.com:8080
docker push harbor.colinshi.com/public/ubuntu:v12

安装ETCD(etcd-v3.5.1)
*证书签发*
创建etcd-peer-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"192.168.2.136",
"192.168.2.137",
"192.168.2.138",
"192.168.2.139"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai"
}
]
}
创建ca-config.json
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
},
"client": {
"expiry": "8760h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"server": {
"expiry": "438000h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
}
}
}
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssljson -bare etcd-peer
创建etcd用户
useradd etcd -M -s /sbin/nologin
解压etcd至/opt/
创建目录授权给etcd用户
mkdir -p /opt/etcd/certs /data/logs/etcd-server
cp ca.pem etcd-peer-key.pem etcd-peer.pem到etcd服务器/opt/etcd/certs/下
etcd启动文件(根据各个服务器修改变量)
#!/bin/bash
ETCD_NAME=etcd01
ETCD_DATA_DIR="/data/etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.2.137:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.2.137:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.2.137:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.2.137:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster1"
ETCD_INITIAL_CLUSTER="etcd01=http://192.168.2.137:2380,etcd02=http://192.168.2.138:2380,etcd03=http://192.168.2.139:2380"
ETCD_CA="/opt/etcd/certs/ca.pem"
ETCD_CERT_FILE="/opt/etcd/certs/etcd-peer.pem"
ETCD_KEY_FILE="/opt/etcd/certs/etcd-peer-key.pem"
nohup /opt/etcd/etcd --log-outputs /data/logs/etcd-server/etcd.log --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" --listen-peer-urls="${ETCD_LISTEN_PEER_URLS}" --advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}" --initial-cluster-token="${ETCD_INITIAL_CLUSTER_TOKEN}" --initial-cluster="${ETCD_INITIAL_CLUSTER}" --initial-cluster-state="${ETCD_INITIAL_CLUSTER_STATE}" --initial-advertise-peer-urls="${ETCD_INITIAL_ADVERTISE_PEER_URLS}" --trusted-ca-file="${ETCD_CA}" --cert-file="${ETCD_CERT_FILE}" --peer-cert-file="${ETCD_CERT_FILE}" --key-file="${ETCD_KEY_FILE}" --peer-key-file="${ETCD_KEY_FILE}" --peer-trusted-ca-file="${ETCD_CA}" --peer-client-cert-auth --client-cert-auth &
检测成功：
/opt/etcd/etcdctl --endpoints=https://192.168.2.137:2379,https://192.168.2.138:2379,https://192.168.2.139:2379 --cert="/opt/etcd/certs/etcd-peer.pem" --cacert="/opt/etcd/certs/ca.pem" --key="/opt/etcd/certs/etcd-peer-key.pem" endpoint health
注意事项：每台etcd的服务器配置有所有同，主要是name和urls等。各个key必须正确，如果重新生成key，建议清空etcd的data目录后重启etcd服务。

安装kubernetes
下载源码包安装
go1.7版本
make编译
kube-apiserver安装
*证书签发*
创建client-csr.json
{
"CN": "k8s-node",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "k8s",
"OU": "DevOps"
}
]
}
签发客户端证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes client-csr.json | cfssljson -bare client
创建apiserver-csr.json
{
"CN": "system:kube-apiserver",
"hosts":[
"127.0.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"192.168.2.136",
"192.168.2.137",
"192.168.2.138",
"192.168.2.139",
"localhost"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "kubernetes",
"OU": "DevOps"
}
]
}
签发api服务端证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssljson -bare apiserver
签发的证书CP到apiserver的证书目录下
kube-apiserver.conf启动配置文件
KUBE_API_ADDRESS="--advertise-address=192.168.2.137 --bind-address=0.0.0.0"
KUBE_API_PORT="--secure-port=6443"
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.2.138:2379,https://192.168.2.139:2379,https://192.168.2.137:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=192.168.3.0/24"
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"
KUBE_API_ARGS=" --allow-privileged=true \
--anonymous-auth=false \
--alsologtostderr \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kube-audit/audit.log \
--audit-policy-file=/opt/kubernetes/conf/audit.yaml \
--authorization-mode=RBAC \
--client-ca-file=/opt/kubernetes/certs/ca.pem \
--token-auth-file=/opt/kubernetes/conf/bootstrap-token.csv \
--enable-bootstrap-token-auth \
--enable-garbage-collector \
--enable-logs-handler \
--endpoint-reconciler-type=lease \
--etcd-cafile=/opt/kubernetes/certs/ca.pem \
--etcd-certfile=/opt/kubernetes/certs/etcd-peer.pem \
--etcd-keyfile=/opt/kubernetes/certs/etcd-peer-key.pem \
--etcd-compaction-interval=0s \
--event-ttl=168h0m0s \
--kubelet-certificate-authority=/opt/kubernetes/certs/ca.pem \
--kubelet-client-certificate=/opt/kubernetes/certs/client.pem \
--kubelet-client-key=/opt/kubernetes/certs/client-key.pem \
--kubelet-timeout=3s \
--runtime-config=api/all=true \
--service-node-port-range=30000-50000 \
--service-account-key-file=/opt/kubernetes/certs/ca-key.pem \
--service-account-signing-key-file=/opt/kubernetes/certs/ca-key.pem \
--service-account-issuer=kubernetes.default.svc \
--tls-cert-file=/opt/kubernetes/certs/apiserver.pem \
--tls-private-key-file=/opt/kubernetes/certs/apiserver-key.pem \
--v=2"
kube-apiserver.service systemctl文件
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf
User=root
ExecStart=/opt/kubernetes/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
systemctl start kube-apiserver
kube-controller-manager安装
*证书签发*
创建manager.json
{
"CN": "system:kube-controller-manager",
"hosts": [
"127.0.0.1",
"192.168.2.136",
"192.168.2.137",
"192.168.2.138",
"192.168.2.139",
"localhost"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "system:kube-controller-manager",
"OU": "System"
}
]
}
签发manager服务证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes manager.json| cfssljson -bare manager
签发的证书CP到manager服务的证书目录下
生成配置文件kube-controller-manager.kubeconfig
/opt/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/certs/ca.pem --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=kube-controller-manager.kubeconfig
配置客户端认证参数
/opt/kubernetes/bin/kubectl config set-credentials system:kube-controller-manager --client-certificate=/opt/kubernetes/certs/controller.pem --embed-certs=true --client-key=/opt/kubernetes/certs/controller-key.pem --kubeconfig=kube-controller-manager.kubeconfig
配置上下文参数
/opt/kubernetes/bin/kubectl config set-context system:kube-controller-manager@kubernetes --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
配置默认上下文
/opt/kubernetes/bin/kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=kube-controller-manager.kubeconfig

kube-controller-manager.conf文件
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 \
--authentication-kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \
--authorization-kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \
--bind-address=0.0.0.0 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/certs/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/certs/ca-key.pem \
--client-ca-file=/opt/kubernetes/certs/ca.pem \
--controllers=*,bootstrapsigner,tokencleaner \
--deployment-controller-sync-period=10s \
--experimental-cluster-signing-duration=87600h0m0s \
--enable-garbage-collector=true \
--kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \
--leader-elect=true \
--node-monitor-grace-period=20s \
--node-monitor-period=5s \
--pod-eviction-timeout=2m0s \
--requestheader-client-ca-file=/opt/kubernetes/certs/ca.pem \
--terminated-pod-gc-threshold=50 \
--tls-cert-file=/opt/kubernetes/certs/controller.pem \
--tls-private-key-file=/opt/kubernetes/certs/controller-key.pem \
--root-ca-file=/opt/kubernetes/certs/ca.pem \
--secure-port=10257 \
--service-cluster-ip-range=192.168.5.0/24 \
--service-account-private-key-file=/opt/kubernetes/certs/ca-key.pem \
--use-service-account-credentials=true \
--v=2"

kube-controller-manager.service文件
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf
User=root
ExecStart=/opt/kubernetes/bin/kube-controller-manager \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

kube-scheduler安装
*证书签发*
创建manager.json
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"192.168.2.136",
"192.168.2.137",
"192.168.2.138",
"192.168.2.139",
"localhost"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "system:kube-scheduler",
"OU": "System"
}
]
}
签发manager服务证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes scheduler.json| cfssljson -bare scheduler
签发的证书CP到manager服务的证书目录下
生成配置文件kube-controller-manager.kubeconfig
/opt/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/certs/ca.pem --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=kube-scheduler.kubeconfig
配置客户端认证参数
/opt/kubernetes/bin/kubectl config set-credentials system:kube-scheduler --client-certificate=/opt/kubernetes/certs/scheduler.pem --embed-certs=true --client-key=/opt/kubernetes/certs/scheduler-key.pem --kubeconfig=kube-scheduler.kubeconfig
配置上下文参数
/opt/kubernetes/bin/kubectl config set-context system:kube-scheduler@kubernetes --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
配置默认上下文
/opt/kubernetes/bin/kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=kube-scheduler.kubeconfig

kube-scheduler.conf文件
KUBE_SCHEDULER_ARGS="--address=127.0.0.1 \
--authentication-kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--authorization-kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--bind-address=0.0.0.0 \
--client-ca-file=/opt/kubernetes/certs/ca.pem \
--kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--requestheader-client-ca-file=/opt/kubernetes/certs/ca.pem \
--secure-port=10259 \
--leader-elect=true \
--tls-cert-file=/opt/kubernetes/certs/scheduler.pem \
--tls-private-key-file=/opt/kubernetes/certs/scheduler-key.pem \
--v=2"

kube-scheduler.service文件
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
User=root
ExecStart=/opt/kubernetes/bin/kube-scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

kubelet安装
*证书签发*
创建kubelet.json
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"192.168.2.136",
"192.168.2.137",
"192.168.2.138",
"192.168.2.139",
"localhost"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "system:kube-scheduler",
"OU": "System"
}
]
}
签发kubel服务证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes scheduler.json| cfssljson -bare scheduler
生成配置文件bootstrap.kubeconfig
/opt/kubernetes/bin/kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/certs/ca.pem --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=bootstrap.kubeconfig
配置客户端认证参数(如果使用token和自己设定一致)
#/opt/kubernetes/bin/kubectl config set-credentials kubelet-bootstrap --token=ee09e39a55334d5c7425d977ae075444 --kubeconfig=bootstrap.kubeconfig
/opt/kubernetes/bin/kubectl config set-credentials kubelet-bootstrap --token=ee09e39a55334d5c7425d977ae075444 --kubeconfig=bootstrap.kubeconfig
配置上下文参数
/opt/kubernetes/bin/kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig
配置默认上下文
/opt/kubernetes/bin/kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

kubelet.conf文件

kubelet.service文件
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
User=root
ExecStart=/opt/kubernetes/bin/kube-scheduler \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_MASTER \
$KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target