Kubernetes网络
kubernetes-Service
1.service存在的意义
1.防止破的失联（服务发现）
2.定义一组pod的访问策略（提供负载均衡）
2.pod与service的关系
1.通过lablel-selector相关联
2.通过service实现pod的负载均衡（TCP/UDP 4层）
负载均衡器类型
1.四层 传输层，基于IP和端口
2.七层 应用层，基于应用协议转发，例如http协议
3. service三种常用类型
1.Cluster：集群内部使用
2.Node port：对外暴露应用
3.Loadbalancer: 对外暴露应用，适合公有云（腾讯云、阿里云、亚马逊等）
ClusterIP：
默认分配一个稳定的IP地址，即VIP只能在集群内部访问（amespace内部pod）
Nodeport:
在每一个节点上启用一个端口来暴露服务，可以在集群外部访问，也会分配一个稳定内部集群IP地址。访问地址<nodeIP: nodeport>
Cluster端口 镜像中使用的端口
生成service的yaml
kubectl expose deploy web --port=80 --target-port=8080 --type=NodePort -o yaml --dry-run=client >service.yaml
cat server.yaml
apiVersion: v1 kind: Service metadata: labels: app: web name: web spec: ports: - port: 80 #clusterIP对应端口 protocol: TCP targetPort: 8080 #容器里面应用的端口 nodePort: 30012 #nodeport使用的节点端口 (配置的固定端口) selector: app: web type: NodePort
loadBalancer（只适合用云平台）
与nodeport类似，在每一个节点上启用一个端口来暴露服务，除此之外，Kubernetes会请求低层云平台删固定负载均衡器，将每个node(nodeIP):[nodePort]作为后端添加上
用户 --->域名 --->负载均衡器(公网IP) --->nodePort(所有节点) --->pod
每个部署一个应用，都需要手工在负载均衡器上配置，比较麻烦
4. service代码模式
service使用iptables或者ipvs进行转发的
kubeadm 方式修改ipvs模式
kubectl edit configmap kube-proxy -n kube-system ... 43 mode: "ipvs" ... kubectl delete pod kube-proxy-btz4p -n kube-system # 删除pod,会自动拉起 kubectl logs kube-proxy-wwqbh -n kube-system # 查看是否启用ipvs
注意：
1.kube-proxy配置文件以configmap方式存储
2.如果让所有节点生效，需要重建所有节点kube-proxy pod
二进制修改ipvs模式：
vim kube-proxy-config.yaml mode: ipvs scheduler: "rr" systemctl restart kube-proxy # 注：参考不同资料，文件名可能不同
安装ipvsadm工具进行查看ipvs
yum install -y ipvsadm lsmod |grep ip_vs #检测是否加载 modprobe ip_vs # 或者更新一下内核 yum update -y
查看规则
[root@k8s-node ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.17.0.1:32487 rr -> 10.244.84.176:80 Masq 1 0 0 TCP 192.168.10.111:30001 rr -> 10.244.247.3:8443 Masq 1 0 0
六城堡流程： 客户端 --> clusterIP(iptables/ipvs负载均衡规则) -->分布在各个节点pod
查看规则负载均衡规则：
iptables
iptables-save |grep <service-name>
ipvs
ipvsadm -L -n
iptables VS ipvs
| iptables |
1.灵活。功能强大 2.规则遍历匹配和更新，呈现性时延时 |
| ipvs |
1.工作在内核态，有更好的性能 2.调度算法丰富：rr、wrr、lc、wlc、ip hash |
5. service DNS名称
DNS服务监视kubernetes API, 为每一个service创建DNS记录用于域名解析
clusterIP A记录格式： <service-name>.<namespace-name>.svc,cluster.local
示例：my-svc.my-namespace.svc.cluster.local
小结：
1.采用Node Port对外暴露应用，前面加一个LB实现统一访问入口
2.优先使用IPVS代理模式
3.集群内应用采用DNS名称访问
-------------------------------------------------------
Kubernetes-Ingress
1. Ingress 为弥补Node Port不足而生
Nodeport存在的不足：
1.一个端口只能一个服务使用，端口需要提前规划
2.只支持4层负载均衡
2. Pod与ingress 的关系
1.通过service Controller实现的负载均衡
2.支持TCP、UDP 4层和HTTP 7层
3. Ingerss Controller部署
1.部署ingress Controller
2.创建ingress规则
ingress controller有很多方式实现，这里采用官方维护的nginx控制器
github代码托管地址：
https://github.com/kunernetes/ingress-nginx
下载到master节点
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
注意事项：
1.镜像地址修改/如果可以外网可以不修改（一般这个很难pull 下来）lizhenliang/nginx-ingress-controller:0.30.0
2.建议直接宿主机网络暴露： hostNetwork: true
设置/查看 污点
格式：kubectl label nodes <node-name> <label-key>=<label-value> 例如：kubectl label nodes k8s-node1 disktype=ssd 验证：kubectl get nodes --show-labels
修改mandatory.yaml
# 主要Deployment 资源下的内容 tolerations: # 由于我这里的边缘节点只有master一个节点，所有需要加上容忍 - operator: "Exists" nodeSelector: # 固定在边缘节点 #disktype: ssd kubernetes.io/os: linux #kubernetes.io/hostname: k8s-master hostNetwork: true # 将pod使用主机网命名空间 containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io #这里的标签和后面创建ingress 资源对象标签对应 securityContext:
其他主流控制器
1.Traefik: HTTP反向代理、负载均衡工具
2.Istio: 服务治理、控制入口流量
启动 Controller
kubectl apply -f mandatroy.yaml # 查看pod状态 kubectl get pods -n ingress-nginx -o wide kubectl describe pod xxxx -n ingress-nginx
创建服务发现
1.启动一个pod
# 导出yaml文件 或者直接创建pod kubectl create deploy java-demo --image=lizhenliang/java-demo -oyaml --dry-run=client > java-demo.yaml kubectl apply -f java-demo.yaml
2.为pod创建/绑定 service
kubectl expose deployment java-demo --port=80 --target-port=8080 --type=NodePort -o yaml --dry-run=client > svc.yaml # java-demo-svc.yaml 文件内容 apiVersion: v1 kind: Service metadata: labels: app: java-demo name: java-demo spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: app: java-demo
3.创建ingress规则（基于http的方式）
cat java-demo-ingress.yaml
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: snippet-test annotations: nginx.ingress.kubernetes.io/server-snippet: | # 对应的mandatroy文件中的标签
if ($http_user_agent ~* '(Android|iPhone)') { rewrite ^/(.*) http://m.baidu.com break; } spec: rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: java-demo port: number: 80
生效ingress资源
kubectl apply -f java-demo-ingress.yaml
4.扩容pod副本/ 并且查看ingress、service资源服务
# 扩容副本 kubectl scale deploy java-demo --replicas=2 # 查新是否被关联到一个负载均衡器里面 kubectl get ep # 查看ingress资源 kubectl get ingress
访问测试：
需要添加hosts解析
curl foo.bar.com
ingerss规则相当于在nginx里创建了一个基于域名虚拟主机。只不过是自动生成，不需人工干预
upstream web { server pod1; server pod2; } server { listen 80; server_name foo.bar.com; location / { cert key proxy_pass http://web; } }
用户--> 域名-->负载均衡器(公网IP) --> ingress controller(nginx实现负载均衡)-->pod
创建ingress规则 （基于https的方式）
https方式通过证书方式进行实现，证书分自签证书和CA证书机构
这里使用自签证书方式进行实现https加密传输
部署生成证书的工具
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
脚本方式生成证书
[root@k8s-master ~]# unzip ingress-https.zip [root@k8s-master ~]# cd ingress/ [root@k8s-master ingress]# ls certs.sh cfssl.sh ingress-controller.yaml ingress.yaml [root@k8s-master ingress]#
修改ceats.sh里面的域名，改成要使用的域名，然后生成证书
cd ingress/ && bash certs.sh #执行过程会将证书保存到secret中调用
查看保存的证书
kubectl get secret
配置https证书访问
cat ingress-https.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: tls-example-ingress spec: tls: - hosts: - foo.bar.com # 以上四行就是加https证书位置 secretName: foo.bar.com #对应保存证书的名称 rules: - host: foo.bar.com http: paths: - path: / backend: serviceName: java-demo servicePort: 80
删除http规则，不删除可能会对https有影响
kubectl delete -f ingress.yaml
生效https规则
kubectl apply -f ingress-https.yaml
kubectl get ing
测试访问 https://foo.bar.com
mandatory.yaml 原yaml的内容： 橙色部分为修改部分
apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount tolerations: # 由于我这里的边缘节点只有master一个节点，所有需要加上容忍 - operator: "Exists" nodeSelector: # 固定在边缘节点 #disktype: ssd kubernetes.io/os: linux #kubernetes.io/hostname: k8s-master hostNetwork: true # 将pod使用主机网命名空间 containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io #这里的标签和后面创建ingress 资源对象标签对应 securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown ---
