k8s service

创建 ClusterIP 类型的 Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: metrics
      department: sales
  replicas: 3
  template:
    metadata:
      labels:
        app: metrics
        department: sales
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:2.0"

将此清单复制到名为 my-deployment.yaml 的文件，然后创建该 Deployment：

kubectl apply -f my-deployment.yaml

root@ubuntu:~/k8s_server# mv nsexec.c  my-deployment.yaml
root@ubuntu:~/k8s_server# kubectl apply -f my-deployment.yaml
deployment.apps/my-deployment created
root@ubuntu:~/k8s_server# kubectl get pods
NAME                             READY   STATUS              RESTARTS   AGE
kata-busybox                     1/1     Running             0          207d
kata-nginx                       1/1     Running             0          205d
my-deployment-68bdbbb5cc-2d2k6   0/1     ContainerCreating   0          7s
my-deployment-68bdbbb5cc-7fkwp   0/1     ContainerCreating   0          7s
my-deployment-68bdbbb5cc-lk5vr   0/1     ContainerCreating   0          7s
root@ubuntu:~/k8s_server#

root@ubuntu:~/k8s_server# kubectl get pods
NAME                             READY   STATUS             RESTARTS   AGE
kata-busybox                     1/1     Running            0          207d
kata-nginx                       1/1     Running            0          205d
my-deployment-68bdbbb5cc-2d2k6   0/1     ImagePullBackOff   0          16m
my-deployment-68bdbbb5cc-7fkwp   0/1     ImagePullBackOff   0          16m
my-deployment-68bdbbb5cc-lk5vr   0/1     ImagePullBackOff   0          16m
root@ubuntu:~/k8s_server# kubectl delete pod my-deployment-68bdbbb5cc-2d2k6  my-deployment-68bdbbb5cc-7fkwp  my-deployment-68bdbbb5cc-lk5vr
pod "my-deployment-68bdbbb5cc-2d2k6" deleted
pod "my-deployment-68bdbbb5cc-7fkwp" deleted
pod "my-deployment-68bdbbb5cc-lk5vr" deleted

root@ubuntu:~/k8s_server# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80

root@ubuntu:~/k8s_server# cat nginx-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: my-nginx

述规约将创建一个 Service，对应具有标签 run: my-nginx 的 Pod，目标 TCP 端口 80，并且在一个抽象的 Service 端口（targetPort：容器接收流量的端口；port：抽象的 Service 端口，可以使任何其它 Pod 访问该 Service 的端口）上暴露。查看 Service API 对象了解 Service 定义支持的字段列表。查看你的 Service 资源:

root@ubuntu:~/k8s_server# kubectl create -f nginx-svc.yaml 
service/my-nginx created
root@ubuntu:~/k8s_server# kubectl get svc my-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx ClusterIP 10.110.79.116 <none> 80/TCP 8s
root@ubuntu:~/k8s_server#

正如前面所提到的，一个 Service 由一组 backend Pod 组成。这些 Pod 通过 endpoints 暴露出来。 Service Selector 将持续评估，结果被 POST 到一个名称为 my-nginx 的 Endpoint 对象上。当 Pod 终止后，它会自动从 Endpoint 中移除，新的能够匹配上 Service Selector 的 Pod 将自动地被添加到 Endpoint 中。检查该 Endpoint，注意到 IP 地址与在第一步创建的 Pod 是相同的。

root@ubuntu:~/k8s_server# kubectl describe svc my-nginx
Name:              my-nginx
Namespace:         default
Labels:            run=my-nginx
Annotations:       <none>
Selector:          run=my-nginx
Type:              ClusterIP
IP:                10.110.79.116
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.0.19:80,10.244.0.20:80
Session Affinity:  None
Events:            <none>

root@ubuntu:~/k8s_server# kubectl get ep my-nginx
NAME       ENDPOINTS                       AGE
my-nginx   10.244.0.19:80,10.244.0.20:80   2m35s
root@ubuntu:~/k8s_server#

现在，能够从集群中任意节点上使用 curl 命令请求 Nginx Service <CLUSTER-IP>:<PORT> 。注意 Service IP 完全是虚拟的，它从来没有走过网络，如果对它如何工作的原理感到好奇，可以进一步阅读服务代理的内容。

root@ubuntu:~/k8s_server# telnet 10.110.79.116 80
Trying 10.110.79.116...
Connected to 10.110.79.116.
Escape character is '^]'.
^CConnection closed by foreign host.
root@ubuntu:~/k8s_server#

访问 Service

Kubernetes支持两种查找服务的主要模式: 环境变量和DNS。前者开箱即用，而后者则需要[CoreDNS集群插件] CoreDNS 集群插件.

说明：如果不需要服务环境变量（因为可能与预期的程序冲突，可能要处理的变量太多，或者仅使用DNS等），则可以通过在 pod spec 上将 enableServiceLinks 标志设置为 false 来禁用此模式。

环境变量

当 Pod 在 Node 上运行时，kubelet 会为每个活跃的 Service 添加一组环境变量。这会有一个顺序的问题。想了解为何，检查正在运行的 Nginx Pod 的环境变量（Pod 名称将不会相同）：

my-nginx-5dc4865748-jqx54        1/1     Running            0          18m
my-nginx-5dc4865748-pcrbg        1/1     Running            0          18m
root@ubuntu:~/k8s_server# kubectl exec my-nginx-5dc4865748-jqx54  -- printenv | grep SERVICE
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
root@ubuntu:~/k8s_server# kubectl exec my-nginx-5dc4865748-pcrbg  -- printenv | grep SERVICE
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
root@ubuntu:~/k8s_server#

暴露 Service

对我们应用的某些部分，可能希望将 Service 暴露在一个外部 IP 地址上。 Kubernetes 支持两种实现方式：NodePort 和 LoadBalancer。在上一段创建的 Service 使用了 NodePort，因此 Nginx https 副本已经就绪，如果使用一个公网 IP，能够处理 Internet 上的流量。

root@ubuntu:~/k8s_server# cat nginx-np.yaml 
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-np
  labels:
     run: my-nginx
spec:
  type: NodePort
  selector:
    app: my-nginx
  ports:
      # 默认情况下，为了方便起见，`targetPort` 被设置为与 `port` 字段相同的值。
    - port: 80
      targetPort: 80
      # 可选字段
      # 默认情况下，为了方便起见，Kubernetes 控制平面会从某个范围内分配一个端口号（默认：30000-32767）
      nodePort: 30007

root@ubuntu:~/k8s_server# kubectl get svc my-nginx -o yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2021-05-12T07:10:09Z"
  labels:
    run: my-nginx
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:run: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:run: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl
    operation: Update
    time: "2021-05-12T07:10:09Z"
  name: my-nginx
  namespace: default
  resourceVersion: "39404283"
  selfLink: /api/v1/namespaces/default/services/my-nginx
  uid: 1bc222be-11ed-47b5-8363-83bf9b9e0f2a
spec:
  clusterIP: 10.110.79.116
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: my-nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}




root@ubuntu:~/k8s_server# kubectl get svc my-nginx-np -o yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2021-05-12T07:24:39Z"
  labels:
    run: my-nginx
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:run: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:nodePort: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl
    operation: Update
    time: "2021-05-12T07:24:39Z"
  name: my-nginx-np
  namespace: default
  resourceVersion: "39406221"
  selfLink: /api/v1/namespaces/default/services/my-nginx-np
  uid: 56ee2fcd-c2dc-4455-b81f-515e9cd0bcce
spec:
  clusterIP: 10.102.202.85
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 30007
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: my-nginx
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

root@ubuntu:~/k8s_server# netstat -pan | grep 30007
tcp        0      0 0.0.0.0:30007           0.0.0.0:*               LISTEN      13725/kube-proxy    
root@ubuntu:~/k8s_server# kubectl get svc my-nginx
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
my-nginx   ClusterIP   10.110.79.116   <none>        80/TCP    18m
root@ubuntu:~/k8s_server# telnet 10.110.79.116 80
Trying 10.110.79.116...
Connected to 10.110.79.116.
Escape character is '^]'.
^CConnection closed by foreign host.
root@ubuntu:~/k8s_server# kubectl get svc my-nginx-np
NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
my-nginx-np   NodePort   10.102.202.85   <none>        80:30007/TCP   4m39s
root@ubuntu:~/k8s_server# telnet 10.102.202.85  30007
Trying 10.102.202.85...
^C

root@ubuntu:~/k8s_server# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        207d
my-nginx      ClusterIP   10.110.79.116   <none>        80/TCP         22m
my-nginx-np   NodePort    10.102.202.85   <none>        80:30007/TCP   7m55s

root@ubuntu:~/k8s_server# kubectl edit svc my-nginx-np

root@ubuntu:~/k8s_server# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        207d
my-nginx      ClusterIP   10.110.79.116   <none>        80/TCP         27m
my-nginx-np   NodePort    10.102.202.85   10.10.16.82   80:30007/TCP   13m
root@ubuntu:~/k8s_server#

节点没有外网IP

root@ubuntu:~/k8s_server# kubectl get nodes -o yaml | grep ExternalIP -C 1
root@ubuntu:~/k8s_server# kubectl get nodes --output wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ubuntu Ready master 207d v1.18.1 10.10.16.82 <none> Ubuntu 18.04.3 LTS 5.0.0-23-generic containerd://1.3.7
root@ubuntu:~/k8s_server#

ClusterIP实现

root@ubuntu:~/k8s_server# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP          207d
my-nginx      ClusterIP   10.110.79.116   <none>        8280/TCP         56m
my-nginx-np   NodePort    10.102.202.85   10.10.16.82   8180:32007/TCP   42m
root@ubuntu:~/k8s_server# telnet 10.110.79.116 8280
Trying 10.110.79.116...
Connected to 10.110.79.116.
Escape character is '^]'.
^CConnection closed by foreign host.

其他机器上添加路由访问 custerip 8280

root@cloud:~/cmd# ip r add 10.110.79.116/32 via 10.10.16.82
root@cloud:~/cmd# telnet 10.110.79.116  8280
Trying 10.110.79.116...
Connected to 10.110.79.116.
Escape character is '^]'.

^CConnection closed by foreign host.
root@cloud:~/cmd#

root@ubuntu:~/k8s_server# tcpdump -i enahisic2i0 tcp and  host 10.10.16.47 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:21:07.145276 IP 10.10.16.47.36654 > 10.110.79.116.8280: Flags [S], seq 1830641387, win 64240, options [mss 1460,sackOK,TS val 903079081 ecr 0,nop,wscale 7], length 0
16:21:07.145395 IP 10.110.79.116.8280 > 10.10.16.47.36654: Flags [S.], seq 941547984, ack 1830641388, win 64308, options [mss 1410,sackOK,TS val 3689254214 ecr 903079081,nop,wscale 7], length 0
16:21:07.145455 IP 10.10.16.47.36654 > 10.110.79.116.8280: Flags [.], ack 1, win 502, options [nop,nop,TS val 903079081 ecr 3689254214], length 0
^C
3 packets captured
11 packets received by filter
0 packets dropped by kernel
root@ubuntu:~/k8s_server#

root@ubuntu:~/k8s_server# tcpdump -i cni0 tcp and  host 10.10.16.47 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:22:14.896119 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [S.], seq 3152243450, ack 2069205458, win 64308, options [mss 1410,sackOK,TS val 3689321964 ecr 903146831,nop,wscale 7], length 0
16:22:30.142312 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [.], ack 3, win 503, options [nop,nop,TS val 3689337211 ecr 903162078], length 0
16:22:31.157763 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [.], ack 7, win 503, options [nop,nop,TS val 3689338226 ecr 903163093], length 0
16:22:31.157900 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [P.], seq 1:312, ack 7, win 503, options [nop,nop,TS val 3689338226 ecr 903163093], length 311: HTTP: HTTP/1.1 400 Bad Request
16:22:31.157957 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [F.], seq 312, ack 7, win 503, options [nop,nop,TS val 3689338226 ecr 903163093], length 0
16:22:31.158084 IP 10.244.0.20.http > 10.10.16.47.36658: Flags [.], ack 8, win 503, options [nop,nop,TS val 3689338226 ecr 903163094], length 0

root@ubuntu:~/k8s_server# conntrack -D

root@ubuntu:~/k8s_server# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP          207d
my-nginx      ClusterIP   10.110.79.116   <none>        8280/TCP         78m
my-nginx-np   NodePort    10.102.202.85   10.10.16.82   8180:32007/TCP   64m

root@ubuntu:~/k8s_server# netstat -pan | grep -E '8280|8180|32007'
tcp        0      0 10.10.16.82:8180        0.0.0.0:*               LISTEN      13725/kube-proxy    
tcp        0      0 0.0.0.0:32007           0.0.0.0:*               LISTEN      13725/kube-proxy

root@ubuntu:~/k8s_server# iptables -S -t nat | grep 8280
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.110.79.116/32 -p tcp -m comment --comment "default/my-nginx: cluster IP" -m tcp --dport 8280 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.110.79.116/32 -p tcp -m comment --comment "default/my-nginx: cluster IP" -m tcp --dport 8280 -j KUBE-SVC-BEPXDJBUHFCSYIC3

Chain KUBE-SVC-BEPXDJBUHFCSYIC3 (1 references)
target     prot opt source               destination         
KUBE-SEP-WURJJT7GEXOC32WR  all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ statistic mode random probability 0.50000000000
KUBE-SEP-UFGUNP44NJC7LB3J  all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */

Chain KUBE-SEP-UFGUNP44NJC7LB3J (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.0.20          0.0.0.0/0            /* default/my-nginx: */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ tcp DNAT [unsupported revision]

Chain KUBE-SEP-WURJJT7GEXOC32WR (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.0.19          0.0.0.0/0            /* default/my-nginx: */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ tcp DNAT [unsupported revision]

NodePort

root@ubuntu:~/k8s_server# cat nginx-np.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-np
  labels:
    run: my-nginx
spec:
  ports:
  - nodePort: 31199
    port: 8080
    protocol: TCP
    targetPort: 80
  selector:
    run: my-nginx
  type: NodePort
status:
  loadBalancer: {}

本地访问port=8080成功

root@ubuntu:~/k8s_server# telnet 10.99.1.231  8080
Trying 10.99.1.231...
Connected to 10.99.1.231.
Escape character is '^]'.
^CConnection closed by foreign host.

远程访问31199

root@ubuntu:~/k8s_server# iptables -S -t nat | grep 31199
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/my-nginx-np:" -m tcp --dport 31199 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/my-nginx-np:" -m tcp --dport 31199 -j KUBE-SVC-T5LLH4C2SBESAK4A
root@ubuntu:~/k8s_server#

通过node ip=10.10.16.82 nodeport=31199访问

远程访问8080

root@cloud:~/cmd# ip r add 10.99.1.231/32 via 10.10.16.82
root@cloud:~/cmd# telnet 10.10.16.82 8080
Trying 10.10.16.82...
telnet: Unable to connect to remote host: Connection refused
root@cloud:~/cmd# telnet 10.99.1.231  8080
Trying 10.99.1.231...
Connected to 10.99.1.231.
Escape character is '^]'.


^CConnection closed by foreign host.
root@cloud:~/cmd#

root@ubuntu:~/k8s_server# iptables -S -t nat | grep 8080
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.99.1.231/32 -p tcp -m comment --comment "default/my-nginx-np: cluster IP" -m tcp --dport 8080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.99.1.231/32 -p tcp -m comment --comment "default/my-nginx-np: cluster IP" -m tcp --dport 8080 -j KUBE-SVC-T5LLH4C2SBESAK4A
root@ubuntu:~/k8s_server#