基于Kubernetes部署nacos配置中心
写在前面的话
作为运维人员，本身对nacos配置中心可能不太熟悉。nacos既是配种中心，又是注册中心，相当于是eureka 和Apollo 的结合体。 应该是可以这么理解的。
Apollo的官方GitHub上，有关于如何基于k8s进行部署和配置的文档以及相应的镜像。但是看了nacos的官网文档之后，感觉一脸懵逼。
官方文档将MySQL也部署在k8s集群内部，建议不要这么做。
一：构建nacos镜像
由于之前的nacos镜像特别的大，而且官网呢，也没有任何的优化。感觉很糊弄。这里我们基于alpine的基础镜像重新初始化了jdk以及nacos的镜像。
项目地址：https://github.com/skymyyang/nacos-docker
这里我主要修改了build目录下的Dockerfile，别的没有动。
构建完成之后，我上传到了自己的阿里云镜像仓库。不为别的，就为拉取快。镜像地址：registry.cn-beijing.aliyuncs.com/skymyyang/nacos:1.3.0
公有的镜像仓库。大家都可以用。
完整的dockerfile如下：
FROM alpine:3.11.2 MAINTAINER skymyyang yang-li@live.cn ENV LANG=C.UTF-8 \ TZ=Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \ && echo $TZ > /etc/timezone \ && sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \ && apk update \ && apk add --update procps \ && apk add --no-cache ca-certificates ttf-dejavu tzdata tini bash ARG NACOS_VERSION=1.3.0 COPY nacos-server-${NACOS_VERSION}.tar.gz /home RUN apk add --no-cache openjdk8-jre \ && rm -rf /home/nacos-server-${NACOS_VERSION}.tar.gz /home/nacos/bin/* /home/nacos/conf/*.properties /home/nacos/conf/*.example /home/nacos/conf/nacos-mysql.sql # set environment ENV MODE="cluster" \ PREFER_HOST_MODE="ip"\ BASE_DIR="/home/nacos" \ CLASSPATH=".:/home/nacos/conf:$CLASSPATH" \ CLUSTER_CONF="/home/nacos/conf/cluster.conf" \ FUNCTION_MODE="all" \ JAVA_HOME="/usr/lib/jvm/java-1.8-openjdk" \ NACOS_USER="nacos" \ JAVA="/usr/lib/jvm/java-1.8-openjdk/bin/java" \ JVM_XMS="2g" \ JVM_XMX="2g" \ JVM_XMN="1g" \ JVM_MS="128m" \ JVM_MMS="320m" \ NACOS_DEBUG="n" \ TOMCAT_ACCESSLOG_ENABLED="false" WORKDIR /$BASE_DIR ADD bin/docker-startup.sh bin/docker-startup.sh ADD conf/application.properties conf/application.properties ADD init.d/custom.properties init.d/custom.properties # set startup log dir RUN mkdir -p logs \ && cd logs \ && touch start.out \ && ln -sf /dev/stdout start.out \ && ln -sf /dev/stderr start.out RUN chmod +x bin/docker-startup.sh \ && rm -rf /var/cache/apk/* EXPOSE 8848 ENTRYPOINT ["bin/docker-startup.sh"
二：配置nfs的存储类
官方文档说，nacos/nacos-peer-finder-plugin:1.0 这个插件是帮助Nacos集群进行动态扩容的，咱也不知道具体的原理，反正搞就对了，如果不需要动态扩容的话，我觉得根本就不需要搞。
关于k8s扩展存储，官方的驱动插件地址：https://github.com/kubernetes-incubator/external-storage
配置nfs的deployment rbac class即可。
deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.50.99 - name: NFS_PATH value: /nfsdata volumes: - name: nfs-client-root nfs: server: 192.168.50.99 path: /nfsdata
rbac.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io
class.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false"
三： 部署nacos
这里我们使用的是default名称空间，建议单独创建一个namespace
nacos-pvc-nfs.yml
--- apiVersion: v1 kind: Service metadata: name: nacos-headless labels: app: nacos annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" spec: ports: - port: 8848 name: server targetPort: 8848 clusterIP: None selector: app: nacos --- apiVersion: v1 kind: ConfigMap metadata: name: nacos-cm data: mysql.host: "192.168.50.99" mysql.db.name: "nacos_devtest" mysql.port: "3306" mysql.user: "nacos" mysql.password: "aixnacos" --- apiVersion: apps/v1 kind: StatefulSet metadata: name: nacos spec: serviceName: nacos-headless replicas: 3 template: metadata: labels: app: nacos annotations: pod.alpha.kubernetes.io/initialized: "true" spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - nacos topologyKey: "kubernetes.io/hostname" serviceAccountName: nfs-client-provisioner initContainers: - name: peer-finder-plugin-install image: nacos/nacos-peer-finder-plugin:1.0 imagePullPolicy: Always volumeMounts: - mountPath: "/home/nacos/plugins/peer-finder" name: plugindir containers: - name: nacos imagePullPolicy: IfNotPresent image: registry.cn-beijing.aliyuncs.com/skymyyang/nacos:1.3.0 resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 8848 name: client-port env: - name: NACOS_REPLICAS value: "3" - name: SERVICE_NAME value: "nacos-headless" - name: DOMAIN_NAME value: "cluster.local" - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: MYSQL_SERVICE_HOST valueFrom: configMapKeyRef: name: nacos-cm key: mysql.host - name: MYSQL_SERVICE_DB_NAME valueFrom: configMapKeyRef: name: nacos-cm key: mysql.db.name - name: MYSQL_SERVICE_PORT valueFrom: configMapKeyRef: name: nacos-cm key: mysql.port - name: MYSQL_SERVICE_USER valueFrom: configMapKeyRef: name: nacos-cm key: mysql.user - name: MYSQL_SERVICE_PASSWORD valueFrom: configMapKeyRef: name: nacos-cm key: mysql.password - name: NACOS_SERVER_PORT value: "8848" - name: NACOS_APPLICATION_PORT value: "8848" - name: PREFER_HOST_MODE value: "hostname" volumeMounts: - name: plugindir mountPath: /home/nacos/plugins/peer-finder - name: datadir mountPath: /home/nacos/data - name: logdir mountPath: /home/nacos/logs volumeClaimTemplates: - metadata: name: plugindir annotations: volume.beta.kubernetes.io/storage-class: "managed-nfs-storage" spec: accessModes: [ "ReadWriteMany" ] resources: requests: storage: 5Gi - metadata: name: datadir annotations: volume.beta.kubernetes.io/storage-class: "managed-nfs-storage" spec: accessModes: [ "ReadWriteMany" ] resources: requests: storage: 5Gi - metadata: name: logdir annotations: volume.beta.kubernetes.io/storage-class: "managed-nfs-storage" spec: accessModes: [ "ReadWriteMany" ] resources: requests: storage: 5Gi selector: matchLabels: app: nacos
创建ingress资源，将服务暴漏
nacos-ingress.yml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nacos-ingress namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: nacos-dev.aixbx.com http: paths: - backend: serviceName: nacos-headless servicePort: 8848
PS: 最后补充一下, 最后在进行MySQL初始化的时候, 如果出现以下报错:
Specified key was too long; max key length is 767 bytes
修改下列语句:
CREATE TABLE `permissions` (
`role` varchar(50) NOT NULL,
`resource` varchar(512) NOT NULL,
`action` varchar(8) NOT NULL,
UNIQUE INDEX `uk_role_permission` (`role`,`resource`,`action`) USING BTREE
)
中resource 字段为 varchar(255)
MySQL5.6中好像有这个问题, MySQL5.7 可以正常初始化.
