k8s部署nacos集群
1.Nacos 集群介绍
1.1 Nacos 简介
Nacos 是阿里巴巴开源的一个更易于构建云原生应用的动态服务发现、配置管理和服务管理平台。它提供了一组简单易用的特性集,帮助开发者快速实现动态服务发现、服务配置、服务元数据及流量管理,在微服务架构中发挥着至关重要的作用。
1.2 Nacos 集群的优势
高可用性:通过部署多个 Nacos 节点组成集群,当部分节点出现故障时,其他节点仍能正常工作,保证服务的持续可用性,避免单点故障对业务造成影响。
可扩展性:随着业务的发展,系统中的服务数量和配置数据量会不断增加。Nacos 集群可以方便地进行水平扩展,通过添加节点来提高系统的处理能力和存储容量。
数据一致性:Nacos 集群采用了分布式一致性算法,确保各个节点之间的数据一致性。在服务注册、配置管理等操作中,能够保证所有节点的数据实时同步,为应用提供准确、一致的服务信息和配置数据。
1.3 在 Kubernetes 中部署 Nacos 集群的意义
Kubernetes 是一个用于自动化部署、扩展和管理容器化应用程序的开源平台。将 Nacos 集群部署在 Kubernetes 中,可以充分利用 Kubernetes 的强大功能,如自动伸缩、负载均衡、故障转移等,实现 Nacos 集群的高效管理和运行。同时,借助 Kubernetes 的容器化技术,能够提高 Nacos 部署的灵活性和可移植性,降低运维成本
2.部署环境
IP | 节点 | 操作系统 | k8s版本 | nacos版本 | docker版本 |
172.16.4.85 | master1 | centos7.8 | 1.23.17 | 20.10.9 | |
172.16.4.86 | node1 | centos7.8 | 1.23.17 | 2.1.1 | 20.10.9 |
172.16.4.87 | node2 | centos7.8 | 1.23.17 | 2.1.1 | 20.10.9 |
172.16.4.89 | node3 | centos7.8 | 1.23.17 | 2.1.1 | 20.10.9 |
3.mysql连接地址
- mysql现在部署的主从,所以只需要连接主库的service即可,如果后边有mysql其他集群再修改
mysql-svc.default.svc.cluster.local 23306
4.nacos集群部署
4.1 nfs部署
- centos7安装nfs
yum install -y nfs-utils
- 创建nfs共享目录
mkdir -p /nfs_share/k8s/nacos/pv{1..3}
- nfs配置文件编辑
[root@localhost nacos]# cat /etc/exports
/nfs_share/k8s/nacos/pv1 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/nacos/pv2 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/nacos/pv3 *(rw,sync,no_subtree_check,no_root_squash)
- 启动nfs服务
# 启动 NFS 服务
systemctl start nfs-server
# 设置 NFS 服务在系统启动时自动启动
systemctl enable nfs-server
- 加载配置文件,并输出
[root@localhost nacos]# exportfs -r
[root@localhost nacos]# exportfs -v
/nfs_share/k8s/nacos/pv1
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/nacos/pv2
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/nacos/pv3
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
4.2 创建namespace
kubectl create namespace nacos
4.3 nacos部署pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: nacos-pv-1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: nacos-nfs-storage
nfs:
path: /nfs_share/k8s/nacos/pv1
server: 172.16.4.60
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nacos-pv-2
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: nacos-nfs-storage
nfs:
path: /nfs_share/k8s/nacos/pv2
server: 172.16.4.60
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nacos-pv-3
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: nacos-nfs-storage
nfs:
path: /nfs_share/k8s/nacos/pv3
server: 172.16.4.60
kubectl apply -f nacos-pv.yaml
4.4 nacos部署configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: nacos-config
namespace: nacos
data:
application.properties: |
# 应用端口
server.port=28848
PREFER_HOST_MODE=ip
# 数据库配置
spring.datasource.platform=mysql
db.num=1
db.url.0=jdbc:mysql://mysql-svc.default.svc.cluster.local:23306/unicom_nacos?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false
db.user=root
db.password=123456
# 鉴权配置
nacos.core.auth.enabled=true
nacos.core.auth.server.identity.key=ks_nacos
nacos.core.auth.server.identity.value=ks_nacos
nacos.core.auth.plugin.nacos.token.secret.key=ksNacosSecretKey012345678901234567890123456789012345678901234567890123456789
kubectl apply -f nacos-cm.yaml
4.5 nacos部署headless
apiVersion: v1
kind: Service
metadata:
name: nacos-headless
namespace: nacos
labels:
app: nacos
spec:
clusterIP: None # Headless Service
ports:
- port: 28848
name: server
- port: 27848
name: raft
selector:
app: nacos
kubectl apply -f nacos-headless.yaml
4.6 nacos部署svc
apiVersion: v1
kind: Service
metadata:
name: nacos-service
namespace: nacos
spec:
type: NodePort
ports:
- port: 28848 # Service 的 ClusterIP 端口(集群内访问)
targetPort: 28848 # 必须与 Nacos 容器实际监听的端口一致
nodePort: 30080 # 节点端口(范围 30000-32767)
selector:
app: nacos
kubectl apply -f nacos-svc-nodeport.yaml
4.7 nacos部署sts
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nacos
namespace: nacos
spec:
serviceName: nacos-headless
replicas: 3 # 至少3节点保证高可用
selector:
matchLabels:
app: nacos
template:
metadata:
labels:
app: nacos
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: [nacos]
topologyKey: kubernetes.io/hostname
containers:
- name: nacos
image: 172.16.4.17:8090/public/nacos-server:v2.1.1
ports:
- containerPort: 28848
name: server
- containerPort: 27848
name: raft
env:
- name: MODE
value: cluster
- name: PREFER_HOST_MODE
value: hostname
- name: NACOS_SERVERS
value: "nacos-0.nacos-headless:28848 nacos-1.nacos-headless:28848 nacos-2.nacos-headless:28848"
volumeMounts:
- name: nacos-config
mountPath: /home/nacos/conf/application.properties
subPath: application.properties
- name: nacos-data
mountPath: /home/nacos/data
volumes:
- name: nacos-config
configMap:
name: nacos-config
volumeClaimTemplates:
- metadata:
name: nacos-data
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: "nacos-nfs-storage" # 根据实际存储类填写
resources:
requests:
storage: 10Gi
kubectl apply -f nacos-ss.yaml
4.8 nacos状态查看
[root@master1 nacos-n6]# kubectl get pv | grep nacos
nacos-pv-1 10Gi RWX Retain Bound nacos/nacos-data-nacos-0 nacos-nfs-storage 76m
nacos-pv-2 10Gi RWX Retain Bound nacos/nacos-data-nacos-1 nacos-nfs-storage 76m
nacos-pv-3 10Gi RWX Retain Bound nacos/nacos-data-nacos-2 nacos-nfs-storage 76m
[root@master1 nacos-n6]# kubectl get pvc -n nacos
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nacos-data-nacos-0 Bound nacos-pv-1 10Gi RWX nacos-nfs-storage 76m
nacos-data-nacos-1 Bound nacos-pv-2 10Gi RWX nacos-nfs-storage 75m
nacos-data-nacos-2 Bound nacos-pv-3 10Gi RWX nacos-nfs-storage 75m
[root@master1 nacos-n6]# kubectl get cm -n nacos
NAME DATA AGE
kube-root-ca.crt 1 17h
nacos-config 1 108m
[root@master1 nacos-n6]# kubectl get svc -n nacos
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nacos-headless ClusterIP None <none> 28848/TCP,27848/TCP 92m
nacos-service ClusterIP 10.103.244.219 <none> 28848/TCP 91m
[root@master1 nacos-n6]# kubectl get sts -n nacos
NAME READY AGE
nacos 3/3 76m
[root@master1 nacos-n6]# kubectl get pods -n nacos -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nacos-0 1/1 Running 0 77m 10.244.135.36 node3 <none> <none>
nacos-1 1/1 Running 0 76m 10.244.104.32 node2 <none> <none>
nacos-2 1/1 Running 0 76m 10.244.166.157 node1 <none> <none>
4.9 nacos集群验证
- 通过nacos-0、nacos-1、nacos-2 pod的日志来查看集群是否创建成功
查看代码
[root@master1 nacos-n6]# kubectl logs -f nacos-0 -n nacos
Nacos is starting, you can docker logs your container
+ exec /usr/lib/jvm/java-1.8.0-openjdk/bin/java -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/nacos/logs/java_heapdump.hprof -XX:-UseLargePages -Dnacos.preferHostnameOverIp=true -Dnacos.member.list= -Djava.ext.dirs=/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext:/usr/lib/jvm/java-1.8.0-openjdk/lib/ext:/home/nacos/plugins/health:/home/nacos/plugins/cmdb:/home/nacos/plugins/mysql -Xloggc:/home/nacos/logs/nacos_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dnacos.home=/home/nacos -jar /home/nacos/target/nacos-server.jar --spring.config.additional-location=file:/home/nacos/conf/ --spring.config.name=application --logging.config=/home/nacos/conf/nacos-logback.xml --server.max-http-header-size=524288
,--.
,--.'|
,--,: : | Nacos 2.1.1
,`--.'`| ' : ,---. Running in cluster mode, All function modules
| : : | | ' ,'\ .--.--. Port: 28848
: | \ | : ,--.--. ,---. / / | / / ' Pid: 1
| : ' '; | / \ / \. ; ,. :| : /`./ Console: http://nacos-0.nacos-headless.nacos.svc.cluster.local:28848/nacos/index.html
' ' ;. ;.--. .-. | / / '' | |: :| : ;_
| | | \ | \__\/: . .. ' / ' | .; : \ \ `. https://nacos.io
' : | ; .' ," .--.; |' ; :__| : | `----. \
| | '`--' / / ,. |' | '.'|\ \ / / /`--' /
' : | ; : .' \ : : `----' '--'. /
; |.' | , .-./\ \ / `--'---'
'---' `--`---' `----'
2025-02-27 13:07:23,376 INFO The server IP list of Nacos is [nacos-0.nacos-headless:28848, nacos-1.nacos-headless:28848, nacos-2.nacos-headless:28848]
2025-02-27 13:07:24,377 INFO Nacos is starting...
2025-02-27 13:07:25,378 INFO Nacos is starting...
2025-02-27 13:07:26,378 INFO Nacos is starting...
2025-02-27 13:07:27,378 INFO Nacos is starting...
2025-02-27 13:07:28,379 INFO Nacos is starting...
2025-02-27 13:07:29,380 INFO Nacos is starting...
2025-02-27 13:07:30,380 INFO Nacos is starting...
2025-02-27 13:07:30,516 INFO Nacos started successfully in cluster mode. use external storage
[root@master1 nacos-n6]# kubectl logs -f nacos-1 -n nacos
Nacos is starting, you can docker logs your container
+ exec /usr/lib/jvm/java-1.8.0-openjdk/bin/java -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/nacos/logs/java_heapdump.hprof -XX:-UseLargePages -Dnacos.preferHostnameOverIp=true -Dnacos.member.list= -Djava.ext.dirs=/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext:/usr/lib/jvm/java-1.8.0-openjdk/lib/ext:/home/nacos/plugins/health:/home/nacos/plugins/cmdb:/home/nacos/plugins/mysql -Xloggc:/home/nacos/logs/nacos_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dnacos.home=/home/nacos -jar /home/nacos/target/nacos-server.jar --spring.config.additional-location=file:/home/nacos/conf/ --spring.config.name=application --logging.config=/home/nacos/conf/nacos-logback.xml --server.max-http-header-size=524288
,--.
,--.'|
,--,: : | Nacos 2.1.1
,`--.'`| ' : ,---. Running in cluster mode, All function modules
| : : | | ' ,'\ .--.--. Port: 28848
: | \ | : ,--.--. ,---. / / | / / ' Pid: 1
| : ' '; | / \ / \. ; ,. :| : /`./ Console: http://nacos-1.nacos-headless.nacos.svc.cluster.local:28848/nacos/index.html
' ' ;. ;.--. .-. | / / '' | |: :| : ;_
| | | \ | \__\/: . .. ' / ' | .; : \ \ `. https://nacos.io
' : | ; .' ," .--.; |' ; :__| : | `----. \
| | '`--' / / ,. |' | '.'|\ \ / / /`--' /
' : | ; : .' \ : : `----' '--'. /
; |.' | , .-./\ \ / `--'---'
'---' `--`---' `----'
2025-02-27 13:07:19,065 INFO The server IP list of Nacos is [nacos-0.nacos-headless:28848, nacos-1.nacos-headless:28848, nacos-2.nacos-headless:28848]
2025-02-27 13:07:20,066 INFO Nacos is starting...
2025-02-27 13:07:21,066 INFO Nacos is starting...
2025-02-27 13:07:22,067 INFO Nacos is starting...
2025-02-27 13:07:23,067 INFO Nacos is starting...
2025-02-27 13:07:24,067 INFO Nacos is starting...
2025-02-27 13:07:25,068 INFO Nacos is starting...
2025-02-27 13:07:26,047 INFO Nacos started successfully in cluster mode. use external storage
[root@master1 nacos-n6]# kubectl logs -f nacos-2 -n nacos
Nacos is starting, you can docker logs your container
+ echo 'Nacos is starting, you can docker logs your container'
+ exec /usr/lib/jvm/java-1.8.0-openjdk/bin/java -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/nacos/logs/java_heapdump.hprof -XX:-UseLargePages -Dnacos.preferHostnameOverIp=true -Dnacos.member.list= -Djava.ext.dirs=/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext:/usr/lib/jvm/java-1.8.0-openjdk/lib/ext:/home/nacos/plugins/health:/home/nacos/plugins/cmdb:/home/nacos/plugins/mysql -Xloggc:/home/nacos/logs/nacos_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dnacos.home=/home/nacos -jar /home/nacos/target/nacos-server.jar --spring.config.additional-location=file:/home/nacos/conf/ --spring.config.name=application --logging.config=/home/nacos/conf/nacos-logback.xml --server.max-http-header-size=524288
,--.
,--.'|
,--,: : | Nacos 2.1.1
,`--.'`| ' : ,---. Running in cluster mode, All function modules
| : : | | ' ,'\ .--.--. Port: 28848
: | \ | : ,--.--. ,---. / / | / / ' Pid: 1
| : ' '; | / \ / \. ; ,. :| : /`./ Console: http://nacos-2.nacos-headless.nacos.svc.cluster.local:28848/nacos/index.html
' ' ;. ;.--. .-. | / / '' | |: :| : ;_
| | | \ | \__\/: . .. ' / ' | .; : \ \ `. https://nacos.io
' : | ; .' ," .--.; |' ; :__| : | `----. \
| | '`--' / / ,. |' | '.'|\ \ / / /`--' /
' : | ; : .' \ : : `----' '--'. /
; |.' | , .-./\ \ / `--'---'
'---' `--`---' `----'
2025-02-27 13:07:15,113 INFO The server IP list of Nacos is [nacos-0.nacos-headless:28848, nacos-1.nacos-headless:28848, nacos-2.nacos-headless:28848]
2025-02-27 13:07:16,114 INFO Nacos is starting...
2025-02-27 13:07:17,114 INFO Nacos is starting...
2025-02-27 13:07:18,115 INFO Nacos is starting...
2025-02-27 13:07:19,115 INFO Nacos is starting...
2025-02-27 13:07:20,116 INFO Nacos is starting...
2025-02-27 13:07:21,116 INFO Nacos is starting...
2025-02-27 13:07:22,104 INFO Nacos started successfully in cluster mode. use external storage
- 通过nacos web控制台查看集群是否创建成功。nacos web 控制台访问地址通过nacos-svc-nodeport的端口30080进行访问:(master1)http://172.16.4.85:30080/nacos、(node1)http://172.16.4.86:30080/nacos、(node2)http://172.16.4.87:30080/nacos、(node3)http://172.16.4.89:30080/nacos,以上地址均可访问。
- 下图中nacos-2.nacos-headless.nacos.svc.cluster.local:28848 地址说明当前访问的是此节点
- 也可以通过web 控制台登陆其中一个节点,创建一些测试数据,在用其他节点登陆,查看是否有同步
4.10 遇到的问题
4.10.1 问题一:Nacos 启动失败 - Invalid boolean value 错误
- 启动日志报错:
Caused by: java.lang.IllegalArgumentException: Invalid boolean value [true # æ ¹æ®éæ±å¼å¯é´æ] - 原因分析
配置格式错误:nacos.core.auth.enabled=true 的配置值与注释写在同一行,导致 Spring 无法正确解析布尔值。 - 修正 ConfigMap 配置:
将布尔值参数与注释分开,确保参数单独成行 - 修改configmap
将configmap中 nacos.core.auth.enabled=true # 根据需求开启鉴权 修改为 # 根据需求开启鉴权 nacos.core.auth.enabled=true
- 重新运行cm和sts
#重新运行configmap kubectl apply -f nacos-cm.yaml -n nacos #重启 StatefulSet: kubectl rollout restart statefulset nacos -n nacos
4.10.2 问题二:Nacos 启动失败 - No DataSource set 错误
- 启动日志报错:
Caused by: java.lang.IllegalStateException: No DataSource set - 原因分析
数据库连接配置错误:MySQL 服务地址、端口、用户名、密码或权限配置错误。
数据库未初始化:未执行 Nacos 初始化 SQL 脚本。
网络策略限制:Nacos Pod 无法访问 MySQL 服务。 - 解决方案,修改configmap的格式,还有数据库名称,我这里是将数据库名称弄错了
apiVersion: v1 kind: ConfigMap metadata: name: nacos-config namespace: nacos data: application.properties: | NACOS_APPLICATION_PORT=28848 PREFER_HOST_MODE=ip SPRING_DATASOURCE_PLATFORM=mysql MYSQL_SERVICE_HOST=mysql-svc.default.svc.cluster.local MYSQL_SERVICE_DB_NAME=nacos_mysql MYSQL_SERVICE_PORT=23306 MYSQL_SERVICE_USER=root MYSQL_SERVICE_PASSWORD=123456 MYSQL_SERVICE_DB_PARAM=characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false nacos.core.auth.enabled=true # 根据需求开启鉴权 nacos.core.auth.server.identity.key=ks_nacos nacos.core.auth.server.identity.value=ks_nacos nacos.core.auth.plugin.nacos.token.secret.key=ksNacosSecretKey012345678901234567890123456789012345678901234567890123456789 #修改为———————————————————————————————————————————————————————————— apiVersion: v1 kind: ConfigMap metadata: name: nacos-config namespace: nacos data: application.properties: | # 应用端口 server.port=28848 PREFER_HOST_MODE=ip # 数据库配置 spring.datasource.platform=mysql db.num=1 db.url.0=jdbc:mysql://mysql-svc.default.svc.cluster.local:23306/unicom_nacos?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false db.user=root db.password=123456 # 鉴权配置 nacos.core.auth.enabled=true nacos.core.auth.server.identity.key=ks_nacos nacos.core.auth.server.identity.value=ks_nacos nacos.core.auth.plugin.nacos.token.secret.key=ksNacosSecretKey012345678901234567890123456789012345678901234567890123456789
5.nacos集群地址
- 节点内部地址
nacos-0.nacos-headless.nacos.svc.cluster.local:8848
nacos-1.nacos-headless.nacos.svc.cluster.local:8848
nacos-2.nacos-headless.nacos.svc.cluster.local:8848
- 集群内部地址
[root@master1 nacos-n6]# kubectl get svc -n nacos
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nacos-headless ClusterIP None <none> 28848/TCP,27848/TCP 3h29m
nacos-service NodePort 10.103.208.19 <none> 28848:30080/TCP 67m
#svc cluster ip 访问内部nacos集群
10.103.208.19:28848
#svc dns 访问内部集群,需要有dns支持nacos-service(svc name). nacos(命名空间). svc.cluster.local(默认后缀)
nacos-service.nacos.svc.cluster.local:28848
- 集群外部地址
#使用service 的nodeport映射端口
http://master1-ip:30080/nacos
http://node1-ip:30080/nacos
http://node2-ip:30080/nacos
http://node3-ip:30080/nacos
至此nacos集群就部署完了!!!
分类:
k8s
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· winform 绘制太阳,地球,月球 运作规律
· TypeScript + Deepseek 打造卜卦网站:技术与玄学的结合
· AI 智能体引爆开源社区「GitHub 热点速览」
· 写一个简单的SQL生成工具
· Manus的开源复刻OpenManus初探