mall-swarm微服务商城
官网地址：https://github.com/macrozheng/mall-swarm
https://www.macrozheng.com/ 参考地址：https://github.com/iKubernetes/learning-k8s/tree/master/Mall-MicroService https://gitee.com/mageedu/mall-microservice
mall ├── mall-common -- 工具类及通用代码模块 ├── mall-mbg -- MyBatisGenerator生成的数据库操作代码模块 ├── mall-auth -- 基于Spring Security Oauth2的统一的认证中心 ├── mall-gateway -- 基于Spring Cloud Gateway的微服务API网关服务 ├── mall-monitor -- 基于Spring Boot Admin的微服务监控中心 ├── mall-admin -- 后台管理系统服务 ├── mall-search -- 基于Elasticsearch的商品搜索系统服务 ├── mall-portal -- 移动端商城系统服务 ├── mall-demo -- 微服务远程调用测试服务 └── config -- 配置中心存储的配置
架构分析
mall-microservice项目实战： 微服务化的电子商城（Spring Cloud） 电子商城后台：mall-microservice 前台：移动端 商家前台：Vue, Web UI, mall-admin-web mall-microservice部署： 后端服务 Prometheus Server (http/https，且仅支持prometheus专用的格式的指标) mysql --> mysql exporter 部署形式：同一个Pod中的多个容器 mysql-server: 主容器 mysql-exporter: adapter容器 Nacos: 依赖于特有jar直接暴露prometheus格式的指标； ... MySQL Nacos --> MySQL 业务数据 --> MySQL Nacos： 服务注册中心 配置中心 ElasticSearch 商品搜索服务: 安装中文分词器 Image: ikubernetes/elasticsearch:7.17.7-ik 日志中心：Fluent-bit, Kibana fluent-bit: DaemonSet Kibana: Web UI Redis master/slave MongoDB ReplicaSet Cluster RabbitMQ MinIO SkyWalking 商城后台： mall-gateway mall-portal mall-serach ... 商城管理端： mall-admin-web --> API --> mall-gateway 移动端：用户前台
一、集群信息
操作系统：ubuntu 22.04 内核：5.15.0-105-generic containerd版本：1.6.31 kubernetes版本：1.28.9 网络插件：cilium cilium版本：1.15.3 pod负载均衡: metallb metallb版本：0.14.5
1.1、节点准备
k8s-cilium-master-01 172.16.88.61 4vcpu 8G 50G k8s-cilium-master-02 172.16.88.62 4vcpu 8G 50G k8s-cilium-master-03 172.16.88.63 4vcpu 8G 50G k8s-cilium-node-01 172.16.88.71 8vcpu 16G 50G k8s-cilium-node-02 172.16.88.72 8vcpu 16G 50G k8s-cilium-node-03 172.16.88.73 8vcpu 16G 50G k8s-cilium-node-04 172.16.88.74 8vcpu 16G 50G k8s-cilium-node-05 172.16.88.75 8vcpu 16G 50G
1.2、初始节点信息
通过ansible批量修改主机名
[root@cyh-dell-rocky9-02 ~]# cat /etc/ansible/hosts [vm1] 172.16.88.61 hostname=k8s-cilium-master-01 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.62 hostname=k8s-cilium-master-02 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.63 hostname=k8s-cilium-master-03 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.71 hostname=k8s-cilium-node-01 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.72 hostname=k8s-cilium-node-02 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.73 hostname=k8s-cilium-node-03 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.74 hostname=k8s-cilium-node-04 ansible_ssh_port=22 ansible_ssh_pass=redhat 172.16.88.75 hostname=k8s-cilium-node-05 ansible_ssh_port=22 ansible_ssh_pass=redhat [vm] 172.16.88.61 172.16.88.62 172.16.88.63 172.16.88.71 172.16.88.72 172.16.88.73 172.16.88.74 172.16.88.75 [root@cyh-dell-rocky9-02 ~]# [root@cyh-dell-rocky9-02 ~]# cat name.yml --- - hosts: vm1 remote_user: root tasks: - name: change name raw: "echo {{hostname|quote}} > /etc/hostname" - name: shell: hostname {{hostname|quote}} [root@cyh-dell-rocky9-02 ~]#
批量修改主机名：ansible-playbook ./name.yaml
在master推送ssh key
root@k8s-cilium-master-01:~#ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa root@k8s-cilium-master-01:~#for i in {62,63,71,72,73,74,75}; do sshpass -p 'redhat' ssh-copy-id -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -p 22 root@172.16.88.$i; done
初始化节点
cat >> /etc/hosts <<EOF 172.16.88.61 k8s-cilium-master-01 172.16.88.62 k8s-cilium-master-02 172.16.88.63 k8s-cilium-master-03 172.16.88.71 k8s-cilium-node-01 172.16.88.72 k8s-cilium-node-02 172.16.88.73 k8s-cilium-node-03 172.16.88.74 k8s-cilium-node-04 172.16.88.75 k8s-cilium-node-05 EOF swapoff -a sed -i 's@/swap.img@#/swap.img@g' /etc/fstab rm -fr /swap.img systemctl mask swap.img.swap ufw disable apt -y install apt-transport-https ca-certificates curl software-properties-common curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add - add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable" apt update && apt-get install containerd.io cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF modprobe overlay && modprobe br_netfilter # sysctl params required by setup, params persist across reboots cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF sysctl -p /etc/sysctl.d/k8s.conf
ansible 'vm' -m script -a "./base.sh"
1.3、安装配置Containerd
首先，运行如下命令打印默认并保存默认配置 ~# mkdir /etc/containerd ~# containerd config default > /etc/containerd/config.toml 接下来，编辑生成的配置文件，完成如下几项相关的配置： 修改containerd使用SystemdCgroup [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true 配置Containerd使用国内Mirror站点上的pause镜像及指定的版本 [plugins."io.containerd.grpc.v1.cri"] sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9" 配置Containerd使用国内的Image加速服务，以加速Image获取 [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://docker.mirrors.ustc.edu.cn", "https://registry.docker-cn.com"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"] endpoint = ["https://registry.aliyuncs.com/google_containers"]
完整config.toml文件
root@k8s-cilium-master-01:~# cat /etc/containerd/config.toml disabled_plugins = [] imports = [] oom_score = 0 plugin_dir = "" required_plugins = [] root = "/var/lib/containerd" state = "/run/containerd" temp = "" version = 2 [cgroup] path = "" [debug] address = "" format = "" gid = 0 level = "" uid = 0 [grpc] address = "/run/containerd/containerd.sock" gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 tcp_address = "" tcp_tls_ca = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0 [metrics] address = "" grpc_histogram = false [plugins] [plugins."io.containerd.gc.v1.scheduler"] deletion_threshold = 0 mutation_threshold = 100 pause_threshold = 0.02 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] device_ownership_from_security_context = false disable_apparmor = false disable_cgroup = false disable_hugetlb_controller = true disable_proc_mount = false disable_tcp_service = true drain_exec_sync_io_timeout = "0s" enable_selinux = false enable_tls_streaming = false enable_unprivileged_icmp = false enable_unprivileged_ports = false ignore_deprecation_warnings = [] ignore_image_defined_volumes = false max_concurrent_downloads = 3 max_container_log_line_size = 16384 netns_mounts_under_state_dir = false restrict_oom_score_adj = false sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9" selinux_category_range = 1024 stats_collect_period = 10 stream_idle_timeout = "4h0m0s" stream_server_address = "127.0.0.1" stream_server_port = "0" systemd_cgroup = false tolerate_missing_hugetlb_controller = true unset_seccomp_profile = "" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" conf_template = "" ip_pref = "" max_conf_num = 1 [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" disable_snapshot_annotations = true discard_unpacked_layers = false ignore_rdt_not_enabled_errors = false no_pivot = false snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 IoUid = 0 NoNewKeyring = false NoPivotRoot = false Root = "" ShimCgroup = "" SystemdCgroup = true [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "node" [plugins."io.containerd.grpc.v1.cri".registry] config_path = "" [plugins."io.containerd.grpc.v1.cri".registry.auths] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.headers] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://docker.mirrors.ustc.edu.cn", "https://registry.docker-cn.com"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"] endpoint = ["https://registry.aliyuncs.com/google_containers"] [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.internal.v1.tracing"] sampling_ratio = 1.0 service_name = "containerd" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] no_shim = false runtime = "runc" runtime_root = "" shim = "containerd-shim" shim_debug = false [plugins."io.containerd.runtime.v2.task"] platforms = ["linux/amd64"] sched_core = false [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.service.v1.tasks-service"] rdt_config_file = "" [plugins."io.containerd.snapshotter.v1.aufs"] root_path = "" [plugins."io.containerd.snapshotter.v1.btrfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.devmapper"] async_remove = false base_image_size = "" discard_blocks = false fs_options = "" fs_type = "" pool_name = "" root_path = "" [plugins."io.containerd.snapshotter.v1.native"] root_path = "" [plugins."io.containerd.snapshotter.v1.overlayfs"] mount_options = [] root_path = "" sync_remove = false upperdir_label = false [plugins."io.containerd.snapshotter.v1.zfs"] root_path = "" [plugins."io.containerd.tracing.processor.v1.otlp"] endpoint = "" insecure = false protocol = "" [proxy_plugins] [stream_processors] [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"] accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar" [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"] accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar+gzip" [timeouts] "io.containerd.timeout.bolt.open" = "0s" "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [ttrpc] address = "" gid = 0 uid = 0 root@k8s-cilium-master-01:~#
最后，同步到其他节点并重新启动containerd服务。
~# for i in {62,63,71,72,73,74,75};do scp /etc/containerd/config.toml root@172.16.88.$i:/etc/containerd/;done ~# systemctl daemon-reload && systemctl restart containerd
配置crictl客户端
cat > /etc/crictl.yaml <<EOF runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: true EOF
1.4、安装配置kubernetes
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list apt-get update && apt-get install -y kubelet kubeadm kubectl 设置kubelet kubeadm kubectl不允许被动态更新 apt-mark hold kubelet kubeadm kubectl
配置kubernetes 控制平面endpoint节点信息,方便做高可用
root@k8s-cilium-master-01:~# cat /etc/hosts 172.16.88.61 k8s-cilium-master-01 kubeapi.k8s.com root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# for i in {62,63,71,72,73,74,75};do scp /etc/hosts root@172.16.88.$i:/etc;done
1.5、初始化kubernetes
kubeadm init \ --control-plane-endpoint="kubeapi.k8s.com" \ --kubernetes-version=v1.28.9 \ --pod-network-cidr=10.244.0.0/16 \ --service-cidr=10.96.0.0/12 \ --image-repository=registry.aliyuncs.com/google_containers \ --cri-socket unix:///var/run/containerd/containerd.sock \ --upload-certs
初始化详情
root@k8s-cilium-master-01:~# kubeadm init \ --control-plane-endpoint="kubeapi.k8s.com" \ --kubernetes-version=v1.28.9 \ --pod-network-cidr=10.244.0.0/16 \ --service-cidr=10.96.0.0/12 \ --image-repository=registry.aliyuncs.com/google_containers \ --cri-socket unix:///var/run/containerd/containerd.sock \ --upload-certs [init] Using Kubernetes version: v1.28.9 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-cilium-master-01 kubeapi.k8s.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.88.61] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-cilium-master-01 localhost] and IPs [172.16.88.61 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-cilium-master-01 localhost] and IPs [172.16.88.61 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 9.005007 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: ba8510df5514ef3178b1d08293f7d4d7a725fdabacc1602a03587c1a8ee244ec [mark-control-plane] Marking the node k8s-cilium-master-01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-cilium-master-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 6i1bqc.gk7sf1alyiunuzry [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join kubeapi.k8s.com:6443 --token 6i1bqc.gk7sf1alyiunuzry \ --discovery-token-ca-cert-hash sha256:77df5c95946f6467da3c9fe2c5360887290f6322d04c40d525970eebea814655 \ --control-plane --certificate-key ba8510df5514ef3178b1d08293f7d4d7a725fdabacc1602a03587c1a8ee244ec Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join kubeapi.k8s.com:6443 --token 6i1bqc.gk7sf1alyiunuzry \ --discovery-token-ca-cert-hash sha256:77df5c95946f6467da3c9fe2c5360887290f6322d04c40d525970eebea814655 root@k8s-cilium-master-01:~#
同步key到master02、master03
master02、03创建etcd目录文件 mkdir -p /etc/kubernetes/pki/etcd/ 拷贝key scp /etc/kubernetes/pki/ca.* 172.16.88.62:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/ca.* 172.16.88.63:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/sa.* 172.16.88.62:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/sa.* 172.16.88.63:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/front-proxy-ca.* 172.16.88.62:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/front-proxy-ca.* 172.16.88.63:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/etcd/ca.* 172.16.88.62:/etc/kubernetes/pki/etcd/ scp /etc/kubernetes/pki/etcd/ca.* 172.16.88.63:/etc/kubernetes/pki/etcd/ # 将master2、master3加入集群，成为控制节点 kubeadm join kubeapi.k8s.com:6443 --token 6i1bqc.gk7sf1alyiunuzry \ --discovery-token-ca-cert-hash sha256:77df5c95946f6467da3c9fe2c5360887290f6322d04c40d525970eebea814655 \ --control-plane --certificate-key ba8510df5514ef3178b1d08293f7d4d7a725fdabacc1602a03587c1a8ee244ec 添加node节点 kubeadm join kubeapi.k8s.com:6443 --token 6i1bqc.gk7sf1alyiunuzry \ --discovery-token-ca-cert-hash sha256:77df5c95946f6467da3c9fe2c5360887290f6322d04c40d525970eebea814655
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@k8s-cilium-master-01:~# kubectl get node -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-cilium-master-01 NotReady control-plane 10m v1.28.9 172.16.88.61 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-master-02 NotReady control-plane 2m8s v1.28.9 172.16.88.62 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-master-03 NotReady control-plane 73s v1.28.9 172.16.88.63 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-node-01 NotReady <none> 74s v1.28.9 172.16.88.71 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-node-02 NotReady <none> 72s v1.28.9 172.16.88.72 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-node-03 NotReady <none> 67s v1.28.9 172.16.88.73 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-node-04 NotReady <none> 65s v1.28.9 172.16.88.74 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 k8s-cilium-node-05 NotReady <none> 62s v1.28.9 172.16.88.75 <none> Ubuntu 22.04.4 LTS 5.15.0-105-generic containerd://1.6.31 root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-66f779496c-j68dq 0/1 Pending 0 10m kube-system coredns-66f779496c-khhw5 0/1 Pending 0 10m kube-system etcd-k8s-cilium-master-01 1/1 Running 7 10m kube-system etcd-k8s-cilium-master-02 1/1 Running 0 2m25s kube-system etcd-k8s-cilium-master-03 1/1 Running 0 70s kube-system kube-apiserver-k8s-cilium-master-01 1/1 Running 7 10m kube-system kube-apiserver-k8s-cilium-master-02 1/1 Running 0 2m24s kube-system kube-controller-manager-k8s-cilium-master-01 1/1 Running 8 (2m14s ago) 10m kube-system kube-controller-manager-k8s-cilium-master-02 1/1 Running 0 2m24s kube-system kube-controller-manager-k8s-cilium-master-03 1/1 Running 0 18s kube-system kube-proxy-4l9s5 1/1 Running 0 91s kube-system kube-proxy-5fg8w 1/1 Running 0 84s kube-system kube-proxy-67z5q 1/1 Running 0 79s kube-system kube-proxy-8668p 1/1 Running 0 89s kube-system kube-proxy-d28rk 1/1 Running 0 10m kube-system kube-proxy-lztwh 1/1 Running 0 2m25s kube-system kube-proxy-qcl2x 1/1 Running 0 82s kube-system kube-proxy-xwp2p 1/1 Running 0 90s kube-system kube-scheduler-k8s-cilium-master-01 1/1 Running 8 (2m9s ago) 10m kube-system kube-scheduler-k8s-cilium-master-02 1/1 Running 0 2m24s kube-system kube-scheduler-k8s-cilium-master-03 1/1 Running 0 29s root@k8s-cilium-master-01:~#
1.6、安装cilium网络插件
root@k8s-cilium-master-01:~# wget https://github.com/cilium/cilium-cli/releases/download/v0.16.4/cilium-linux-amd64.tar.gz root@k8s-cilium-master-01:~# tar -xf cilium-linux-amd64.tar.gz && mv cilium /usr/local/bin/ root@k8s-cilium-master-01:~# cilium version cilium-cli: v0.16.4 compiled with go1.22.1 on linux/amd64 cilium image (default): v1.15.3 cilium image (stable): unknown cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: 1 errors \__/¯¯\__/ Operator: disabled /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: disabled \__/ ClusterMesh: disabled Containers: cilium cilium-operator Cluster Pods: 0/2 managed by Cilium Helm chart version: Errors: cilium cilium daemonsets.apps "cilium" not found status check failed: [daemonsets.apps "cilium" not found] root@k8s-cilium-master-01:~#
#配置cilium vxlan隧道模式，并开启ingress、hubble
cilium install \ --set kubeProxyReplacement=true \ --set ingressController.enabled=true \ --set ingressController.loadbalancerMode=dedicated \ --set ipam.mode=kubernetes \ --set routingMode=tunnel \ --set tunnelProtocol=vxlan \ --set ipam.operator.clusterPoolIPv4PodCIDRList=10.244.0.0/16 \ --set ipam.Operator.ClusterPoolIPv4MaskSize=24 \ --set hubble.enabled="true" \ --set hubble.listenAddress=":4244" \ --set hubble.relay.enabled="true" \ --set hubble.ui.enabled="true" \ --set prometheus.enabled=true \ --set operator.prometheus.enabled=true \ --set hubble.metrics.port=9665 \ --set hubble.metrics.enableOpenMetrics=true \ --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"
安装详情
root@k8s-cilium-master-01:~# cilium install \ --set kubeProxyReplacement=true \ --set ingressController.enabled=true \ --set ingressController.loadbalancerMode=dedicated \ --set ipam.mode=kubernetes \ --set routingMode=tunnel \ --set tunnelProtocol=vxlan \ --set ipam.operator.clusterPoolIPv4PodCIDRList=10.244.0.0/16 \ --set ipam.Operator.ClusterPoolIPv4MaskSize=24 \ --set hubble.enabled="true" \ --set hubble.listenAddress=":4244" \ --set hubble.relay.enabled="true" \ --set hubble.ui.enabled="true" \ --set prometheus.enabled=true \ --set operator.prometheus.enabled=true \ --set hubble.metrics.port=9665 \ --set hubble.metrics.enableOpenMetrics=true \ --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" ℹ️ Using Cilium version 1.15.3 🔮 Auto-detected cluster name: kubernetes 🔮 Auto-detected kube-proxy has been installed root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: OK \__/ ClusterMesh: disabled Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 DaemonSet cilium Desired: 8, Ready: 8/8, Available: 8/8 Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 8 hubble-relay Running: 1 cilium-operator Running: 1 hubble-ui Running: 1 Cluster Pods: 4/4 managed by Cilium Helm chart version: Image versions cilium quay.io/cilium/cilium:v1.15.3@sha256:da74ab61d1bc665c1c088dff41d5be388d252ca5800f30c7d88844e6b5e440b0: 8 hubble-relay quay.io/cilium/hubble-relay:v1.15.3@sha256:b9c6431aa4f22242a5d0d750c621d9d04bdc25549e4fb1116bfec98dd87958a2: 1 cilium-operator quay.io/cilium/operator-generic:v1.15.3@sha256:c97f23161906b82f5c81a2d825b0646a5aa1dfb4adf1d49cbb87815079e69d61: 1 hubble-ui quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1 root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 121m kube-system cilium-agent ClusterIP None <none> 9964/TCP 108s kube-system cilium-ingress LoadBalancer 10.106.144.101 <pending> 80:31461/TCP,443:30840/TCP 108s kube-system hubble-metrics ClusterIP None <none> 9665/TCP 108s kube-system hubble-peer ClusterIP 10.102.70.39 <none> 443/TCP 108s kube-system hubble-relay ClusterIP 10.106.193.178 <none> 80/TCP 108s kube-system hubble-ui ClusterIP 10.109.144.197 <none> 80/TCP 108s kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 121m root@k8s-cilium-master-01:~#
1.7、安装配置metallb
root@k8s-cilium-master-01:~# wget https://github.com/metallb/metallb/archive/refs/tags/v0.14.5.tar.gz root@k8s-cilium-master-01:~# tar -xf v0.14.5.tar.gz && cd /root/metallb-0.14.5/config/manifests root@k8s-cilium-master-01:~/metallb-0.14.5/config/manifests# kubectl apply -f metallb-native.yaml namespace/metallb-system created customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created customresourcedefinition.apiextensions.k8s.io/servicel2statuses.metallb.io created serviceaccount/controller created serviceaccount/speaker created role.rbac.authorization.k8s.io/controller created role.rbac.authorization.k8s.io/pod-lister created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/controller created rolebinding.rbac.authorization.k8s.io/pod-lister created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created configmap/metallb-excludel2 created secret/metallb-webhook-cert created service/metallb-webhook-service created deployment.apps/controller created daemonset.apps/speaker created validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created root@k8s-cilium-master-01:~/metallb-0.14.5/config/manifests# root@k8s-cilium-master-01:~/metallb-0.14.5/config/manifests# kubectl get pod -n metallb-system NAME READY STATUS RESTARTS AGE controller-56bb48dcd4-ctfhp 1/1 Running 0 2m32s speaker-2fht9 1/1 Running 0 2m32s speaker-bjvfw 1/1 Running 0 2m32s speaker-d7kx9 1/1 Running 0 2m32s speaker-dbxtf 1/1 Running 0 2m32s speaker-fdw9s 1/1 Running 0 2m32s speaker-kxmfk 1/1 Running 0 2m32s speaker-p8qwv 1/1 Running 0 2m32s speaker-t975r 1/1 Running 0 2m32s root@k8s-cilium-master-01:~/metallb-0.14.5/config/manifests#
配置metallb
root@k8s-cilium-master-01:~# git clone https://github.com/iKubernetes/learning-k8s.git root@k8s-cilium-master-01:~/learning-k8s/MetalLB# cat metallb-ipaddresspool.yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: localip-pool namespace: metallb-system spec: addresses: - 172.16.88.90-172.16.88.100 autoAssign: true avoidBuggyIPs: true root@k8s-cilium-master-01:~/learning-k8s/MetalLB# root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl apply -f metallb-ipaddresspool.yaml ipaddresspool.metallb.io/localip-pool created root@k8s-cilium-master-01:~/learning-k8s/MetalLB# root@k8s-cilium-master-01:~/learning-k8s/MetalLB# cat metallb-l2advertisement.yaml apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: localip-pool-l2a namespace: metallb-system spec: ipAddressPools: - localip-pool interfaces: - enp1s0 root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl apply -f metallb-l2advertisement.yaml l2advertisement.metallb.io/localip-pool-l2a created root@k8s-cilium-master-01:~/learning-k8s/MetalLB# root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl get ipaddresspool -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES localip-pool true true ["172.16.88.90-172.16.88.100"] root@k8s-cilium-master-01:~/learning-k8s/MetalLB#
查看cilium ingress loadbalanecer
root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 144m kube-system cilium-agent ClusterIP None <none> 9964/TCP 24m kube-system cilium-ingress LoadBalancer 10.106.144.101 172.16.88.90 80:31461/TCP,443:30840/TCP 24m kube-system hubble-metrics ClusterIP None <none> 9665/TCP 24m kube-system hubble-peer ClusterIP 10.102.70.39 <none> 443/TCP 24m kube-system hubble-relay ClusterIP 10.106.193.178 <none> 80/TCP 24m kube-system hubble-ui ClusterIP 10.109.144.197 <none> 80/TCP 24m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 144m metallb-system metallb-webhook-service ClusterIP 10.98.152.230 <none> 443/TCP 7m55s root@k8s-cilium-master-01:~/learning-k8s/MetalLB#
启用hubble ui
root@k8s-cilium-master-01:~/learning-k8s/MetalLB# cilium hubble enable --ui root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl get ingressclass NAME CONTROLLER PARAMETERS AGE cilium cilium.io/ingress-controller <none> 32m root@k8s-cilium-master-01:~/learning-k8s/MetalLB# root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl create ingress hubble-ui --rule='hubble.k8s.com/*=hubble-ui:80' --class='cilium' -n kube-system --dry-run=client -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: hubble-ui namespace: kube-system spec: ingressClassName: cilium rules: - host: hubble.k8s.com http: paths: - backend: service: name: hubble-ui port: number: 80 path: / pathType: Prefix status: loadBalancer: {} root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl create ingress hubble-ui --rule='hubble.k8s.com/*=hubble-ui:80' --class='cilium' -n kube-system ingress.networking.k8s.io/hubble-ui created root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl get ingress -n kube-system NAME CLASS HOSTS ADDRESS PORTS AGE hubble-ui cilium hubble.k8s.com 172.16.88.91 80 65s root@k8s-cilium-master-01:~/learning-k8s/MetalLB# root@k8s-cilium-master-01:~/learning-k8s/MetalLB# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 157m kube-system cilium-agent ClusterIP None <none> 9964/TCP 37m kube-system cilium-ingress LoadBalancer 10.106.144.101 172.16.88.90 80:31461/TCP,443:30840/TCP 37m kube-system cilium-ingress-hubble-ui LoadBalancer 10.108.201.184 172.16.88.91 80:31462/TCP,443:30975/TCP 95s kube-system hubble-metrics ClusterIP None <none> 9665/TCP 37m kube-system hubble-peer ClusterIP 10.102.70.39 <none> 443/TCP 37m kube-system hubble-relay ClusterIP 10.106.193.178 <none> 80/TCP 37m kube-system hubble-ui ClusterIP 10.109.144.197 <none> 80/TCP 37m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 157m metallb-system metallb-webhook-service ClusterIP 10.98.152.230 <none> 443/TCP 21m root@k8s-cilium-master-01:~/learning-k8s/MetalLB#
添加本地域名解析
编辑本地hosts
172.16.88.91 hubble.k8s.com
浏览器访问：http://hubble.k8s.com
1.8、安装openebs存储环境
# apply this yaml kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml root@k8s-cilium-master-01:~# kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml namespace/openebs created serviceaccount/openebs-maya-operator created clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created customresourcedefinition.apiextensions.k8s.io/blockdevices.openebs.io created customresourcedefinition.apiextensions.k8s.io/blockdeviceclaims.openebs.io created configmap/openebs-ndm-config created daemonset.apps/openebs-ndm created deployment.apps/openebs-ndm-operator created deployment.apps/openebs-ndm-cluster-exporter created service/openebs-ndm-cluster-exporter-service created daemonset.apps/openebs-ndm-node-exporter created service/openebs-ndm-node-exporter-service created deployment.apps/openebs-localpv-provisioner created storageclass.storage.k8s.io/openebs-hostpath created storageclass.storage.k8s.io/openebs-device created root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# kubectl get pod -n openebs NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-6787b599b9-l9czz 1/1 Running 0 17m openebs-ndm-2lgz5 1/1 Running 0 17m openebs-ndm-6ptjr 1/1 Running 0 17m openebs-ndm-77nwx 1/1 Running 0 17m openebs-ndm-cluster-exporter-7bfd5746f4-52mnw 1/1 Running 0 17m openebs-ndm-h9qj6 1/1 Running 0 17m openebs-ndm-node-exporter-2dl7w 1/1 Running 0 17m openebs-ndm-node-exporter-4b6kc 1/1 Running 0 17m openebs-ndm-node-exporter-dbxgt 1/1 Running 0 17m openebs-ndm-node-exporter-dk6w2 1/1 Running 0 17m openebs-ndm-node-exporter-xc44x 1/1 Running 0 17m openebs-ndm-operator-845b8858db-t2j25 1/1 Running 0 17m openebs-ndm-t7cqx 1/1 Running 0 17m root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# kubectl get sc -A NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 18m openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 18m root@k8s-cilium-master-01:~#
指定默认存储类
root@k8s-cilium-master-01:~# kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' storageclass.storage.k8s.io/openebs-hostpath patched root@k8s-cilium-master-01:~# kubectl get sc -A NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 106m openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 106m root@k8s-cilium-master-01:~#
nfs存储类安装(可选)
服务端： apt-get install -y nfs-kernel-server mkdir -pv /data/nfs echo "/data/nfs 172.16.88.0/24(rw,fsid=0,async,no_subtree_check,no_auth_nlm,insecure,no_root_squash)" >> /etc/exports exportfs -arv 客户端： apt-get install -y nfs-common k8s部署csi-driver-nfs [root@cyh-dell-rocky9-02 ~]# ansible 'vm' -m shell -a "apt-get install -y nfs-common"ansible 'vm' -m shell -a "apt-get install -y nfs-common" root@k8s-cilium-master-01:~# git clone https://github.com/kubernetes-csi/csi-driver-nfs.git cd /root/csi-driver-nfs/deploy kubectl apply -f v4.6.0/ 创建存储类 vi nfs-sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-csi provisioner: nfs.csi.k8s.io parameters: server: 172.16.88.81 share: /data/nfs reclaimPolicy: Delete volumeBindingMode: Immediate kubectl apply -f nfs-sc.yaml
1.9、安装Prometheus监控
部署Prometheus监控系统
cd /root/learning-k8s/Mall-MicroService/infra-services-with-prometheus
kubectl apply -f namespace.yaml
kubectl apply -f prometheus-server/ -n prom
# ll -h prometheus-server/ total 40K drwxr-xr-x 2 root root 4.0K Apr 29 06:59 ./ drwxr-xr-x 8 root root 4.0K Apr 28 08:00 ../ -rw-r--r-- 1 root root 6.0K Apr 28 08:00 prometheus-cfg.yaml -rw-r--r-- 1 root root 1.9K Apr 28 08:00 prometheus-deploy.yaml -rw-r--r-- 1 root root 685 Apr 29 06:59 prometheus-ingress.yaml -rw-r--r-- 1 root root 728 Apr 28 08:00 prometheus-rbac.yaml -rw-r--r-- 1 root root 7.3K Apr 28 08:00 prometheus-rules.yaml -rw-r--r-- 1 root root 357 Apr 28 08:00 prometheus-svc.yaml #
---
apiVersion: v1
kind: Namespace
metadata:
name: prom
--- kind: ConfigMap apiVersion: v1 metadata: labels: app: prometheus name: prometheus-config namespace: prom data: prometheus.yml: |- global: scrape_interval: 15s scrape_timeout: 10s evaluation_interval: 15s rule_files: - /etc/prometheus/prometheus.rules alerting: alertmanagers: - scheme: http static_configs: - targets: - "alertmanager.prom.svc:9093" scrape_configs: # Scape config for prometheus node exporters. # - job_name: 'node-exporter' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_endpoints_name] regex: 'node-exporter' action: keep # Scrape config for kube-state-metrics. # - job_name: 'kube-state-metrics' static_configs: - targets: ['kube-state-metrics.prom.svc:8080'] # Scrape config for kubernetes apiservers. # - job_name: 'kubernetes-apiservers' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https # Scrape config for nodes (kubelet). # - job_name: 'kubernetes-nodes' scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics # Example scrape config for pods. # # The relabeling allows the actual pod scrape endpoint to be configured via the # following annotations: # # * `prometheus.io/scrape`: Only scrape pods that have a value of `true` # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the # - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name # Scrape config for Kubelet cAdvisor. # - job_name: 'kubernetes-cadvisor' scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor # Scrape config for service endpoints. # # The relabeling allows the actual service scrape endpoint to be configured # via the following annotations: # # * `prometheus.io/scrape`: Only scrape services that have a value of `true` # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need # to set this to `https` & most likely set the `tls_config` of the scrape config. # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. # * `prometheus.io/port`: If the metrics are exposed on a different port to the # service then set this appropriately. # - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name
--- apiVersion: apps/v1 kind: Deployment metadata: name: prometheus-server namespace: prom labels: app: prometheus spec: replicas: 1 selector: matchLabels: app: prometheus component: server #matchExpressions: #- {key: app, operator: In, values: [prometheus]} #- {key: component, operator: In, values: [server]} template: metadata: labels: app: prometheus component: server annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9090' spec: serviceAccountName: prometheus containers: - name: prometheus image: prom/prometheus:v2.50.1 imagePullPolicy: Always command: - prometheus - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention=720h ports: - containerPort: 9090 protocol: TCP resources: limits: memory: 2Gi volumeMounts: - mountPath: /etc/prometheus/prometheus.yml name: prometheus-config subPath: prometheus.yml - mountPath: /etc/prometheus/prometheus.rules name: prometheus-rules subPath: prometheus.rules - mountPath: /prometheus/ name: prometheus-storage-volume volumes: - name: prometheus-config configMap: name: prometheus-config items: - key: prometheus.yml path: prometheus.yml mode: 0644 - name: prometheus-rules configMap: name: prometheus-rules items: - key: prometheus.rules path: prometheus.rules mode: 0644 - name: prometheus-storage-volume emptyDir: {}
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus namespace: prom labels: app: prometheus annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: 'cilium' rules: - host: prom.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus port: number: 9090 - host: prometheus.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus port: number: 9090
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: prom --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: prom
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-rules namespace: prom data: prometheus.rules: | groups: - name: kube-state-metrics rules: - alert: KubernetesNodeReady expr: kube_node_status_condition{condition="Ready",status="true"} == 0 for: 10m labels: severity: critical annotations: summary: Kubernetes Node ready (instance {{ $labels.instance }}) description: "Node {{ $labels.node }} has been unready for a long time\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesMemoryPressure expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1 for: 2m labels: severity: critical annotations: summary: Kubernetes memory pressure (instance {{ $labels.instance }}) description: "{{ $labels.node }} has MemoryPressure condition\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesDiskPressure expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1 for: 2m labels: severity: critical annotations: summary: Kubernetes disk pressure (instance {{ $labels.instance }}) description: "{{ $labels.node }} has DiskPressure condition\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesNetworkUnavailable expr: kube_node_status_condition{condition="NetworkUnavailable",status="true"} == 1 for: 2m labels: severity: critical annotations: summary: Kubernetes network unavailable (instance {{ $labels.instance }}) description: "{{ $labels.node }} has NetworkUnavailable condition\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesOutOfCapacity expr: sum by (node) ((kube_pod_status_phase{phase="Running"} == 1) + on(uid) group_left(node) (0 * kube_pod_info{pod_template_hash=""})) / sum by (node) (kube_node_status_allocatable{resource="pods"}) * 100 > 90 for: 2m labels: severity: warning annotations: summary: Kubernetes out of capacity (instance {{ $labels.instance }}) description: "{{ $labels.node }} is out of capacity\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesContainerOomKiller expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) == 1 for: 0m labels: severity: warning annotations: summary: Kubernetes container oom killer (instance {{ $labels.instance }}) description: "Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesPersistentvolumeclaimPending expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1 for: 2m labels: severity: warning annotations: summary: Kubernetes PersistentVolumeClaim pending (instance {{ $labels.instance }}) description: "PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is pending\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesVolumeOutOfDiskSpace expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes * 100 < 10 for: 2m labels: severity: warning annotations: summary: Kubernetes Volume out of disk space (instance {{ $labels.instance }}) description: "Volume is almost full (< 10% left)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesPersistentvolumeError expr: kube_persistentvolume_status_phase{phase=~"Failed|Pending", job="kube-state-metrics"} > 0 for: 0m labels: severity: critical annotations: summary: Kubernetes PersistentVolume error (instance {{ $labels.instance }}) description: "Persistent volume is in bad state\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesPodNotHealthy expr: min_over_time(sum by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"})[15m:1m]) > 0 for: 0m labels: severity: critical annotations: summary: Kubernetes Pod not healthy (instance {{ $labels.instance }}) description: "Pod has been in a non-ready state for longer than 15 minutes.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesPodCrashLooping expr: increase(kube_pod_container_status_restarts_total[1m]) > 3 for: 2m labels: severity: warning annotations: summary: Kubernetes pod crash looping (instance {{ $labels.instance }}) description: "Pod {{ $labels.pod }} is crash looping\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesApiServerErrors expr: sum(rate(apiserver_request_total{job="apiserver",code=~"^(?:5..)$"}[1m])) / sum(rate(apiserver_request_total{job="apiserver"}[1m])) * 100 > 3 for: 2m labels: severity: critical annotations: summary: Kubernetes API server errors (instance {{ $labels.instance }}) description: "Kubernetes API server is experiencing high error rate\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesApiClientErrors expr: (sum(rate(rest_client_requests_total{code=~"(4|5).."}[1m])) by (instance, job) / sum(rate(rest_client_requests_total[1m])) by (instance, job)) * 100 > 1 for: 2m labels: severity: critical annotations: summary: Kubernetes API client errors (instance {{ $labels.instance }}) description: "Kubernetes API client is experiencing high error rate\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesClientCertificateExpiresNextWeek expr: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 7*24*60*60 for: 0m labels: severity: warning annotations: summary: Kubernetes client certificate expires next week (instance {{ $labels.instance }}) description: "A client certificate used to authenticate to the apiserver is expiring next week.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" - alert: KubernetesClientCertificateExpiresSoon expr: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 24*60*60 for: 0m labels: severity: critical annotations: summary: Kubernetes client certificate expires soon (instance {{ $labels.instance }}) description: "A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
--- apiVersion: v1 kind: Service metadata: name: prometheus namespace: prom annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9090' labels: app: prometheus spec: type: NodePort ports: - port: 9090 targetPort: 9090 nodePort: 30090 protocol: TCP selector: app: prometheus component: server
部署node-exporter
kubectl apply -f node-exporter/
# ll -h node_exporter/ total 16K drwxr-xr-x 2 root root 4.0K Apr 28 08:00 ./ drwxr-xr-x 8 root root 4.0K Apr 28 08:00 ../ -rw-r--r-- 1 root root 782 Apr 28 08:00 node-exporter-ds.yaml -rw-r--r-- 1 root root 383 Apr 28 08:00 node-exporter-svc.yaml #
apiVersion: apps/v1 kind: DaemonSet metadata: name: prometheus-node-exporter namespace: prom labels: app: prometheus component: node-exporter spec: selector: matchLabels: app: prometheus component: node-exporter template: metadata: name: prometheus-node-exporter labels: app: prometheus component: node-exporter spec: tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master containers: - image: prom/node-exporter:v1.7.0 #image: registry.magedu.com/prom/node-exporter:v1.5.0 name: prometheus-node-exporter ports: - name: prom-node-exp containerPort: 9100 hostPort: 9100 hostNetwork: true hostPID: true
apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: 'true' name: prometheus-node-exporter namespace: prom labels: app: prometheus component: node-exporter spec: clusterIP: None ports: - name: prometheus-node-exporter port: 9100 protocol: TCP selector: app: prometheus component: node-exporter type: ClusterIP
部署Kube-State-Metrics
部署kube-state-metrics，监控Kubernetes集群的服务指标。
kubectl apply -f kube-state-metrics/
# ll -h kube-state-metrics/ total 20K drwxr-xr-x 2 root root 4.0K Apr 28 08:00 ./ drwxr-xr-x 8 root root 4.0K Apr 28 08:00 ../ -rw-r--r-- 1 root root 523 Apr 28 08:00 kube-state-metrics-deploy.yaml -rw-r--r-- 1 root root 1007 Apr 28 08:00 kube-state-metrics-rbac.yaml -rw-r--r-- 1 root root 318 Apr 28 08:00 kube-state-metrics-svc.yaml #
apiVersion: apps/v1 kind: Deployment metadata: name: kube-state-metrics namespace: prom spec: replicas: 1 selector: matchLabels: app: kube-state-metrics template: metadata: labels: app: kube-state-metrics spec: serviceAccountName: kube-state-metrics containers: - name: kube-state-metrics image: gcmirrors/kube-state-metrics:v1.9.5 #image: registry.magedu.com/gcmirrors/kube-state-metrics-amd64:v1.7.1 ports: - containerPort: 8080
--- apiVersion: v1 kind: ServiceAccount metadata: name: kube-state-metrics namespace: prom --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-state-metrics rules: - apiGroups: [""] resources: ["nodes", "pods", "services", "resourcequotas", "replicationcontrollers", "limitranges", "persistentvolumeclaims", "persistentvolumes", "namespaces", "endpoints"] verbs: ["list", "watch"] - apiGroups: ["apps"] resources: ["daemonsets", "deployments", "replicasets", "statefulsets"] verbs: ["list", "watch"] - apiGroups: ["batch"] resources: ["cronjobs", "jobs"] verbs: ["list", "watch"] - apiGroups: ["autoscaling"] resources: ["horizontalpodautoscalers"] verbs: ["list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-state-metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: prom
apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: 'true' prometheus.io/port: '8080' name: kube-state-metrics namespace: prom labels: app: kube-state-metrics spec: ports: - name: kube-state-metrics port: 8080 protocol: TCP selector: app: kube-state-metrics
部署AlertManager
部署AlertManager，为Prometheus-Server提供可用的告警发送服务。
kubectl apply -f alertmanager/
# ll -h alertmanager/ total 28K drwxr-xr-x 2 root root 4.0K Apr 28 08:00 ./ drwxr-xr-x 8 root root 4.0K Apr 28 08:00 ../ -rw-r--r-- 1 root root 1.3K Apr 28 08:00 alertmanager-cfg.yaml -rw-r--r-- 1 root root 1.2K Apr 28 08:00 alertmanager-deployment.yaml -rw-r--r-- 1 root root 274 Apr 28 08:00 alertmanager-service.yaml -rw-r--r-- 1 root root 4.2K Apr 28 08:00 alertmanager-templates-cfg.yaml #
# Maintainer: MageEdu <mage@magedu.com> # --- kind: ConfigMap apiVersion: v1 metadata: name: alertmanager-config namespace: prom data: config.yml: |- global: templates: - '/etc/alertmanager-templates/*.tmpl' route: receiver: ops-team group_by: ['alertname', 'priority'] group_wait: 10s repeat_interval: 30m routes: - receiver: ops-manager match: severity: critical group_wait: 10s repeat_interval: 30m receivers: - name: ops-team email_configs: - to: ops@magedu.com send_resolved: false from: monitoring@magedu.com smarthost: smtp.magedu.com:25 require_tls: false headers: subject: "{{ .Status | toUpper }} {{ .CommonLabels.env }}:{{ .CommonLabels.cluster }} {{ .CommonLabels.alertname }}" html: '{{ template "email.default.html" . }}' - name: ops-manager webhook_configs: - url: http://prometheus-webhook-dingtalk:8060/dingtalk/webhook1/send send_resolved: true inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'job'] ---
apiVersion: apps/v1 kind: Deployment metadata: name: alertmanager namespace: prom spec: replicas: 1 selector: matchLabels: app: alertmanager template: metadata: name: alertmanager labels: app: alertmanager spec: containers: - name: alertmanager image: prom/alertmanager:v0.27.0 #image: registry.magedu.com/prom/alertmanager:v0.25.0 args: - "--config.file=/etc/alertmanager/config.yml" - "--storage.path=/alertmanager" ports: - name: alertmanager containerPort: 9093 resources: requests: cpu: 500m memory: 500M limits: cpu: 1 memory: 1Gi volumeMounts: - name: config-volume mountPath: /etc/alertmanager - name: templates-volume mountPath: /etc/alertmanager-templates - name: alertmanager mountPath: /alertmanager volumes: - name: config-volume configMap: name: alertmanager-config - name: templates-volume configMap: name: alertmanager-templates - name: alertmanager emptyDir: {}
apiVersion: v1 kind: Service metadata: name: alertmanager namespace: prom annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9093' spec: selector: app: alertmanager type: LoadBalancer ports: - port: 9093 targetPort: 9093
# Maintainer: MageEdu <mage@magedu.com> # --- apiVersion: v1 kind: ConfigMap metadata: name: alertmanager-templates namespace: prom data: dingtalk_template.tmpl: "{{ define \"dingtalk.default.message\" }}\n{{- if gt (len .Alerts.Firing) 0 -}}\n{{- range $index, $alert := .Alerts -}}\n{{- if eq $index 0 }}\n========= 监控报警 ========= \n告警状态：{{ .Status }} \n告警级别：{{ .Labels.severity }} \n告警类型：{{ $alert.Labels.alertname }} \n故障主机: {{ $alert.Labels.instance }} \ \n告警主题: {{ $alert.Annotations.summary }} \n告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}} \n触发阈值：{{ .Annotations.value }} \n故障时间: {{ ($alert.StartsAt.Add 28800e9).Format \"2006-01-02 15:04:05\" }} \n========= = end = ========= \n{{- end }}\n{{- end }}\n{{- end }}\n\n{{- if gt (len .Alerts.Resolved) 0 -}}\n{{- range $index, $alert := .Alerts -}}\n{{- if eq $index 0 }}\n========= 异常恢复 ========= \n告警类型：{{ .Labels.alertname }} \n告警状态：{{ .Status }} \n告警主题: {{ $alert.Annotations.summary }} \n告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}} \ \n故障时间: {{ ($alert.StartsAt.Add 28800e9).Format \"2006-01-02 15:04:05\" }} \n恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format \"2006-01-02 15:04:05\" }} \n{{- if gt (len $alert.Labels.instance) 0 }} \n实例信息: {{ $alert.Labels.instance }} \n{{- end }} \n========= = end = ========= \n{{- end }}\n{{- end }}\n{{- end }}\n{{- end }}\n" email_template.tmpl: "{{ define \"email.default.html\" }}\n{{- if gt (len .Alerts.Firing) 0 -}}\n{{ range .Alerts }}\n=========start==========<br>\n告警程序: prometheus_alert <br>\n告警级别: {{ .Labels.severity }} <br>\n告警类型: {{ .Labels.alertname }} <br>\n告警主机: {{ .Labels.instance }} <br>\n告警主题: {{ .Annotations.summary }} <br>\n告警详情: {{ .Annotations.description }} <br>\n触发时间: {{ .StartsAt.Format \"2006-01-02 15:04:05\" }} <br>\n=========end==========<br>\n{{ end }}{{ end -}}\n \n{{- if gt (len .Alerts.Resolved) 0 -}}\n{{ range .Alerts }}\n=========start==========<br>\n告警程序: prometheus_alert <br>\n告警级别: {{ .Labels.severity }} <br>\n告警类型: {{ .Labels.alertname }} <br>\n告警主机: {{ .Labels.instance }} <br>\n告警主题: {{ .Annotations.summary }} <br>\n告警详情: {{ .Annotations.description }} <br>\n触发时间: {{ .StartsAt.Format \"2006-01-02 15:04:05\" }} <br>\n恢复时间: {{ .EndsAt.Format \"2006-01-02 15:04:05\" }} <br>\n=========end==========<br>\n{{ end }}{{ end -}}\n{{- end }}\n" wechat_template.tmpl: | {{ define "wechat.default.message" }} {{- if gt (len .Alerts.Firing) 0 -}} {{- range $index, $alert := .Alerts -}} {{- if eq $index 0 }} ========= 监控报警 ========= 告警状态：{{ .Status }} 告警级别：{{ .Labels.severity }} 告警类型：{{ $alert.Labels.alertname }} 故障主机: {{ $alert.Labels.instance }} 告警主题: {{ $alert.Annotations.summary }} 告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}; 触发阈值：{{ .Annotations.value }} 故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} ========= = end = ========= {{- end }} {{- end }} {{- end }} {{- if gt (len .Alerts.Resolved) 0 -}} {{- range $index, $alert := .Alerts -}} {{- if eq $index 0 }} ========= 异常恢复 ========= 告警类型：{{ .Labels.alertname }} 告警状态：{{ .Status }} 告警主题: {{ $alert.Annotations.summary }} 告警详情: {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}; 故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} 恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }} {{- if gt (len $alert.Labels.instance) 0 }} 实例信息: {{ $alert.Labels.instance }} {{- end }} ========= = end = ========= {{- end }} {{- end }} {{- end }} {{- end }} ---
安装grafana
# kubectl apply -f grafana/
# ll -h grafana/ total 28K drwxr-xr-x 2 root root 4.0K Apr 28 08:00 ./ drwxr-xr-x 8 root root 4.0K Apr 28 08:00 ../ -rw-r--r-- 1 root root 532 Apr 28 08:00 01-grafana-cfg.yaml -rw-r--r-- 1 root root 270 Apr 28 08:00 02-grafana-service.yaml -rw-r--r-- 1 root root 210 Apr 28 08:00 03-grafana-pvc.yaml -rw-r--r-- 1 root root 1.6K Apr 28 08:00 04-grafana-deployment.yaml -rw-r--r-- 1 root root 478 Apr 28 08:00 05-grafana-ingress.yaml #
# Maintainer: MageEdu <mage@magedu.com> apiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources namespace: prom data: prometheus.yaml: |- { "apiVersion": 1, "datasources": [ { "access":"proxy", "editable": true, "name": "prometheus", "orgId": 1, "type": "prometheus", "url": "http://prometheus.prom.svc.cluster.local.:9090", "version": 1 } ] } ---
--- apiVersion: v1 kind: Service metadata: name: grafana namespace: prom annotations: prometheus.io/scrape: 'true' prometheus.io/port: '3000' spec: selector: app: grafana type: NodePort ports: - port: 3000 targetPort: 3000 ---
# Maintainer: MageEdu <mage@magedu.com> # --- apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: prom spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: name: grafana labels: app: grafana spec: initContainers: - name: fix-permissions image: alpine command: ["sh", "-c", "chown -R 472:472 /var/lib/grafana/"] securityContext: privileged: true volumeMounts: - name: grafana-storage mountPath: /var/lib/grafana containers: - name: grafana image: grafana/grafana:10.2.5 #image: registry.magedu.com/grafana/grafana:9.5.13 imagePullPolicy: IfNotPresent ports: - name: grafana containerPort: 3000 resources: limits: cpu: 200m memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /etc/grafana/provisioning/datasources name: grafana-datasources readOnly: false #- mountPath: /etc/grafana/provisioning/dashboards # name: grafana-dashboards - mountPath: /var/lib/grafana name: grafana-storage volumes: - name: grafana-datasources configMap: defaultMode: 420 name: grafana-datasources #- configMap: # defaultMode: 420 # name: grafana-dashboards - name: grafana-storage persistentVolumeClaim: claimName: grafana-pvc
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: grafana namespace: prom labels: app: grafana annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: 'cilium' rules: - host: grafana.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc namespace: prom spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: openebs-hostpath
安装后详细
root@k8s-cilium-master-01:~# kubectl get sc -A NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 145m openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 145m root@k8s-cilium-master-01:~# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE prom grafana-pvc Bound pvc-9529a417-586b-4178-8c1b-36d53f8805d3 5Gi RWO openebs-hostpath 19m root@k8s-cilium-master-01:~# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h kube-system cilium-agent ClusterIP None <none> 9964/TCP 25h kube-system cilium-ingress LoadBalancer 10.103.136.93 172.16.88.90 80:30857/TCP,443:31638/TCP 25h kube-system cilium-ingress-hubble-ui LoadBalancer 10.99.117.51 172.16.88.91 80:31222/TCP,443:30301/TCP 25h kube-system hubble-metrics ClusterIP None <none> 9665/TCP 25h kube-system hubble-peer ClusterIP 10.100.154.82 <none> 443/TCP 25h kube-system hubble-relay ClusterIP 10.110.242.159 <none> 80/TCP 25h kube-system hubble-ui ClusterIP 10.111.100.166 <none> 80/TCP 25h kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 25h metallb-system metallb-webhook-service ClusterIP 10.109.66.191 <none> 443/TCP 25h openebs openebs-ndm-cluster-exporter-service ClusterIP None <none> 9100/TCP 145m openebs openebs-ndm-node-exporter-service ClusterIP None <none> 9101/TCP 145m prom grafana NodePort 10.96.83.159 <none> 3000:31211/TCP 19m prom kube-state-metrics ClusterIP 10.108.85.55 <none> 8080/TCP 100m prom prometheus NodePort 10.97.173.12 <none> 9090:30090/TCP 132m prom prometheus-node-exporter ClusterIP None <none> 9100/TCP 103m root@k8s-cilium-master-01:~# kubectl get pod -n prom NAME READY STATUS RESTARTS AGE grafana-69465f69d6-tttqn 1/1 Running 0 19m kube-state-metrics-7bfbbcdbb7-xk9s5 1/1 Running 0 100m prometheus-node-exporter-52bdq 1/1 Running 0 103m prometheus-node-exporter-8cqg7 1/1 Running 0 103m prometheus-node-exporter-gnvd5 1/1 Running 0 103m prometheus-node-exporter-sjg8d 1/1 Running 0 103m prometheus-node-exporter-xp776 1/1 Running 0 103m prometheus-server-5974664c5d-bz7jk 1/1 Running 0 91m root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# kubectl get ingress -n prom NAME CLASS HOSTS ADDRESS PORTS AGE grafana cilium grafana.k8s.com 172.16.88.90 80 20m prometheus cilium prom.k8s.com,prometheus.k8s.com 172.16.88.90 80 133m root@k8s-cilium-master-01:~#
二、业务环境部署
2.1、安装部署nacos
创建名称空间
kubectl create namespace nacos
kubectl apply -f 01-secrets-mysql.yaml -f 02-mysql-persistent.yaml -n nacos
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# pwd /root/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# ll -h total 52K drwxr-xr-x 3 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 265 May 1 07:56 01-secrets-mysql.yaml -rw-r--r-- 1 root root 8.2K May 1 07:56 02-mysql-persistent.yaml -rw-r--r-- 1 root root 3.2K May 1 07:56 03-nacos-cfg.yaml -rw-r--r-- 1 root root 4.2K May 1 07:56 04-nacos-persistent.yaml -rw-r--r-- 1 root root 342 May 1 07:56 05-nacos-service.yaml -rw-r--r-- 1 root root 430 May 1 07:56 06-nacos-ingress.yaml drwxr-xr-x 2 root root 4.0K May 1 07:56 examples/ -rw-r--r-- 1 root root 2.7K May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
--- apiVersion: v1 kind: Secret metadata: name: mysql-secret data: database.name: bmFjb3NkYg== # DB name: nacosdb root.password: "" # root password: null user.name: bmFjb3M= # username: nacos user.password: bWFnZWR1LmNvbQo= # password: magedu.com
--- apiVersion: v1 kind: Secret metadata: name: mysql-secret data: database.name: bmFjb3NkYg== # DB name: nacosdb root.password: "" # root password: null user.name: bmFjb3M= # username: nacos user.password: bWFnZWR1LmNvbQo= # password: magedu.com root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# cat 02-mysql-persistent.yaml # Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # MySQL Replication Cluster for Nacos --- apiVersion: v1 kind: ConfigMap metadata: name: mysql data: primary.cnf: | # Apply this config only on the primary. [mysql] default-character-set=utf8mb4 [mysqld] log-bin character-set-server=utf8mb4 #innodb-file-per-table=on [client] default-character-set=utf8mb4 replica.cnf: | # Apply this config only on replicas. [mysql] default-character-set=utf8mb4 [mysqld] super-read-only character-set-server=utf8mb4 [client] default-character-set=utf8mb4 --- # Headless service for stable DNS entries of StatefulSet members. apiVersion: v1 kind: Service metadata: name: mysql spec: ports: - name: mysql port: 3306 clusterIP: None selector: app: mysql --- # Client service for connecting to any MySQL instance for reads. # For writes, you must instead connect to the primary: mysql-0.mysql. apiVersion: v1 kind: Service metadata: name: mysql-read labels: app: mysql spec: ports: - name: mysql port: 3306 selector: app: mysql --- # StatefulSet for mysql replication cluster apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql spec: selector: matchLabels: app: mysql serviceName: mysql replicas: 2 template: metadata: labels: app: mysql annotations: prometheus.io/scrape: "true" prometheus.io/port: "9104" prometheus.io/path: "/metrics" spec: initContainers: - name: init-mysql #image: registry.magedu.com/nacos/nacos-mysql:5.7 image: nacos/nacos-mysql:5.7 command: - bash - "-c" - | set -ex # Generate mysql server-id from pod ordinal index. [[ $(cat /proc/sys/kernel/hostname) =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo [mysqld] > /mnt/conf.d/server-id.cnf # Add an offset to avoid reserved server-id=0 value. echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf # Copy appropriate conf.d files from config-map to emptyDir. if [[ $ordinal -eq 0 ]]; then cp /mnt/config-map/primary.cnf /mnt/conf.d/ else cp /mnt/config-map/replica.cnf /mnt/conf.d/ fi volumeMounts: - name: conf mountPath: /mnt/conf.d - name: config-map mountPath: /mnt/config-map - name: clone-mysql image: ikubernetes/xtrabackup:1.0 command: - bash - "-c" - | set -ex # Skip the clone if data already exists. [[ -d /var/lib/mysql/mysql ]] && exit 0 # Skip the clone on primary (ordinal index 0). [[ $(cat /proc/sys/kernel/hostname) =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} [[ $ordinal -eq 0 ]] && exit 0 # Clone data from previous peer. ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql # Prepare the backup. xtrabackup --prepare --target-dir=/var/lib/mysql volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d containers: - name: mysql #image: registry.magedu.com/nacos/nacos-mysql:5.7 image: nacos/nacos-mysql:5.7 env: - name: LANG value: "C.UTF-8" - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: root.password - name: MYSQL_ALLOW_EMPTY_PASSWORD value: "1" - name: MYSQL_USER valueFrom: secretKeyRef: name: mysql-secret key: user.name - name: MYSQL_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: user.password - name: MYSQL_DATABASE valueFrom: secretKeyRef: name: mysql-secret key: database.name ports: - name: mysql containerPort: 3306 volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d livenessProbe: exec: command: ["mysqladmin", "ping"] initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 readinessProbe: exec: # Check we can execute queries over TCP (skip-networking is off). command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"] initialDelaySeconds: 5 periodSeconds: 2 timeoutSeconds: 1 - name: xtrabackup image: ikubernetes/xtrabackup:1.0 ports: - name: xtrabackup containerPort: 3307 command: - bash - "-c" - | set -ex cd /var/lib/mysql # Determine binlog position of cloned data, if any. if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then # XtraBackup already generated a partial "CHANGE MASTER TO" query # because we're cloning from an existing replica. (Need to remove the tailing semicolon!) cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in # Ignore xtrabackup_binlog_info in this case (it's useless). rm -f xtrabackup_slave_info xtrabackup_binlog_info elif [[ -f xtrabackup_binlog_info ]]; then # We're cloning directly from primary. Parse binlog position. [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1 rm -f xtrabackup_binlog_info xtrabackup_slave_info echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\ MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in fi # Check if we need to complete a clone by starting replication. if [[ -f change_master_to.sql.in ]]; then echo "Waiting for mysqld to be ready (accepting connections)" until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done echo "Initializing replication from clone position" mysql -h 127.0.0.1 \ -e "$(<change_master_to.sql.in), \ MASTER_HOST='mysql-0.mysql', \ MASTER_USER='root', \ MASTER_PASSWORD='', \ MASTER_CONNECT_RETRY=10; \ START SLAVE;" || exit 1 # In case of container restart, attempt this at-most-once. mv change_master_to.sql.in change_master_to.sql.orig fi # Start a server to send backups when requested by peers. exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \ "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root" volumeMounts: - name: data mountPath: /var/lib/mysql subPath: mysql - name: conf mountPath: /etc/mysql/conf.d - name: mysqld-exporter image: prom/mysqld-exporter:v0.14.0 #image: registry.magedu.com/prom/mysqld-exporter:v0.14.0 env: - name: DATA_SOURCE_NAME value: 'root:""@(localhost:3306)/' args: - --collect.info_schema.innodb_metrics - --collect.info_schema.innodb_tablespaces - --collect.perf_schema.eventsstatementssum - --collect.perf_schema.memory_events - --collect.global_status - --collect.engine_innodb_status - --collect.binlog_size ports: - name: mysqld-exporter containerPort: 9104 volumes: - name: conf emptyDir: {} - name: config-map configMap: name: mysql volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] # Modify StorageClass before deployment storageClassName: "openebs-hostpath" resources: requests: storage: 10Gi
安装详情
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl create namespace nacos namespace/nacos created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl apply -f 01-secrets-mysql.yaml -f 02-mysql-persistent.yaml -n nacos secret/mysql-secret created configmap/mysql created service/mysql created service/mysql-read created statefulset.apps/mysql created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get pod -n nacos NAME READY STATUS RESTARTS AGE mysql-0 3/3 Running 0 60m mysql-1 3/3 Running 0 34m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nacos data-mysql-0 Bound pvc-84cfc00e-a254-4e23-b718-cf9a38c65ac0 10Gi RWO openebs-hostpath 61m nacos data-mysql-1 Bound pvc-89fa2b48-51f6-406e-b2e7-cef565c2777d 10Gi RWO openebs-hostpath 34m prom grafana-pvc Bound pvc-9529a417-586b-4178-8c1b-36d53f8805d3 5Gi RWO openebs-hostpath 2d root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
访问入口
读请求：mysql-read.nacos.svc.cluster.local
写请求：mysql-0.mysql.nacos.svc.cluster.local
创建用户账号
在mysql上创建nacos专用的用户账号，本示例中，Naocs默认使用nacos用户名和"magedu.com"密码访问mysql服务上的nacosdb数据库。
kubectl exec -it mysql-0 -n nacos -- mysql -uroot -hlocalhost
在mysql的提示符下运行如下SQL语句后退出即可
mysql> GRANT ALL ON nacosdb.* TO nacos@'%' IDENTIFIED BY 'magedu.com';
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl exec -it mysql-0 -n nacos -- mysql -uroot -hlocalhost Defaulted container "mysql" out of: mysql, xtrabackup, mysqld-exporter, init-mysql (init), clone-mysql (init) Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1583 Server version: 5.7.39-log MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> GRANT ALL ON nacosdb.* TO nacos@'%' IDENTIFIED BY 'magedu.com'; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> exit Bye root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
部署Nacos
kubectl apply -f 03-nacos-cfg.yaml -f 04-nacos-persistent.yaml -f 05-nacos-service.yaml -n nacos
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # Nacos Cluster --- apiVersion: v1 kind: ConfigMap metadata: name: nacos-cfg data: application.properties: | # spring server.servlet.contextPath=${SERVER_SERVLET_CONTEXTPATH:/nacos} server.contextPath=/nacos server.port=${NACOS_APPLICATION_PORT:8848} server.tomcat.accesslog.max-days=30 server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D %{User-Agent}i %{Request-Source}i server.tomcat.accesslog.enabled=${TOMCAT_ACCESSLOG_ENABLED:false} server.error.include-message=ALWAYS # default current work dir server.tomcat.basedir=file:. #*************** Config Module Related Configurations ***************# ### Deprecated configuration property, it is recommended to use `spring.sql.init.platform` replaced. #spring.datasource.platform=${SPRING_DATASOURCE_PLATFORM:} spring.sql.init.platform=${SPRING_DATASOURCE_PLATFORM:} nacos.cmdb.dumpTaskInterval=3600 nacos.cmdb.eventTaskInterval=10 nacos.cmdb.labelTaskInterval=300 nacos.cmdb.loadDataAtStart=false db.num=${MYSQL_DATABASE_NUM:1} db.url.0=jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT:3306}/${MYSQL_SERVICE_DB_NAME}?${MYSQL_SERVICE_DB_PARAM:characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false} db.user.0=${MYSQL_SERVICE_USER} db.password.0=${MYSQL_SERVICE_PASSWORD} ### The auth system to use, currently only 'nacos' and 'ldap' is supported: nacos.core.auth.system.type=${NACOS_AUTH_SYSTEM_TYPE:nacos} ### worked when nacos.core.auth.system.type=nacos ### The token expiration in seconds: nacos.core.auth.plugin.nacos.token.expire.seconds=${NACOS_AUTH_TOKEN_EXPIRE_SECONDS:18000} ### The default token: nacos.core.auth.plugin.nacos.token.secret.key=${NACOS_AUTH_TOKEN:} ### Turn on/off caching of auth information. By turning on this switch, the update of auth information would have a 15 seconds delay. nacos.core.auth.caching.enabled=${NACOS_AUTH_CACHE_ENABLE:false} nacos.core.auth.enable.userAgentAuthWhite=${NACOS_AUTH_USER_AGENT_AUTH_WHITE_ENABLE:false} nacos.core.auth.server.identity.key=${NACOS_AUTH_IDENTITY_KEY:} nacos.core.auth.server.identity.value=${NACOS_AUTH_IDENTITY_VALUE:} ## spring security config ### turn off security nacos.security.ignore.urls=${NACOS_SECURITY_IGNORE_URLS:/,/error,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/**,/v1/console/health/**,/actuator/**,/v1/console/server/**} # metrics for elastic search # meterics path is "/nacos/actuator/prometheus" management.endpoints.web.exposure.include=* management.metrics.export.elastic.enabled=false management.metrics.export.influx.enabled=false nacos.naming.distro.taskDispatchThreadCount=10 nacos.naming.distro.taskDispatchPeriod=200 nacos.naming.distro.batchSyncKeyCount=1000 nacos.naming.distro.initDataRatio=0.9 nacos.naming.distro.syncRetryDelay=5000 nacos.naming.data.warmup=true #mysql.host: "mysql-0.mysql.mall.svc.cluster.local" mysql.host: "mysql-0.mysql" # mysql master host mysql.port: "3306"
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # Nacos Cluster --- apiVersion: v1 kind: Service metadata: name: nacos-headless labels: app: nacos spec: publishNotReadyAddresses: true ports: - port: 8848 name: server targetPort: 8848 - port: 9848 name: client-rpc targetPort: 9848 - port: 9849 name: raft-rpc targetPort: 9849 ## 兼容1.4.x版本的选举端口 - port: 7848 name: old-raft-rpc targetPort: 7848 clusterIP: None selector: app: nacos --- apiVersion: apps/v1 kind: StatefulSet metadata: name: nacos spec: podManagementPolicy: Parallel serviceName: nacos-headless replicas: 3 selector: matchLabels: app: nacos template: metadata: labels: app: nacos annotations: pod.alpha.kubernetes.io/initialized: "true" prometheus.io/scrape: "true" prometheus.io/port: "8848" prometheus.io/path: "/nacos/actuator/prometheus" spec: initContainers: - name: peer-finder-plugin-install #image: registry.magedu.com/nacos/nacos-peer-finder-plugin:1.1 image: nacos/nacos-peer-finder-plugin:1.1 imagePullPolicy: Always volumeMounts: - mountPath: /home/nacos/plugins/peer-finder name: data subPath: peer-finder containers: - name: nacos imagePullPolicy: Always #image: registry.magedu.com/nacos/nacos-server:v2.2.3-slim image: nacos/nacos-server:v2.3.1-slim resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 8848 name: client-port - containerPort: 9848 name: client-rpc - containerPort: 9849 name: raft-rpc - containerPort: 7848 name: old-raft-rpc env: - name: NACOS_REPLICAS value: "3" - name: SERVICE_NAME value: "nacos-headless" - name: DOMAIN_NAME value: "cluster.local" - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: MYSQL_SERVICE_HOST valueFrom: configMapKeyRef: name: nacos-cfg key: mysql.host - name: MYSQL_SERVICE_DB_NAME valueFrom: secretKeyRef: name: mysql-secret key: database.name - name: MYSQL_SERVICE_PORT valueFrom: configMapKeyRef: name: nacos-cfg key: mysql.port - name: MYSQL_SERVICE_USER valueFrom: secretKeyRef: name: mysql-secret key: user.name - name: MYSQL_SERVICE_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: user.password - name: SPRING_DATASOURCE_PLATFORM value: "mysql" - name: NACOS_SERVER_PORT value: "8848" - name: NACOS_APPLICATION_PORT value: "8848" - name: PREFER_HOST_MODE value: "hostname" volumeMounts: - name: properties mountPath: /home/nacos/conf/application.properties subPath: application.properties - name: data mountPath: /home/nacos/plugins/peer-finder subPath: peer-finder - name: data mountPath: /home/nacos/data subPath: data - name: data mountPath: /home/nacos/logs subPath: logs volumes: - name: properties configMap: name: nacos-cfg volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] # Modify StorageClass before deployment. storageClassName: "openebs-hostpath" resources: requests: storage: 10Gi ---
apiVersion: v1 kind: Service metadata: name: nacos labels: app: nacos annotations: {} #prometheus.io/scrape: "true" #prometheus.io/port: "8848" #prometheus.io/path: "/nacos/actuator/prometheus" spec: #type: LoadBalancer ports: - port: 8848 name: server targetPort: 8848 selector: app: nacos
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl apply -f 03-nacos-cfg.yaml -f 04-nacos-persistent.yaml -f 05-nacos-service.yaml -n nacos configmap/nacos-cfg created service/nacos-headless created statefulset.apps/nacos created service/nacos created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get pod -n nacos NAME READY STATUS RESTARTS AGE mysql-0 3/3 Running 0 96m mysql-1 3/3 Running 0 69m nacos-0 1/1 Running 0 26m nacos-1 1/1 Running 0 26m nacos-2 1/1 Running 0 26m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nacos data-mysql-0 Bound pvc-84cfc00e-a254-4e23-b718-cf9a38c65ac0 10Gi RWO openebs-hostpath 96m nacos data-mysql-1 Bound pvc-89fa2b48-51f6-406e-b2e7-cef565c2777d 10Gi RWO openebs-hostpath 69m nacos data-nacos-0 Bound pvc-00ce3f9b-c937-4efb-88a6-52b56e007398 10Gi RWO openebs-hostpath 26m nacos data-nacos-1 Bound pvc-d66a4ad9-5222-4d0f-bf1a-64288ce1a0fb 10Gi RWO openebs-hostpath 26m nacos data-nacos-2 Bound pvc-3f9a0d55-4171-466b-a1e6-673a5c6b7b05 10Gi RWO openebs-hostpath 26m prom grafana-pvc Bound pvc-9529a417-586b-4178-8c1b-36d53f8805d3 5Gi RWO openebs-hostpath 2d root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
登录访问nacos
配置loadbalancer入口访问
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# cat 06-nacos-ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nacos annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: nacos.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: nacos port: number: 8848 root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl apply -f 06-nacos-ingress.yaml -n nacos ingress.networking.k8s.io/nacos created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get ingress -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE kube-system hubble-ui cilium hubble.k8s.com 172.16.88.91 80 3d1h nacos nacos cilium nacos.k8s.com 172.16.88.90 80 7s prom grafana cilium grafana.k8s.com 172.16.88.90 80 2d prom prometheus cilium prom.k8s.com,prometheus.k8s.com 172.16.88.90 80 2d2h root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
http://nacos.k8s.com/nacos
导入数据
curl --location -XPOST 'http://10.244.6.191:8848/nacos/v1/cs/configs?import=true&namespace=public' \
--form 'policy=OVERWRITE' --form 'file=@"examples/nacos_config_20231029.zip"'
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# ll -h total 52K drwxr-xr-x 3 root root 4.0K May 1 09:44 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 265 May 1 07:56 01-secrets-mysql.yaml -rw-r--r-- 1 root root 8.2K May 1 07:56 02-mysql-persistent.yaml -rw-r--r-- 1 root root 3.2K May 1 07:56 03-nacos-cfg.yaml -rw-r--r-- 1 root root 4.2K May 1 07:56 04-nacos-persistent.yaml -rw-r--r-- 1 root root 342 May 1 07:56 05-nacos-service.yaml -rw-r--r-- 1 root root 427 May 1 09:43 06-nacos-ingress.yaml drwxr-xr-x 2 root root 4.0K May 1 07:56 examples/ -rw-r--r-- 1 root root 2.7K May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# ll -h examples/ total 32K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 3 root root 4.0K May 1 09:44 ../ -rw-r--r-- 1 root root 478 May 1 07:56 cloud-nacos-registry.yaml -rw-r--r-- 1 root root 12K May 1 07:56 nacos_config_20230806.zip -rw-r--r-- 1 root root 3.2K May 1 07:56 nacos_config_20230808.zip -rw-r--r-- 1 root root 3.0K May 1 07:56 nacos_config_20231029.zip root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# kubectl get pod -n nacos -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-0 3/3 Running 0 133m 10.244.4.235 k8s-cilium-node-04 <none> <none> mysql-1 3/3 Running 0 106m 10.244.2.35 k8s-cilium-node-02 <none> <none> nacos-0 1/1 Running 0 63m 10.244.6.191 k8s-cilium-node-05 <none> <none> nacos-1 1/1 Running 0 63m 10.244.2.205 k8s-cilium-node-02 <none> <none> nacos-2 1/1 Running 0 63m 10.244.4.185 k8s-cilium-node-04 <none> <none> root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos# curl --location -XPOST 'http://10.244.6.191:8848/nacos/v1/cs/configs?import=true&namespace=public' \
--form 'policy=OVERWRITE' --form 'file=@"examples/nacos_config_20231029.zip"' {"code":200,"message":"导入成功","data":{"succCount":5,"skipCount":0}}
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/01-Nacos#
2.2、安装elk
/root/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# ll -h total 36K drwxr-xr-x 4 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 3.9K May 1 07:56 01-elasticsearch-cluster-persistent.yaml -rw-r--r-- 1 root root 5.5K May 1 07:56 02-fluentbit.yaml -rw-r--r-- 1 root root 1.4K May 1 07:56 03-kibana.yaml drwxr-xr-x 2 root root 4.0K May 1 07:56 filebeat/ drwxr-xr-x 2 root root 4.0K May 1 07:56 grafana-dashboards/ -rw-r--r-- 1 root root 1.3K May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack#
创建名称空间
kubectl create namespace elastic
部署elasticsearch
kubectl apply -f 01-elasticsearch-cluster-persistent.yaml -n elastic
# Maintainer: MageEdu <mage@magedu.com> # Site: https://www.magedu.com # ElasticSearch Deployment. --- kind: Service apiVersion: v1 metadata: name: elasticsearch labels: app: elasticsearch spec: selector: app: elasticsearch clusterIP: None ports: - port: 9200 name: rest - port: 9300 name: inter-node --- apiVersion: apps/v1 kind: StatefulSet metadata: name: es-cluster spec: serviceName: elasticsearch replicas: 3 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch annotations: prometheus.io/scrape: "true" prometheus.io/port: "9114" prometheus.io/path: "/metrics" # Grafana dashboard: https://github.com/prometheus-community/elasticsearch_exporter/blob/master/examples/grafana/dashboard.json spec: containers: - name: elasticsearch image: ikubernetes/elasticsearch:7.17.7-ik #image: registry.magedu.com/elastic/elasticsearch:7.17.7-ik imagePullPolicy: IfNotPresent resources: limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: rest protocol: TCP - containerPort: 9300 name: inter-node protocol: TCP volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data env: - name: cluster.name value: k8s-logs - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: discovery.seed_hosts value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch" - name: cluster.initial_master_nodes value: "es-cluster-0" - name: ES_JAVA_OPTS value: "-Xms1g -Xmx1g" - name: network.host value: "_site_" - name: xpack.security.enabled value: 'false' - name: xpack.security.transport.ssl.enabled value: 'false' - name: elasticsearch-exporter #image: registry.magedu.com/prometheuscommunity/elasticsearch-exporter:v1.6.0 image: prometheuscommunity/elasticsearch-exporter:v1.7.0 args: - '--es.uri=http://localhost:9200' - '--es.shards' - '--es.indices' - '--es.indices_settings' ports: - name: es-exporter containerPort: 9114 livenessProbe: httpGet: path: /healthz port: 9114 initialDelaySeconds: 30 timeoutSeconds: 10 readinessProbe: httpGet: path: /healthz port: 9114 initialDelaySeconds: 10 timeoutSeconds: 10 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 25m memory: 64Mi initContainers: - name: fix-permissions image: ikubernetes/admin-box:v1.2 command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] securityContext: privileged: true volumeMounts: - name: data mountPath: /usr/share/elasticsearch/data - name: increase-vm-max-map image: ikubernetes/admin-box:v1.2 command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true - name: increase-fd-ulimit image: ikubernetes/admin-box:v1.2 command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true volumeClaimTemplates: - metadata: name: data labels: app: elasticsearch spec: accessModes: [ "ReadWriteOnce" ] storageClassName: openebs-hostpath resources: requests: storage: 10Gi
验证es集群正常情况
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# kubectl get pod -n elastic -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES es-cluster-0 2/2 Running 0 15h 10.244.1.182 k8s-cilium-node-01 <none> <none> es-cluster-1 2/2 Running 0 15h 10.244.3.126 k8s-cilium-node-03 <none> <none> es-cluster-2 2/2 Running 0 14h 10.244.6.166 k8s-cilium-node-05 <none> <none> root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# curl http://10.244.1.182:9200 { "name" : "es-cluster-0", "cluster_name" : "k8s-logs", "cluster_uuid" : "rjrAG022Tra4LpLnYlCPFA", "version" : { "number" : "7.17.7", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "78dcaaa8cee33438b91eca7f5c7f56a70fec9e80", "build_date" : "2022-10-17T15:29:54.167373105Z", "build_snapshot" : false, "lucene_version" : "8.11.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# curl http://10.244.1.182:9200/_cat/ =^.^= /_cat/allocation /_cat/shards /_cat/shards/{index} /_cat/master /_cat/nodes /_cat/tasks /_cat/indices /_cat/indices/{index} /_cat/segments /_cat/segments/{index} /_cat/count /_cat/count/{index} /_cat/recovery /_cat/recovery/{index} /_cat/health /_cat/pending_tasks /_cat/aliases /_cat/aliases/{alias} /_cat/thread_pool /_cat/thread_pool/{thread_pools} /_cat/plugins /_cat/fielddata /_cat/fielddata/{fields} /_cat/nodeattrs /_cat/repositories /_cat/snapshots/{repository} /_cat/templates /_cat/ml/anomaly_detectors /_cat/ml/anomaly_detectors/{job_id} /_cat/ml/trained_models /_cat/ml/trained_models/{model_id} /_cat/ml/datafeeds /_cat/ml/datafeeds/{datafeed_id} /_cat/ml/data_frame/analytics /_cat/ml/data_frame/analytics/{id} /_cat/transforms /_cat/transforms/{transform_id} root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack#
待ES的相关Pod就绪后，即可部署fluentd
kubectl apply -f 02-fluentbit.yaml -n elastic
--- apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit labels: app: fluent-bit --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluent-bit labels: app: fluent-bit rules: - apiGroups: [""] resources: - pods - namespaces verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluent-bit namespace: kube-logging --- apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-conf labels: k8s-app: fluent-bit data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-elasticsearch.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off output-elasticsearch.conf: | [OUTPUT] Name es Match * Host elasticsearch Port 9200 Logstash_Format On Replace_Dots On Retry_Limit False parsers.conf: | [PARSER] Name apache Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit spec: selector: matchLabels: k8s-app: fluent-bit version: v2 template: metadata: labels: k8s-app: fluent-bit version: v2 annotations: prometheus.io/scrape: "true" prometheus.io/port: "2020" prometheus.io/path: /api/v1/metrics/prometheus spec: containers: - name: fluent-bit image: fluent/fluent-bit:2.2.2 #image: registry.magedu.com/fluent/fluent-bit:2.1.8 imagePullPolicy: IfNotPresent ports: - containerPort: 2020 env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-conf serviceAccountName: fluent-bit tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - operator: "Exists" effect: "NoExecute" - operator: "Exists" effect: "NoSchedule"
(可选)filebeat进行收集
--- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: container paths: - /var/log/containers/*.log processors: - add_kubernetes_metadata: host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this: #filebeat.autodiscover: # providers: # - type: kubernetes # node: ${NODE_NAME} # hints.enabled: true # hints.default_config: # type: container # paths: # - /var/log/containers/*${data.kubernetes.container.id}.log processors: - add_cloud_metadata: - add_host_metadata: output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] --- apiVersion: apps/v1 kind: DaemonSet metadata: name: filebeat labels: k8s-app: filebeat spec: selector: matchLabels: k8s-app: filebeat template: metadata: labels: k8s-app: filebeat spec: serviceAccountName: filebeat tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - operator: "Exists" effect: "NoExecute" - operator: "Exists" effect: "NoSchedule" terminationGracePeriodSeconds: 30 hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: filebeat image: elastic/filebeat:7.17.12 args: [ "-c", "/etc/filebeat.yml", "-e", ] env: - name: ELASTICSEARCH_HOST value: elasticsearch - name: ELASTICSEARCH_PORT value: "9200" # - name: ELASTICSEARCH_USERNAME # value: elastic # - name: ELASTICSEARCH_PASSWORD # value: changeme - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: runAsUser: 0 # If using Red Hat OpenShift uncomment this: #privileged: true resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: data mountPath: /usr/share/filebeat/data - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: varlog mountPath: /var/log readOnly: true volumes: - name: config configMap: defaultMode: 0640 name: filebeat-config - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: varlog hostPath: path: /var/log # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart - name: data hostPath: # When filebeat runs as non-root user, this directory needs to be writable by group (g+w). path: /var/lib/filebeat-data type: DirectoryOrCreate --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat roleRef: kind: Role name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: filebeat-kubeadm-config subjects: - kind: ServiceAccount name: filebeat roleRef: kind: Role name: filebeat-kubeadm-config apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: filebeat labels: k8s-app: filebeat rules: - apiGroups: [""] # "" indicates the core API group resources: - namespaces - pods - nodes verbs: - get - watch - list - apiGroups: ["apps"] resources: - replicasets verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: filebeat # should be the namespace where filebeat is running labels: k8s-app: filebeat rules: - apiGroups: - coordination.k8s.io resources: - leases verbs: ["get", "create", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: filebeat-kubeadm-config labels: k8s-app: filebeat rules: - apiGroups: [""] resources: - configmaps resourceNames: - kubeadm-config verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: filebeat labels: k8s-app: filebeat ---
部署kibana
kubectl apply -f 03-kibana.yaml -n elastic
# Maintainer: MageEdu <mage@magedu.com> # --- apiVersion: v1 kind: Service metadata: name: kibana labels: app: kibana spec: ports: - port: 5601 targetPort: 5601 selector: app: kibana type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment metadata: name: kibana labels: app: kibana spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: containers: - name: kibana #image: registry.magedu.com/elastic/kibana:7.17.7 image: elastic/kibana:7.17.7 imagePullPolicy: IfNotPresent resources: limits: cpu: 2000m memory: "2048Mi" requests: cpu: 500m memory: "1024Mi" env: - name: ELASTICSEARCH_URL value: http://elasticsearch:9200 ports: - containerPort: 5601 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: kibana annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: kibana.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: kibana port: number: 5601
访问地址：http://kibana.k8s.com/
安装详情
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# kubectl create namespace elastic namespace/elastic created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# kubectl apply -f 01-elasticsearch-cluster-persistent.yaml -n elastic service/elasticsearch created statefulset.apps/es-cluster created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# kubectl get pod -n elastic NAME READY STATUS RESTARTS AGE es-cluster-0 2/2 Running 0 69m es-cluster-1 2/2 Running 0 46m es-cluster-2 2/2 Running 0 19m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/02-ElasticStack# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# kubectl apply -f 02-fluentbit.yaml -n elastic serviceaccount/fluent-bit created clusterrole.rbac.authorization.k8s.io/fluent-bit created clusterrolebinding.rbac.authorization.k8s.io/fluent-bit created configmap/fluent-bit-conf created daemonset.apps/fluent-bit created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# kubectl get pod -n elastic NAME READY STATUS RESTARTS AGE es-cluster-0 2/2 Running 0 148m es-cluster-1 2/2 Running 0 125m es-cluster-2 2/2 Running 0 99m fluent-bit-6pc95 1/1 Running 0 77m fluent-bit-b8x85 1/1 Running 0 77m fluent-bit-j2h98 1/1 Running 0 77m fluent-bit-lbrb7 1/1 Running 0 77m fluent-bit-w8qd7 1/1 Running 0 77m fluent-bit-wg9pq 1/1 Running 0 77m fluent-bit-zfw78 1/1 Running 0 77m fluent-bit-zmmjh 1/1 Running 0 77m kibana-5ddc4d94fc-gxnhk 1/1 Running 0 67m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# kubectl get ingress -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE elastic kibana cilium kibana.k8s.com 172.16.88.90 80 68m kube-system hubble-ui cilium hubble.k8s.com 172.16.88.91 80 3d5h nacos nacos cilium nacos.k8s.com 172.16.88.90 80 4h13m prom grafana cilium grafana.k8s.com 172.16.88.90 80 2d5h prom prometheus cilium prom.k8s.com,prometheus.k8s.com 172.16.88.90 80 2d7h root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE elastic data-es-cluster-0 Bound pvc-433e72cf-4f56-4919-a15e-253534da3fbd 10Gi RWO openebs-hostpath 153m elastic data-es-cluster-1 Bound pvc-f69be8d5-016f-41b1-8a78-2a06167f57e6 10Gi RWO openebs-hostpath 131m elastic data-es-cluster-2 Bound pvc-c3739d0e-8e23-4112-ab02-f1ef58411e59 10Gi RWO openebs-hostpath 104m nacos data-mysql-0 Bound pvc-84cfc00e-a254-4e23-b718-cf9a38c65ac0 10Gi RWO openebs-hostpath 6h nacos data-mysql-1 Bound pvc-89fa2b48-51f6-406e-b2e7-cef565c2777d 10Gi RWO openebs-hostpath 5h33m nacos data-nacos-0 Bound pvc-00ce3f9b-c937-4efb-88a6-52b56e007398 10Gi RWO openebs-hostpath 4h50m nacos data-nacos-1 Bound pvc-d66a4ad9-5222-4d0f-bf1a-64288ce1a0fb 10Gi RWO openebs-hostpath 4h50m nacos data-nacos-2 Bound pvc-3f9a0d55-4171-466b-a1e6-673a5c6b7b05 10Gi RWO openebs-hostpath 4h50m prom grafana-pvc Bound pvc-9529a417-586b-4178-8c1b-36d53f8805d3 5Gi RWO openebs-hostpath 2d5h root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/02-ElasticStack#
2.3、安装redis
部署redis
首先，运行如下命令，创建名称空间。
kubectl create namespace redis
接着，运行如下命令，部署redis replication cluster。
kubectl apply -f . -n redis
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/03-Redis# ll -h total 72K drwxr-xr-x 3 root root 4.0K May 1 07:56 ./ drwxr-xr-x 9 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 38K May 1 07:56 00-configmap-redis.yaml -rw-r--r-- 1 root root 2.8K May 1 07:56 01-configmap-sentinel.yaml -rw-r--r-- 1 root root 223 May 1 07:56 02-secret-redis.yaml -rw-r--r-- 1 root root 723 May 1 07:56 03-services-redis.yaml -rw-r--r-- 1 root root 1.9K May 1 07:56 04-statefulset-redis.yaml -rw-r--r-- 1 root root 761 May 1 07:56 README.md drwxr-xr-x 2 root root 4.0K May 1 07:56 sentinel/ root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/03-Redis#
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # --- apiVersion: v1 kind: ConfigMap metadata: name: redis-conf data: # Variable needed for sentinel init container REDIS_NODES: "redis-0.redis,redis-1.redis,redis-2.redis" # Config used in all redises redis.conf: | # Redis configuration file example. # # Note that in order to read the configuration file, Redis must be # started with the file path as first argument: # # ./redis-server /path/to/redis.conf #slaveof redis-master-0.redis-master.redis.svc.cluster.local 6379 # masterauth a-very-complex-password-here # requirepass a-very-complex-password-here # Note on units: when memory size is needed, it is possible to specify # it in the usual form of 1k 5GB 4M and so forth: # # 1k => 1000 bytes # 1kb => 1024 bytes # 1m => 1000000 bytes # 1mb => 1024*1024 bytes # 1g => 1000000000 bytes # 1gb => 1024*1024*1024 bytes # # units are case insensitive so 1GB 1Gb 1gB are all the same. ################################## INCLUDES ################################### # Include one or more other config files here. This is useful if you # have a standard template that goes to all Redis servers but also need # to customize a few per-server settings. Include files can include # other files, so use this wisely. # # Notice option "include" won't be rewritten by command "CONFIG REWRITE" # from admin or Redis Sentinel. Since Redis always uses the last processed # line as value of a configuration directive, you'd better put includes # at the beginning of this file to avoid overwriting config change at runtime. # # If instead you are interested in using includes to override configuration # options, it is better to use include as the last line. # # include /path/to/local.conf # include /path/to/other.conf ################################## MODULES ##################################### # Load modules at startup. If the server is not able to load modules # it will abort. It is possible to use multiple loadmodule directives. # # loadmodule /path/to/my_module.so # loadmodule /path/to/other_module.so ################################## NETWORK ##################################### # By default, if no "bind" configuration directive is specified, Redis listens # for connections from all the network interfaces available on the server. # It is possible to listen to just one or multiple selected interfaces using # the "bind" configuration directive, followed by one or more IP addresses. # # Examples: # # bind 192.168.1.100 10.0.0.1 # bind 127.0.0.1 ::1 # # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the # internet, binding to all the interfaces is dangerous and will expose the # instance to everybody on the internet. So by default we uncomment the # following bind directive, that will force Redis to listen only into # the IPv4 loopback interface address (this means Redis will be able to # accept connections only from clients running into the same computer it # is running). # # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES # JUST COMMENT THE FOLLOWING LINE. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ bind 0.0.0.0 # Protected mode is a layer of security protection, in order to avoid that # Redis instances left open on the internet are accessed and exploited. # # When protected mode is on and if: # # 1) The server is not binding explicitly to a set of addresses using the # "bind" directive. # 2) No password is configured. # # The server only accepts connections from clients connecting from the # IPv4 and IPv6 loopback addresses 127.0.0.1 and ::1, and from Unix domain # sockets. # # By default protected mode is enabled. You should disable it only if # you are sure you want clients from other hosts to connect to Redis # even if no authentication is configured, nor a specific set of interfaces # are explicitly listed using the "bind" directive. protected-mode no # Accept connections on the specified port, default is 6379 (IANA #815344). # If port 0 is specified Redis will not listen on a TCP socket. port 6379 # TCP listen() backlog. # # In high requests-per-second environments you need an high backlog in order # to avoid slow clients connections issues. Note that the Linux kernel # will silently truncate it to the value of /proc/sys/net/core/somaxconn so # make sure to raise both the value of somaxconn and tcp_max_syn_backlog # in order to get the desired effect. tcp-backlog 511 # Unix socket. # # Specify the path for the Unix socket that will be used to listen for # incoming connections. There is no default, so Redis will not listen # on a unix socket when not specified. # # unixsocket /tmp/redis.sock # unixsocketperm 700 # Close the connection after a client is idle for N seconds (0 to disable) timeout 10 # TCP keepalive. # # If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence # of communication. This is useful for two reasons: # # 1) Detect dead peers. # 2) Take the connection alive from the point of view of network # equipment in the middle. # # On Linux, the specified value (in seconds) is the period used to send ACKs. # Note that to close the connection the double of the time is needed. # On other kernels the period depends on the kernel configuration. # # A reasonable value for this option is 300 seconds, which is the new # Redis default starting with Redis 3.2.1. tcp-keepalive 30 ################################# TLS/SSL ##################################### # By default, TLS/SSL is disabled. To enable it, the "tls-port" configuration # directive can be used to define TLS-listening ports. To enable TLS on the # default port, use: # # port 0 # tls-port 6379 # Configure a X.509 certificate and private key to use for authenticating the # server to connected clients, masters or cluster peers. These files should be # PEM formatted. # # tls-cert-file redis.crt # tls-key-file redis.key # Configure a DH parameters file to enable Diffie-Hellman (DH) key exchange: # # tls-dh-params-file redis.dh # Configure a CA certificate(s) bundle or directory to authenticate TLS/SSL # clients and peers. Redis requires an explicit configuration of at least one # of these, and will not implicitly use the system wide configuration. # # tls-ca-cert-file ca.crt # tls-ca-cert-dir /etc/ssl/certs # By default, clients (including replica servers) on a TLS port are required # to authenticate using valid client side certificates. # # It is possible to disable authentication using this directive. # # tls-auth-clients no # By default, a Redis replica does not attempt to establish a TLS connection # with its master. # # Use the following directive to enable TLS on replication links. # # tls-replication yes # By default, the Redis Cluster bus uses a plain TCP connection. To enable # TLS for the bus protocol, use the following directive: # # tls-cluster yes # Explicitly specify TLS versions to support. Allowed values are case insensitive # and include "TLSv1", "TLSv1.1", "TLSv1.2", "TLSv1.3" (OpenSSL >= 1.1.1) or # any combination. To enable only TLSv1.2 and TLSv1.3, use: # # tls-protocols "TLSv1.2 TLSv1.3" # Configure allowed ciphers. See the ciphers(1ssl) manpage for more information # about the syntax of this string. # # Note: this configuration applies only to <= TLSv1.2. # # tls-ciphers DEFAULT:!MEDIUM # Configure allowed TLSv1.3 ciphersuites. See the ciphers(1ssl) manpage for more # information about the syntax of this string, and specifically for TLSv1.3 # ciphersuites. # # tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256 # When choosing a cipher, use the server's preference instead of the client # preference. By default, the server follows the client's preference. # # tls-prefer-server-ciphers yes # By default, TLS session caching is enabled to allow faster and less expensive # reconnections by clients that support it. Use the following directive to disable # caching. # # tls-session-caching no # Change the default number of TLS sessions cached. A zero value sets the cache # to unlimited size. The default size is 20480. # # tls-session-cache-size 5000 # Change the default timeout of cached TLS sessions. The default timeout is 300 # seconds. # # tls-session-cache-timeout 60 ################################# GENERAL ##################################### # By default Redis does not run as a daemon. Use 'yes' if you need it. # Note that Redis will write a pid file in /var/run/redis.pid when daemonized. daemonize no # If you run Redis from upstart or systemd, Redis can interact with your # supervision tree. Options: # supervised no - no supervision interaction # supervised upstart - signal upstart by putting Redis into SIGSTOP mode # supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET # supervised auto - detect upstart or systemd method based on # UPSTART_JOB or NOTIFY_SOCKET environment variables # Note: these supervision methods only signal "process is ready." # They do not enable continuous liveness pings back to your supervisor. supervised no # If a pid file is specified, Redis writes it where specified at startup # and removes it at exit. # # When the server runs non daemonized, no pid file is created if none is # specified in the configuration. When the server is daemonized, the pid file # is used even if not specified, defaulting to "/var/run/redis.pid". # # Creating a pid file is best effort: if Redis is not able to create it # nothing bad happens, the server will start and run normally. pidfile "/var/run/redis_6379.pid" # Specify the server verbosity level. # This can be one of: # debug (a lot of information, useful for development/testing) # verbose (many rarely useful info, but not a mess like the debug level) # notice (moderately verbose, what you want in production probably) # warning (only very important / critical messages are logged) loglevel notice # Specify the log file name. Also the empty string can be used to force # Redis to log on the standard output. Note that if you use standard # output for logging but daemonize, logs will be sent to /dev/null logfile "" # To enable logging to the system logger, just set 'syslog-enabled' to yes, # and optionally update the other syslog parameters to suit your needs. # syslog-enabled no # Specify the syslog identity. # syslog-ident redis # Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7. # syslog-facility local0 # Set the number of databases. The default database is DB 0, you can select # a different one on a per-connection basis using SELECT <dbid> where # dbid is a number between 0 and 'databases'-1 databases 16 # By default Redis shows an ASCII art logo only when started to log to the # standard output and if the standard output is a TTY. Basically this means # that normally a logo is displayed only in interactive sessions. # # However it is possible to force the pre-4.0 behavior and always show a # ASCII art logo in startup logs by setting the following option to yes. always-show-logo yes ################################ SNAPSHOTTING ################################ # # Save the DB on disk: # # save <seconds> <changes> # # Will save the DB if both the given number of seconds and the given # number of write operations against the DB occurred. # # In the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # Note: you can disable saving completely by commenting out all "save" lines. # # It is also possible to remove all the previously configured save # points by adding a save directive with a single empty string argument # like in the following example: # # # save "" save "" # save 900 1 # save 300 10 # save 60 10000 # By default Redis will stop accepting writes if RDB snapshots are enabled # (at least one save point) and the latest background save failed. # This will make the user aware (in a hard way) that data is not persisting # on disk properly, otherwise chances are that no one will notice and some # disaster will happen. # # If the background saving process will start working again Redis will # automatically allow writes again. # # However if you have setup your proper monitoring of the Redis server # and persistence, you may want to disable this feature so that Redis will # continue to work as usual even if there are problems with disk, # permissions, and so forth. stop-writes-on-bgsave-error yes # Compress string objects using LZF when dump .rdb databases? # For default that's set to 'yes' as it's almost always a win. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. rdbcompression yes # Since version 5 of RDB a CRC64 checksum is placed at the end of the file. # This makes the format more resistant to corruption but there is a performance # hit to pay (around 10%) when saving and loading RDB files, so you can disable it # for maximum performances. # # RDB files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the check. rdbchecksum yes # The filename where to dump the DB # dbfilename "dump.rdb" # Remove RDB files used by replication in instances without persistence # enabled. By default this option is disabled, however there are environments # where for regulations or other security concerns, RDB files persisted on # disk by masters in order to feed replicas, or stored on disk by replicas # in order to load them for the initial synchronization, should be deleted # ASAP. Note that this option ONLY WORKS in instances that have both AOF # and RDB persistence disabled, otherwise is completely ignored. # # An alternative (and sometimes better) way to obtain the same effect is # to use diskless replication on both master and replicas instances. However # in the case of replicas, diskless is not always an option. rdb-del-sync-files no # The working directory. # # The DB will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # # The Append Only File will also be created inside this directory. # # Note that you must specify a directory here, not a file name. dir "/data" ################################# REPLICATION ################################# # Master-Replica replication. Use replicaof to make a Redis instance a copy of # another Redis server. A few things to understand ASAP about Redis replication. # # +------------------+ +---------------+ # | Master | ---> | Replica | # | (receive writes) | | (exact copy) | # +------------------+ +---------------+ # # 1) Redis replication is asynchronous, but you can configure a master to # stop accepting writes if it appears to be not connected with at least # a given number of replicas. # 2) Redis replicas are able to perform a partial resynchronization with the # master if the replication link is lost for a relatively small amount of # time. You may want to configure the replication backlog size (see the next # sections of this file) with a sensible value depending on your needs. # 3) Replication is automatic and does not need user intervention. After a # network partition replicas automatically try to reconnect to masters # and resynchronize with them. # # replicaof <masterip> <masterport> # If the master is password protected (using the "requirepass" configuration # directive below) it is possible to tell the replica to authenticate before # starting the replication synchronization process, otherwise the master will # refuse the replica request. # # # However this is not enough if you are using Redis ACLs (for Redis version # 6 or greater), and the default user is not capable of running the PSYNC # command and/or other commands needed for replication. In this case it's # better to configure a special user to use with replication, and specify the # masteruser configuration as such: # #masteruser master # # When masteruser is specified, the replica will authenticate against its # master using the new AUTH form: AUTH <username> <password>. # When a replica loses its connection with the master, or when the replication # is still in progress, the replica can act in two different ways: # # 1) if replica-serve-stale-data is set to 'yes' (the default) the replica will # still reply to client requests, possibly with out of date data, or the # data set may just be empty if this is the first synchronization. # # 2) if replica-serve-stale-data is set to 'no' the replica will reply with # an error "SYNC with master in progress" to all the kind of commands # but to INFO, replicaOF, AUTH, PING, SHUTDOWN, REPLCONF, ROLE, CONFIG, # SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PUBLISH, PUBSUB, # COMMAND, POST, HOST: and LATENCY. # replica-serve-stale-data yes # You can configure a replica instance to accept writes or not. Writing against # a replica instance may be useful to store some ephemeral data (because data # written on a replica will be easily deleted after resync with the master) but # may also cause problems if clients are writing to it because of a # misconfiguration. # # Since Redis 2.6 by default replicas are read-only. # # Note: read only replicas are not designed to be exposed to untrusted clients # on the internet. It's just a protection layer against misuse of the instance. # Still a read only replica exports by default all the administrative commands # such as CONFIG, DEBUG, and so forth. To a limited extent you can improve # security of read only replicas using 'rename-command' to shadow all the # administrative / dangerous commands. replica-read-only yes # Replication SYNC strategy: disk or socket. # # New replicas and reconnecting replicas that are not able to continue the # replication process just receiving differences, need to do what is called a # "full synchronization". An RDB file is transmitted from the master to the # replicas. # # The transmission can happen in two different ways: # # 1) Disk-backed: The Redis master creates a new process that writes the RDB # file on disk. Later the file is transferred by the parent # process to the replicas incrementally. # 2) Diskless: The Redis master creates a new process that directly writes the # RDB file to replica sockets, without touching the disk at all. # # With disk-backed replication, while the RDB file is generated, more replicas # can be queued and served with the RDB file as soon as the current child # producing the RDB file finishes its work. With diskless replication instead # once the transfer starts, new replicas arriving will be queued and a new # transfer will start when the current one terminates. # # When diskless replication is used, the master waits a configurable amount of # time (in seconds) before starting the transfer in the hope that multiple # replicas will arrive and the transfer can be parallelized. # # With slow disks and fast (large bandwidth) networks, diskless replication # works better. repl-diskless-sync no # When diskless replication is enabled, it is possible to configure the delay # the server waits in order to spawn the child that transfers the RDB via socket # to the replicas. # # This is important since once the transfer starts, it is not possible to serve # new replicas arriving, that will be queued for the next RDB transfer, so the # server waits a delay in order to let more replicas arrive. # # The delay is specified in seconds, and by default is 5 seconds. To disable # it entirely just set it to 0 seconds and the transfer will start ASAP. repl-diskless-sync-delay 5 # ----------------------------------------------------------------------------- # WARNING: RDB diskless load is experimental. Since in this setup the replica # does not immediately store an RDB on disk, it may cause data loss during # failovers. RDB diskless load + Redis modules not handling I/O reads may also # cause Redis to abort in case of I/O errors during the initial synchronization # stage with the master. Use only if your do what you are doing. # ----------------------------------------------------------------------------- # # Replica can load the RDB it reads from the replication link directly from the # socket, or store the RDB to a file and read that file after it was completely # recived from the master. # # In many cases the disk is slower than the network, and storing and loading # the RDB file may increase replication time (and even increase the master's # Copy on Write memory and salve buffers). # However, parsing the RDB file directly from the socket may mean that we have # to flush the contents of the current database before the full rdb was # received. For this reason we have the following options: # # "disabled" - Don't use diskless load (store the rdb file to the disk first) # "on-empty-db" - Use diskless load only when it is completely safe. # "swapdb" - Keep a copy of the current db contents in RAM while parsing # the data directly from the socket. note that this requires # sufficient memory, if you don't have it, you risk an OOM kill. repl-diskless-load disabled # Replicas send PINGs to server in a predefined interval. It's possible to # change this interval with the repl_ping_replica_period option. The default # value is 10 seconds. # appendonly no # repl-ping-replica-period 10 # The following option sets the replication timeout for: # # 1) Bulk transfer I/O during SYNC, from the point of view of replica. # 2) Master timeout from the point of view of replicas (data, pings). # 3) Replica timeout from the point of view of masters (REPLCONF ACK pings). # # It is important to make sure that this value is greater than the value # specified for repl-ping-replica-period otherwise a timeout will be detected # every time there is low traffic between the master and the replica. # # repl-timeout 60 # Disable TCP_NODELAY on the replica socket after SYNC? # # If you select "yes" Redis will use a smaller number of TCP packets and # less bandwidth to send data to replicas. But this can add a delay for # the data to appear on the replica side, up to 40 milliseconds with # Linux kernels using a default configuration. # # If you select "no" the delay for data to appear on the replica side will # be reduced but more bandwidth will be used for replication. # # By default we optimize for low latency, but in very high traffic conditions # or when the master and replicas are many hops away, turning this to "yes" may # be a good idea. repl-disable-tcp-nodelay no # Set the replication backlog size. The backlog is a buffer that accumulates # replica data when replicas are disconnected for some time, so that when a # replica wants to reconnect again, often a full resync is not needed, but a # partial resync is enough, just passing the portion of data the replica # missed while disconnected. # # The bigger the replication backlog, the longer the time the replica can be # disconnected and later be able to perform a partial resynchronization. # # The backlog is only allocated once there is at least a replica connected. # # repl-backlog-size 1mb # After a master has no longer connected replicas for some time, the backlog # will be freed. The following option configures the amount of seconds that # need to elapse, starting from the time the last replica disconnected, for # the backlog buffer to be freed. # # Note that replicas never free the backlog for timeout, since they may be # promoted to masters later, and should be able to correctly "partially # resynchronize" with the replicas: hence they should always accumulate backlog. # # A value of 0 means to never release the backlog. # # repl-backlog-ttl 3600 # The replica priority is an integer number published by Redis in the INFO # output. It is used by Redis Sentinel in order to select a replica to promote # into a master if the master is no longer working correctly. # # A replica with a low priority number is considered better for promotion, so # for instance if there are three replicas with priority 10, 100, 25 Sentinel # will pick the one with priority 10, that is the lowest. # # However a special priority of 0 marks the replica as not able to perform the # role of master, so a replica with priority of 0 will never be selected by # Redis Sentinel for promotion. # # By default the priority is 100. replica-priority 100 # It is possible for a master to stop accepting writes if there are less than # N replicas connected, having a lag less or equal than M seconds. # # The N replicas need to be in "online" state. # # The lag in seconds, that must be <= the specified value, is calculated from # the last ping received from the replica, that is usually sent every second. # # This option does not GUARANTEE that N replicas will accept the write, but # will limit the window of exposure for lost writes in case not enough replicas # are available, to the specified number of seconds. # # For example to require at least 3 replicas with a lag <= 10 seconds use: # # min-replicas-to-write 3 # min-replicas-max-lag 10 # # Setting one or the other to 0 disables the feature. # # By default min-replicas-to-write is set to 0 (feature disabled) and # min-replicas-max-lag is set to 10. # A Redis master is able to list the address and port of the attached # replicas in different ways. For example the "INFO replication" section # offers this information, which is used, among other tools, by # Redis Sentinel in order to discover replica instances. # Another place where this info is available is in the output of the # "ROLE" command of a master. # # The listed IP and address normally reported by a replica is obtained # in the following way: # # IP: The address is auto detected by checking the peer address # of the socket used by the replica to connect with the master. # # Port: The port is communicated by the replica during the replication # handshake, and is normally the port that the replica is using to # listen for connections. # # However when port forwarding or Network Address Translation (NAT) is # used, the replica may be actually reachable via different IP and port # pairs. The following two options can be used by a replica in order to # report to its master a specific set of IP and port, so that both INFO # and ROLE will report those values. # # There is no need to use both the options if you need to override just # the port or the IP address. # # replica-announce-ip 5.5.5.5 # replica-announce-port 1234 ############################### KEYS TRACKING ################################# # Redis implements server assisted support for client side caching of values. # This is implemented using an invalidation table that remembers, using # 16 millions of slots, what clients may have certain subsets of keys. In turn # this is used in order to send invalidation messages to clients. Please # to understand more about the feature check this page: # # https://redis.io/topics/client-side-caching # # When tracking is enabled for a client, all the read only queries are assumed # to be cached: this will force Redis to store information in the invalidation # table. When keys are modified, such information is flushed away, and # invalidation messages are sent to the clients. However if the workload is # heavily dominated by reads, Redis could use more and more memory in order # to track the keys fetched by many clients. # # For this reason it is possible to configure a maximum fill value for the # invalidation table. By default it is set to 1M of keys, and once this limit # is reached, Redis will start to evict keys in the invalidation table # even if they were not modified, just to reclaim memory: this will in turn # force the clients to invalidate the cached values. Basically the table # maximum size is a trade off between the memory you want to spend server # side to track information about who cached what, and the ability of clients # to retain cached objects in memory. # # If you set the value to 0, it means there are no limits, and Redis will # retain as many keys as needed in the invalidation table. # In the "stats" INFO section, you can find information about the number of # keys in the invalidation table at every given moment. # # Note: when key tracking is used in broadcasting mode, no memory is used # in the server side so this setting is useless. # # tracking-table-max-keys 1000000 ################################## SECURITY ################################### # Warning: since Redis is pretty fast an outside user can try up to # 1 million passwords per second against a modern box. This means that you # should use very strong passwords, otherwise they will be very easy to break. # Note that because the password is really a shared secret between the client # and the server, and should not be memorized by any human, the password # can be easily a long string from /dev/urandom or whatever, so by using a # long and unguessable password no brute force attack will be possible. # Redis ACL users are defined in the following format: # # user <username> ... acl rules ... # # For example: # # user worker +@list +@connection ~jobs:* on >ffa9203c493aa99 # # The special username "default" is used for new connections. If this user # has the "nopass" rule, then new connections will be immediately authenticated # as the "default" user without the need of any password provided via the # AUTH command. Otherwise if the "default" user is not flagged with "nopass" # the connections will start in not authenticated state, and will require # AUTH (or the HELLO command AUTH option) in order to be authenticated and # start to work. # # The ACL rules that describe what an user can do are the following: # # on Enable the user: it is possible to authenticate as this user. # off Disable the user: it's no longer possible to authenticate # with this user, however the already authenticated connections # will still work. # +<command> Allow the execution of that command # -<command> Disallow the execution of that command # +@<category> Allow the execution of all the commands in such category # with valid categories are like @admin, @set, @sortedset, ... # and so forth, see the full list in the server.c file where # the Redis command table is described and defined. # The special category @all means all the commands, but currently # present in the server, and that will be loaded in the future # via modules. # +<command>|subcommand Allow a specific subcommand of an otherwise # disabled command. Note that this form is not # allowed as negative like -DEBUG|SEGFAULT, but # only additive starting with "+". # allcommands Alias for +@all. Note that it implies the ability to execute # all the future commands loaded via the modules system. # nocommands Alias for -@all. # ~<pattern> Add a pattern of keys that can be mentioned as part of # commands. For instance ~* allows all the keys. The pattern # is a glob-style pattern like the one of KEYS. # It is possible to specify multiple patterns. # allkeys Alias for ~* # resetkeys Flush the list of allowed keys patterns. # ><password> Add this passowrd to the list of valid password for the user. # For example >mypass will add "mypass" to the list. # This directive clears the "nopass" flag (see later). # <<password> Remove this password from the list of valid passwords. # nopass All the set passwords of the user are removed, and the user # is flagged as requiring no password: it means that every # password will work against this user. If this directive is # used for the default user, every new connection will be # immediately authenticated with the default user without # any explicit AUTH command required. Note that the "resetpass" # directive will clear this condition. # resetpass Flush the list of allowed passwords. Moreover removes the # "nopass" status. After "resetpass" the user has no associated # passwords and there is no way to authenticate without adding # some password (or setting it as "nopass" later). # reset Performs the following actions: resetpass, resetkeys, off, # -@all. The user returns to the same state it has immediately # after its creation. # # ACL rules can be specified in any order: for instance you can start with # passwords, then flags, or key patterns. However note that the additive # and subtractive rules will CHANGE MEANING depending on the ordering. # For instance see the following example: # # user alice on +@all -DEBUG ~* >somepassword # # This will allow "alice" to use all the commands with the exception of the # DEBUG command, since +@all added all the commands to the set of the commands # alice can use, and later DEBUG was removed. However if we invert the order # of two ACL rules the result will be different: # # user alice on -DEBUG +@all ~* >somepassword # # Now DEBUG was removed when alice had yet no commands in the set of allowed # commands, later all the commands are added, so the user will be able to # execute everything. # # Basically ACL rules are processed left-to-right. # # For more information about ACL configuration please refer to # the Redis web site at https://redis.io/topics/acl # ACL LOG # # The ACL Log tracks failed commands and authentication events associated # with ACLs. The ACL Log is useful to troubleshoot failed commands blocked # by ACLs. The ACL Log is stored in memory. You can reclaim memory with # ACL LOG RESET. Define the maximum entry length of the ACL Log below. acllog-max-len 128 # Using an external ACL file # # Instead of configuring users here in this file, it is possible to use # a stand-alone file just listing users. The two methods cannot be mixed: # if you configure users here and at the same time you activate the exteranl # ACL file, the server will refuse to start. # # The format of the external ACL user file is exactly the same as the # format that is used inside redis.conf to describe users. # # aclfile /etc/redis/users.acl # IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatiblity # layer on top of the new ACL system. The option effect will be just setting # the password for the default user. Clients will still authenticate using # AUTH <password> as usually, or more explicitly with AUTH default <password> # if they follow the new protocol: both will work. #
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # --- apiVersion: v1 kind: ConfigMap metadata: name: sentinel-scripts data: sentinel_init.sh: | #!/bin/bash for i in ${REDIS_NODES//,/ } do echo "finding master at $i" MASTER=$(redis-cli --no-auth-warning --raw -h $i -a ${REDIS_PASSWORD} info replication | awk '{print $1}' | grep master_host: | cut -d ":" -f2) if [ "${MASTER}" == "" ]; then echo "no master found" MASTER= else echo "found ${MASTER}" break fi done #echo "sentinel monitor mymaster MASTER 6379 2" >> /tmp/master echo "port 5000 sentinel resolve-hostnames yes sentinel announce-hostnames yes sentinel monitor mymaster MASTER_HOST 6379 2 sentinel down-after-milliseconds mymaster 1000 sentinel failover-timeout mymaster 10000 sentinel parallel-syncs mymaster 1 sentinel sentinel-pass ${REDIS_PASSWORD} sentinel auth-pass mymaster ${REDIS_PASSWORD} requirepass ${REDIS_PASSWORD} sentinel announce-ip ${HOSTNAME}.sentinel sentinel announce-port 5000 " > /etc/redis/sentinel.conf cat /etc/redis/sentinel.conf sed -i "s@MASTER_HOST@${MASTER}@g" /etc/redis/sentinel.conf redis_init.sh: | #!/bin/bash cp /tmp/redis/redis.conf /etc/redis/redis.conf echo "requirepass ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "masterauth ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "replica-announce-ip ${HOSTNAME}.redis" >> /etc/redis/redis.conf echo "replica-announce-port 6379 " >> /etc/redis/redis.conf echo "finding master..." if [ "$(timeout 5 redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} ping)" != "PONG" ]; then echo "sentinel not found, defaulting to redis-0" if [ ${HOSTNAME} == "redis-0" ]; then echo "this is redis-0, not updating config..." else echo "updating redis.conf..." echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof redis-0.redis 6379" >> /etc/redis/redis.conf fi else echo "sentinel found, finding master" MASTER="$(redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} sentinel get-master-addr-by-name mymaster | grep -E '(^redis-*)|([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})')" if [ "${HOSTNAME}.redis" == ${MASTER} ]; then echo "this is master, not updating config..." else echo "master found : ${MASTER}, updating redis.conf" echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof ${MASTER} 6379" >> /etc/redis/redis.conf echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf fi fi
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # --- kind: Secret apiVersion: v1 metadata: name: redis-secret type: Opaque data: REDIS_PASSWORD: "bWFnZWR1LmNvbQ==" #REDIS_PASSWORD: "magedu.com"
# Headless service so sentinel could access redisses using syntax <pod-name>.<service-name> apiVersion: v1 kind: Service metadata: name: redis labels: app: redis app.kubernetes.io/component: redis app.kubernetes.io/instance: redis spec: clusterIP: None ports: - port: 6379 targetPort: 6379 name: redis selector: app: redis --- # Sentinel service used for project pod connection apiVersion: v1 kind: Service metadata: name: sentinel labels: app: sentinel app.kubernetes.io/component: sentinel app.kubernetes.io/instance: sentinel spec: type: ClusterIP sessionAffinity: None ports: - port: 5000 targetPort: 5000 name: sentinel selector: app: sentinel
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # --- apiVersion: apps/v1 kind: StatefulSet metadata: name: redis spec: serviceName: redis replicas: 3 selector: matchLabels: app: redis template: metadata: labels: app: redis annotations: prometheus.io/scrape: "true" prometheus.io/port: "9121" prometheus.io/path: "/metrics" # Grafana Dashboard ID: 763 spec: initContainers: - name: config #image: registry.magedu.com/redis/redis:7.0 image: redis:7.2 env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD command: [ "sh", "-c", "/scripts/redis_init.sh" ] volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: config mountPath: /tmp/redis/ - name: init-script mountPath: /scripts/ containers: - name: redis #image: registry.magedu.com/redis/redis:7.0 image: redis:7.2 command: ["redis-server"] args: ["/etc/redis/redis.conf"] ports: - containerPort: 6379 name: redis volumeMounts: - name: data mountPath: /data - name: redis-config mountPath: /etc/redis/ - name: redis-exporter #image: registry.magedu.com/oliver006/redis_exporter:v1.55.0-alpine image: oliver006/redis_exporter:v1.55.0-alpine env: - name: REDIS_ADDR value: "redis://localhost:6379" - name: REDIS_USER value: "default" - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD ports: - name: redis-exporter containerPort: 9121 volumes: - name: data persistentVolumeClaim: claimName: data - name: redis-config emptyDir: {} - name: init-script configMap: name: sentinel-scripts defaultMode: 0777 items: - key: redis_init.sh path: redis_init.sh - name: config configMap: name: redis-conf items: - key: redis.conf path: redis.conf volumeClaimTemplates: - metadata: name: data labels: app: redis spec: accessModes: [ "ReadWriteOnce" ] storageClassName: openebs-hostpath resources: requests: storage: 10Gi
部署完成后，其master的访问地址为“redis-0.redis.redis.svc”，客户端可通过此地址向redis发起存取请求。
部署sentinel（可选）
最后，部署redis sentinel。此为可选步骤。
kubectl apply -f ./sentinel/ -n redis
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# ll -h sentinel/ total 16K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 3 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 2.8K May 1 07:56 01-configmap-sentinel.yaml -rw-r--r-- 1 root root 1.8K May 1 07:56 02-statefulset-sentinel.yaml root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis#
# Maintainer: MageEdu <mage@magedu.com> # Site: http://www.magedu.com # --- apiVersion: v1 kind: ConfigMap metadata: name: sentinel-scripts data: sentinel_init.sh: | #!/bin/bash for i in ${REDIS_NODES//,/ } do echo "finding master at $i" MASTER=$(redis-cli --no-auth-warning --raw -h $i -a ${REDIS_PASSWORD} info replication | awk '{print $1}' | grep master_host: | cut -d ":" -f2) if [ "${MASTER}" == "" ]; then echo "no master found" MASTER= else echo "found ${MASTER}" break fi done #echo "sentinel monitor mymaster MASTER 6379 2" >> /tmp/master echo "port 5000 sentinel resolve-hostnames yes sentinel announce-hostnames yes sentinel monitor mymaster MASTER_HOST 6379 2 sentinel down-after-milliseconds mymaster 1000 sentinel failover-timeout mymaster 10000 sentinel parallel-syncs mymaster 1 sentinel sentinel-pass ${REDIS_PASSWORD} sentinel auth-pass mymaster ${REDIS_PASSWORD} requirepass ${REDIS_PASSWORD} sentinel announce-ip ${HOSTNAME}.sentinel sentinel announce-port 5000 " > /etc/redis/sentinel.conf cat /etc/redis/sentinel.conf sed -i "s@MASTER_HOST@${MASTER}@g" /etc/redis/sentinel.conf redis_init.sh: | #!/bin/bash cp /tmp/redis/redis.conf /etc/redis/redis.conf echo "requirepass ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "masterauth ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "replica-announce-ip ${HOSTNAME}.redis" >> /etc/redis/redis.conf echo "replica-announce-port 6379 " >> /etc/redis/redis.conf echo "finding master..." if [ "$(timeout 5 redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} ping)" != "PONG" ]; then echo "sentinel not found, defaulting to redis-0" if [ ${HOSTNAME} == "redis-0" ]; then echo "this is redis-0, not updating config..." else echo "updating redis.conf..." echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof redis-0.redis 6379" >> /etc/redis/redis.conf fi else echo "sentinel found, finding master" MASTER="$(redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} sentinel get-master-addr-by-name mymaster | grep -E '(^redis-*)|([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})')" if [ "${HOSTNAME}.redis" == ${MASTER} ]; then echo "this is master, not updating config..." else echo "master found : ${MASTER}, updating redis.conf" echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof ${MASTER} 6379" >> /etc/redis/redis.conf echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf fi fi
apiVersion: apps/v1 kind: StatefulSet metadata: name: sentinel spec: serviceName: sentinel replicas: 3 selector: matchLabels: app: sentinel template: metadata: labels: app: sentinel spec: initContainers: - name: config #image: registry.magedu.com/redis/redis:7.0 image: redis:7.0 env: - name: REDIS_NODES valueFrom: configMapKeyRef: name: redis-conf key: REDIS_NODES - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD command: [ "sh", "-c", "/scripts/sentinel_init.sh" ] volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: init-script mountPath: /scripts/ containers: - name: sentinel #image: registry.magedu.com/redis/redis:7.0 image: redis:7.0 command: ["redis-sentinel"] args: ["/etc/redis/sentinel.conf"] ports: - containerPort: 5000 name: sentinel volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: data mountPath: /data volumes: - name: init-script configMap: name: sentinel-scripts defaultMode: 0777 items: - key: sentinel_init.sh path: sentinel_init.sh - name: redis-config emptyDir: {} - name: data persistentVolumeClaim: claimName: data volumeClaimTemplates: - metadata: name: data labels: app: redis-sentinel spec: accessModes: [ "ReadWriteOnce" ] storageClassName: openebs-hostpath resources: requests: storage: 10Gi
安装详情
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/03-Redis# kubectl create namespace redis namespace/redis created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services/03-Redis# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# kubectl apply -f . -n redis configmap/redis-conf created configmap/sentinel-scripts created secret/redis-secret created service/redis created service/sentinel created statefulset.apps/redis created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# kubectl apply -f ./sentinel/ -n redis configmap/sentinel-scripts unchanged statefulset.apps/sentinel created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# kubectl get pod -n redis -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES redis-0 2/2 Running 0 11h 10.244.3.145 k8s-cilium-node-03 <none> <none> redis-1 2/2 Running 0 11h 10.244.6.219 k8s-cilium-node-05 <none> <none> redis-2 2/2 Running 0 10h 10.244.1.225 k8s-cilium-node-01 <none> <none> sentinel-0 1/1 Running 0 8m39s 10.244.1.169 k8s-cilium-node-01 <none> <none> sentinel-1 1/1 Running 0 8m36s 10.244.4.178 k8s-cilium-node-04 <none> <none> sentinel-2 1/1 Running 0 8m33s 10.244.2.13 k8s-cilium-node-02 <none> <none> root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis# kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE elastic data-es-cluster-0 Bound pvc-433e72cf-4f56-4919-a15e-253534da3fbd 10Gi RWO openebs-hostpath 14h elastic data-es-cluster-1 Bound pvc-f69be8d5-016f-41b1-8a78-2a06167f57e6 10Gi RWO openebs-hostpath 13h elastic data-es-cluster-2 Bound pvc-c3739d0e-8e23-4112-ab02-f1ef58411e59 10Gi RWO openebs-hostpath 13h nacos data-mysql-0 Bound pvc-84cfc00e-a254-4e23-b718-cf9a38c65ac0 10Gi RWO openebs-hostpath 17h nacos data-mysql-1 Bound pvc-89fa2b48-51f6-406e-b2e7-cef565c2777d 10Gi RWO openebs-hostpath 17h nacos data-nacos-0 Bound pvc-00ce3f9b-c937-4efb-88a6-52b56e007398 10Gi RWO openebs-hostpath 16h nacos data-nacos-1 Bound pvc-d66a4ad9-5222-4d0f-bf1a-64288ce1a0fb 10Gi RWO openebs-hostpath 16h nacos data-nacos-2 Bound pvc-3f9a0d55-4171-466b-a1e6-673a5c6b7b05 10Gi RWO openebs-hostpath 16h prom grafana-pvc Bound pvc-9529a417-586b-4178-8c1b-36d53f8805d3 5Gi RWO openebs-hostpath 2d16h redis data-redis-0 Bound pvc-41e1c427-4177-4b3b-8c38-c29ae255e6a2 10Gi RWO openebs-hostpath 11h redis data-redis-1 Bound pvc-af0ac6fd-5ca8-406c-bbf0-6cc15b7d8184 10Gi RWO openebs-hostpath 11h redis data-redis-2 Bound pvc-be7aaa8b-c3d1-4686-828c-5e0da038cdf6 10Gi RWO openebs-hostpath 10h redis data-sentinel-0 Bound pvc-01105804-5575-4074-8fd6-b971357d1a73 10Gi RWO openebs-hostpath 11h redis data-sentinel-1 Bound pvc-326c7597-eb24-4be6-9dc1-a2c97f2ccea6 10Gi RWO openebs-hostpath 11h redis data-sentinel-2 Bound pvc-609fc9e4-7c5e-4384-a8f2-0206689b21be 10Gi RWO openebs-hostpath 11m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/03-Redis#
2.4、部署rabbitmq集群
首先，运行如下命令，创建名称空间。
kubectl create namespace rabbit
而后，运行如下命令，部署Rabbit Cluster。
kubectl apply -f ./ -n rabbit
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ# ll -h total 40K drwxr-xr-x 3 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 551 May 1 07:56 01-configmap-rabbitmq.yaml -rw-r--r-- 1 root root 467 May 1 07:56 02-rbac-rabbitmq.yaml -rw-r--r-- 1 root root 234 May 1 07:56 03-secret-rabbitmq.yaml -rw-r--r-- 1 root root 454 May 1 07:56 04-service-rabbitmq.yaml -rw-r--r-- 1 root root 3.0K May 1 07:56 05-statefulset-rabbitmq.yaml -rw-r--r-- 1 root root 883 May 1 07:56 06-ingress-rabbitmq.yaml drwxr-xr-x 2 root root 4.0K May 1 07:56 manifests/ -rw-r--r-- 1 root root 1000 May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ#
apiVersion: v1 kind: ConfigMap metadata: name: rabbitmq-config data: enabled_plugins: | [rabbitmq_federation,rabbitmq_management,rabbitmq_management_agent,rabbitmq_peer_discovery_k8s,rabbitmq_prometheus]. rabbitmq.conf: | loopback_users.guest = false listeners.tcp.default = 5672 cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s cluster_formation.k8s.host = kubernetes.default.svc.cluster.local cluster_formation.k8s.address_type = hostname cluster_formation.node_cleanup.only_log_warning = true
--- apiVersion: v1 kind: ServiceAccount metadata: name: rabbitmq --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rabbitmq rules: - apiGroups: - "" resources: - endpoints verbs: - get - list - watch --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rabbitmq subjects: - kind: ServiceAccount name: rabbitmq roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rabbitmq
apiVersion: v1 kind: Secret metadata: name: rabbit-secret type: Opaque data: erlang_cookie: U2l0ZS9NYWdlRWR1LmNvbQo= # Default User admin default_user: "YWRtaW4=" # Default Pass magedu.com default_pass: "bWFnZWR1LmNvbQ=="
--- apiVersion: v1 kind: Service metadata: name: rabbitmq annotations: prometheus.io/scrape: "true" prometheus.io/port: "15692" prometheus.io/path: "/metrics" # Grafana Dashboard ID 10991 and 11340. spec: clusterIP: None ports: - port: 15672 targetPort: 15672 name: discovery - port: 15692 targetPort: 15692 name: prometheus - port: 5672 targetPort: 5672 name: amqp selector: app: rabbitmq ---
# --- apiVersion: apps/v1 kind: StatefulSet metadata: name: rabbitmq spec: serviceName: rabbitmq replicas: 3 selector: matchLabels: app: rabbitmq template: metadata: labels: app: rabbitmq annotations: {} #prometheus.io/scrape: "true" #prometheus.io/port: "15692" #prometheus.io/path: "/metrics" # Grafana Dashboard ID 10991 and 11340. spec: serviceAccountName: rabbitmq initContainers: - name: config image: ikubernetes/admin-box:v1.2 command: ['/bin/sh', '-c', 'cp /tmp/config/rabbitmq.conf /config/rabbitmq.conf && ls -l /config/ && cp /tmp/config/enabled_plugins /etc/rabbitmq/enabled_plugins'] volumeMounts: - name: config mountPath: /tmp/config/ readOnly: false - name: config-file mountPath: /config/ - name: plugins-file mountPath: /etc/rabbitmq/ containers: - name: rabbitmq #image: registry.magedu.com/rabbitmq/rabbitmq:3.12-management image: rabbitmq:3.13-management ports: - containerPort: 15672 name: discovery - containerPort: 15692 name: prometheus - containerPort: 5672 name: amqp env: - name: RABBIT_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: RABBIT_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: RABBITMQ_NODENAME value: rabbit@$(RABBIT_POD_NAME).rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local - name: RABBITMQ_USE_LONGNAME value: "true" - name: RABBITMQ_CONFIG_FILE value: "/config/rabbitmq" - name: RABBITMQ_ERLANG_COOKIE valueFrom: secretKeyRef: name: rabbit-secret key: erlang_cookie - name: RABBITMQ_DEFAULT_USER valueFrom: secretKeyRef: name: rabbit-secret key: default_user - name: RABBITMQ_DEFAULT_PASS valueFrom: secretKeyRef: name: rabbit-secret key: default_pass - name: K8S_HOSTNAME_SUFFIX value: .rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local volumeMounts: - name: data mountPath: /var/lib/rabbitmq readOnly: false - name: config-file mountPath: /config/ - name: plugins-file mountPath: /etc/rabbitmq/ volumes: - name: config-file emptyDir: {} - name: plugins-file emptyDir: {} - name: config configMap: name: rabbitmq-config defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: openebs-hostpath resources: requests: storage: 5Gi ---
--- apiVersion: v1 kind: Service metadata: name: rabbitmq-svc annotations: prometheus.io/scrape: "true" prometheus.io/port: "15692" prometheus.io/path: "/metrics" # Grafana Dashboard ID 10991 and 11340. spec: ports: - port: 15672 targetPort: 15672 name: discovery - port: 15692 targetPort: 15692 name: prometheus - port: 5672 targetPort: 5672 name: amqp selector: app: rabbitmq --- --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rabbitmq annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: rabbitmq.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: rabbitmq-svc port: number: 15672
类似如下的URL可用于访问RabbitMQ内置的管理Web UI。 http://rabbitmq.k8s.com
安装详情：
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ# kubectl create namespace rabbit
namespace/rabbit created
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ#
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ# kubectl apply -f ./ -n rabbit
configmap/rabbitmq-config created
serviceaccount/rabbitmq created
role.rbac.authorization.k8s.io/rabbitmq created
rolebinding.rbac.authorization.k8s.io/rabbitmq created
secret/rabbit-secret created
service/rabbitmq created
statefulset.apps/rabbitmq created
service/rabbitmq-svc created
ingress.networking.k8s.io/rabbitmq created
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ#
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ# kubectl get pvc -n rabbit
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-rabbitmq-0 Bound pvc-a154978e-0f1e-4806-a781-79279cd3bd8a 5Gi RWO openebs-hostpath 123m
data-rabbitmq-1 Bound pvc-1988ec6c-b278-4590-998f-f5cb693f8683 5Gi RWO openebs-hostpath 96m
data-rabbitmq-2 Bound pvc-edc54669-8089-41ee-b665-26ef1e106d37 5Gi RWO openebs-hostpath 74m
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ# kubectl get ingress -n rabbit
NAME CLASS HOSTS ADDRESS PORTS AGE
rabbitmq cilium rabbitmq.k8s.com 172.16.88.90 80 123m
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/04-RabbitMQ#
为mall-microservice提供服务时,需要创建新的用户malladmin/magedu.com,并创建新的vhost，名称为/mall，并授权给malladmin用户
添加虚拟主机
授权malladmin
通过admin查看malladmin可以访问mall主机
2.5、部署MongoDB ReplicaSet集群
部署方法
直接将各配置文件创建在集群上即可，建议使用专用的namespace；
kubectl create namespace mongo
kubectl apply -f . -n mongo
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# ll -h total 24K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 409 May 1 07:56 01-configmap-mongodb.yaml -rw-r--r-- 1 root root 206 May 1 07:56 02-service-mongodb.yaml -rw-r--r-- 1 root root 2.3K May 1 07:56 03-statefulset-mongodb.yaml -rw-r--r-- 1 root root 918 May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB#
apiVersion: v1 kind: ConfigMap metadata: name: mongo-config data: script.sh: |- #!/bin/bash if [[ "$POD_NAME" = "mongodb-0" ]]; then export MONGODB_REPLICA_SET_MODE="primary" else export MONGODB_INITIAL_PRIMARY_PORT_NUMBER="27017" export MONGODB_REPLICA_SET_MODE="secondary" fi exec /opt/bitnami/scripts/mongodb/entrypoint.sh /opt/bitnami/scripts/mongodb/run.sh
apiVersion: v1 kind: Service metadata: name: mongodb spec: clusterIP: None publishNotReadyAddresses: true ports: - name: mongodb port: 27017 targetPort: 27017 selector: app: mongodb
apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb spec: serviceName: mongodb selector: matchLabels: app: mongodb replicas: 3 template: metadata: labels: app: mongodb annotations: prometheus.io/scrape: "true" prometheus.io/port: "9216" prometheus.io/path: "/metrics" spec: securityContext: fsGroup: 1001 containers: - name: mongodb #image: registry.magedu.com/bitnami/mongodb:5.0.19 image: bitnami/mongodb:5.0.24 imagePullPolicy: IfNotPresent env: - name: MONGODB_DISABLE_SYSTEM_LOG value: "false" - name: MONGODB_SYSTEM_LOG_VERBOSITY value: "1" - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MONGODB_REPLICA_SET_NAME value: "replicaset" - name: MONGODB_INITIAL_PRIMARY_HOST value: "mongodb-0.mongodb" - name: MONGODB_ADVERTISED_HOSTNAME value: "$(POD_NAME).mongodb" - name: ALLOW_EMPTY_PASSWORD value: "yes" command: - /scripts/script.sh securityContext: runAsUser: 1001 volumeMounts: - name: config mountPath: /scripts - name: data mountPath: /bitnami/mongodb - name: mongodb-exporter #image: registry.magedu.com/percona/mongodb_exporter:0.39 image: percona/mongodb_exporter:0.40 args: - --collect-all env: - name: MONGODB_URI value: "mongodb://127.0.0.1:27017/admin?ssl=false" ports: - name: mongo-exporter containerPort: 9216 volumes: - name: config configMap: name: mongo-config defaultMode: 0755 items: - key: script.sh path: script.sh volumeClaimTemplates: - metadata: name: data spec: storageClassName: "openebs-hostpath" accessModes: [ReadWriteOnce] resources: requests: storage: 10Gi
查看集群状态
kubectl exec -it mongodb-0 -n mongo -- mongo
mongo> rs.status()
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# kubectl create namespace mongo namespace/mongo created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# kubectl apply -f . -n mongo configmap/mongo-config created service/mongodb created statefulset.apps/mongodb created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# kubectl exec -it mongodb-0 -n mongo -- mongo Defaulted container "mongodb" out of: mongodb, mongodb-exporter MongoDB shell version v5.0.24 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("55981e1f-c16a-49d4-a424-ca4dc14c1f42") } MongoDB server version: 5.0.24 ================ Warning: the "mongo" shell has been superseded by "mongosh", which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in an upcoming release. For installation instructions, see https://docs.mongodb.com/mongodb-shell/install/ ================ --- The server generated these startup warnings when booting: 2024-05-02T03:00:14.194+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem --- > rs.status() { "ok" : 0, "errmsg" : "no replset config has been received", "code" : 94, "codeName" : "NotYetInitialized" } > exit root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB# kubectl get pvc -n mongo NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-mongodb-0 Bound pvc-400ce2e1-1f57-4c8e-87c0-cc0ac1aecd4a 10Gi RWO openebs-hostpath 63m data-mongodb-1 Bound pvc-4e34d569-31b5-4f0e-b8fa-a4cc5d5c9906 10Gi RWO openebs-hostpath 52m data-mongodb-2 Bound pvc-bd21864c-d021-4a4b-a3de-0318dbf553fb 10Gi RWO openebs-hostpath 27m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/05-MongoDB#
2.6、部署MinIO
将配置清单中定义的资源对象部署于Kubernetes集群上即可，需要手动指定名称空间；
kubectl create namespace minio
kubectl apply -f ./ -n minio
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# ll -h total 28K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 483 May 1 07:56 01-services-minio.yaml -rw-r--r-- 1 root root 196 May 1 07:56 02-secret-minio.yaml -rw-r--r-- 1 root root 2.2K May 1 07:56 03-statefulset-minio.yaml -rw-r--r-- 1 root root 429 May 1 07:56 04-ingress-minio.yaml -rw-r--r-- 1 root root 2.0K May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO#
--- kind: Service apiVersion: v1 metadata: name: minio-headless labels: app: minio spec: clusterIP: None #publishNotReadyAddresses: true selector: app: minio ports: - name: http port: 9000 targetPort: 9000 --- apiVersion: v1 kind: Service metadata: name: minio spec: type: LoadBalancer selector: app: minio ports: - port: 9000 targetPort: 9000 name: http - port: 9001 targetPort: 9001 name: console ---
apiVersion: v1 kind: Secret metadata: name: minio-secret data: MINIO_ROOT_USER: bWluaW9hZG1pbgo= # username: minioadmin MINIO_ROOT_PASSWORD: bWFnZWR1LmNvbQ== # root password: magedu.com
--- apiVersion: apps/v1 kind: StatefulSet metadata: name: minio labels: app: minio spec: serviceName: "minio-headless" replicas: 4 podManagementPolicy: "Parallel" selector: matchLabels: app: minio template: metadata: labels: app: minio annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: "/minio/v2/metrics/node" #prometheus.io/path: "/minio/v2/metrics/cluster" #prometheus.io/path: "/minio/v2/metrics/bucket" spec: containers: - name: minio #image: registry.magedu.com/minio/minio:RELEASE.2023-08-04T17-40-21Z image: minio/minio:RELEASE.2024-03-10T02-53-48Z volumeMounts: - name: data mountPath: /data args: - server - http://minio-{0...3}.minio-headless.$(MINIO_POD_NAMESPACE).svc.cluster.local/data - "--address=:9000" - "--console-address=:9001" env: # MinIO access key and secret key - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-secret key: MINIO_ROOT_USER - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-secret key: MINIO_ROOT_PASSWORD - name: MINIO_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MINIO_PROMETHEUS_AUTH_TYPE value: "public" livenessProbe: failureThreshold: 3 httpGet: path: /minio/health/live port: http scheme: HTTP initialDelaySeconds: 120 periodSeconds: 15 successThreshold: 1 timeoutSeconds: 10 ports: - containerPort: 9000 name: http protocol: TCP volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: "openebs-hostpath" # This field references the existing StorageClass
--- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: minio.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: minio port: number: 9001
访问console
通过Ingress定义的Host访问，地址如下，注意要使用https协议。
https://minio.k8s.com/
默认的用户名和密码是“minioadmin/magedu.com”。
安装详情：
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# kubectl create namespace minio namespace/minio created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# kubectl apply -f ./ -n minio service/minio-headless created service/minio created secret/minio-secret created statefulset.apps/minio created ingress.networking.k8s.io/minio created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# kubectl get pvc -n minio NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-minio-0 Bound pvc-4ee77390-8775-4eb8-8927-3873626864d7 10Gi RWO openebs-hostpath 62m data-minio-1 Bound pvc-6ae32673-c563-4649-9716-328d8a190849 10Gi RWO openebs-hostpath 62m data-minio-2 Bound pvc-f88be4bb-b780-4cb3-a246-8a3e66af3212 10Gi RWO openebs-hostpath 62m data-minio-3 Bound pvc-4f5682b4-8a7d-4617-8f25-2b2c96eddcb8 10Gi RWO openebs-hostpath 62m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO# kubectl get ingress -n minio NAME CLASS HOSTS ADDRESS PORTS AGE minio cilium minio.k8s.com 172.16.88.90 80 62m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/06-MinIO#
2.7、部署SkyWalking及UI
首先，创建专用的名称空间，以部署Skywalking及相关组件。
kubectl create namespace tracing
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# ll -h total 20K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 10 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 1.2K May 1 07:56 01-skywalking-oap.yaml -rw-r--r-- 1 root root 1.3K May 1 07:56 02-skywalking-ui.yaml -rw-r--r-- 1 root root 513 May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking#
apiVersion: apps/v1 kind: Deployment metadata: name: skywalking-oap labels: app: skywalking-oap spec: selector: matchLabels: app: skywalking-oap template: metadata: labels: app: skywalking-oap spec: containers: - name: skywalking-oap #image: registry.magedu.com/apache/skywalking-oap-server:9.6.0 image: apache/skywalking-oap-server:9.7.0 env: - name: SW_HEALTH_CHECKER value: default - name: SW_STORAGE value: elasticsearch - name: SW_STORAGE_ES_CLUSTER_NODES value: elasticsearch.elastic.svc:9200 readinessProbe: exec: command: - /skywalking/bin/swctl - health initialDelaySeconds: 30 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: labels: name: skywalking-oap name: skywalking-oap spec: ports: - port: 11800 targetPort: 11800 name: grpc - port: 12800 targetPort: 12800 name: http-query selector: app: skywalking-oap ---
--- apiVersion: apps/v1 kind: Deployment metadata: name: skywalking-ui labels: app: skywalking-ui spec: selector: matchLabels: app: skywalking-ui template: metadata: labels: app: skywalking-ui spec: containers: - name: skywalking-ui #image: registry.magedu.com/apache/skywalking-ui:9.6.0 image: apache/skywalking-ui:9.7.0 env: - name: SW_OAP_ADDRESS value: http://skywalking-oap:12800 readinessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 30 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: labels: name: skywalking-ui name: skywalking-ui spec: ports: - port: 8080 targetPort: 8080 name: http selector: app: skywalking-ui --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: skywalking annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: skywalking.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: skywalking-ui port: number: 8080
而后，运行如下命令，部署Skywalking OAP。需要说明的是，下面命令中用到的配置文件，依赖于部署在elastic名称空间中的elasticsearch服务。
kubectl apply -f 01-skywalking-oap.yaml -n tracing
最后，运行如下命令，部署Skywalking UI。
kubectl apply -f 02-skywalking-ui.yaml -n tracing
安装详情：
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# kubectl create namespace tracing namespace/tracing created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# kubectl apply -f 01-skywalking-oap.yaml -n tracing deployment.apps/skywalking-oap created service/skywalking-oap created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# kubectl apply -f 02-skywalking-ui.yaml -n tracing deployment.apps/skywalking-ui created service/skywalking-ui created ingress.networking.k8s.io/skywalking created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking# kubectl get ingress -n tracing NAME CLASS HOSTS ADDRESS PORTS AGE skywalking cilium skywalking.k8s.com 172.16.88.90 80 74m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/infra-services-with-prometheus/07-Skywalking#
三、商城业务部署
3.1、导入商城数据库
root@k8s-cilium-master-01:~# git clone https://gitee.com/mageedu/mall-microservice.git Cloning into 'mall-microservice'... remote: Enumerating objects: 1127, done. remote: Total 1127 (delta 0), reused 0 (delta 0), pack-reused 1127 Receiving objects: 100% (1127/1127), 3.57 MiB | 525.00 KiB/s, done. Resolving deltas: 100% (509/509), done. root@k8s-cilium-master-01:~# root@k8s-cilium-master-01:~# cd mall-microservice/document/sql/ root@k8s-cilium-master-01:~/mall-microservice/document/sql# ll -h total 228K drwxr-xr-x 2 root root 4.0K May 2 07:03 ./ drwxr-xr-x 10 root root 4.0K May 2 07:03 ../ -rw-r--r-- 1 root root 216K May 2 07:03 mall.sql -rw-r--r-- 1 root root 38 May 2 07:03 README.md root@k8s-cilium-master-01:~/mall-microservice/document/sql# apt-get install mysql-client -y root@k8s-cilium-master-01:~/mall-microservice/document/sql# kubectl get pod -n nacos -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-0 3/3 Running 3 (20h ago) 23h 10.244.4.249 k8s-cilium-node-04 <none> <none> mysql-1 3/3 Running 3 (20h ago) 22h 10.244.2.188 k8s-cilium-node-02 <none> <none> nacos-0 1/1 Running 1 (20h ago) 21h 10.244.6.210 k8s-cilium-node-05 <none> <none> nacos-1 1/1 Running 1 (20h ago) 21h 10.244.2.155 k8s-cilium-node-02 <none> <none> nacos-2 1/1 Running 1 (20h ago) 21h 10.244.4.112 k8s-cilium-node-04 <none> <none> root@k8s-cilium-master-01:~/mall-microservice/document/sql# mysql -uroot -p -h10.244.4.249 < ./mall.sql Enter password: root@k8s-cilium-master-01:~/mall-microservice/document/sql# 验证数据库 root@k8s-cilium-master-01:~/mall-microservice/document/sql# mysql -uroot -p -h10.244.4.249 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 55550 Server version: 5.7.39-log MySQL Community Server (GPL) Copyright (c) 2000, 2024, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> show databases; +------------------------+ | Database | +------------------------+ | information_schema | | mall | | mysql | | nacosdb | | performance_schema | | sys | | xtrabackup_backupfiles | +------------------------+ 7 rows in set (0.01 sec) mysql> use mall; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +-----------------------------------------+ | Tables_in_mall | +-----------------------------------------+ | cms_help | | cms_help_category | | cms_member_report | | cms_prefrence_area | | cms_prefrence_area_product_relation | | cms_subject | | cms_subject_category | | cms_subject_comment | | cms_subject_product_relation | | cms_topic | | cms_topic_category | | cms_topic_comment | | oms_cart_item | | oms_company_address | | oms_order | | oms_order_item | | oms_order_operate_history | | oms_order_return_apply | | oms_order_return_reason | | oms_order_setting | | pms_album | | pms_album_pic | | pms_brand | | pms_comment | | pms_comment_replay | | pms_feight_template | | pms_member_price | | pms_product | | pms_product_attribute | | pms_product_attribute_category | | pms_product_attribute_value | | pms_product_category | | pms_product_category_attribute_relation | | pms_product_full_reduction | | pms_product_ladder | | pms_product_operate_log | | pms_product_vertify_record | | pms_sku_stock | | sms_coupon | | sms_coupon_history | | sms_coupon_product_category_relation | | sms_coupon_product_relation | | sms_flash_promotion | | sms_flash_promotion_log | | sms_flash_promotion_product_relation | | sms_flash_promotion_session | | sms_home_advertise | | sms_home_brand | | sms_home_new_product | | sms_home_recommend_product | | sms_home_recommend_subject | | ums_admin | | ums_admin_login_log | | ums_admin_permission_relation | | ums_admin_role_relation | | ums_growth_change_history | | ums_integration_change_history | | ums_integration_consume_setting | | ums_member | | ums_member_level | | ums_member_login_log | | ums_member_member_tag_relation | | ums_member_product_category_relation | | ums_member_receive_address | | ums_member_rule_setting | | ums_member_statistics_info | | ums_member_tag | | ums_member_task | | ums_menu | | ums_permission | | ums_resource | | ums_resource_category | | ums_role | | ums_role_menu_relation | | ums_role_permission_relation | | ums_role_resource_relation | +-----------------------------------------+ 76 rows in set (0.01 sec) mysql> mysql> exit Bye root@k8s-cilium-master-01:~/mall-microservice/document/sql#
3.2、部署商城
创建名称空间，用以部署各服务。
kubectl create namespace mall
运行如下命令，部署各服务。
kubectl apply -f ./ -n mall
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# pwd /root/learning-k8s/Mall-MicroService/mall-and-skywalking root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# ll -h total 40K drwxr-xr-x 2 root root 4.0K May 1 07:56 ./ drwxr-xr-x 5 root root 4.0K May 1 07:56 ../ -rw-r--r-- 1 root root 1.4K May 1 07:56 mall-admin-web.yaml -rw-r--r-- 1 root root 2.0K May 1 07:56 mall-admin.yaml -rw-r--r-- 1 root root 2.0K May 1 07:56 mall-auth.yaml -rw-r--r-- 1 root root 2.7K May 1 07:56 mall-gateway.yaml -rw-r--r-- 1 root root 2.5K May 1 07:56 mall-monitor.yaml -rw-r--r-- 1 root root 2.0K May 1 07:56 mall-portal.yaml -rw-r--r-- 1 root root 2.0K May 1 07:56 mall-search.yaml -rw-r--r-- 1 root root 1.2K May 1 07:56 README.md root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking#
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-admin-web labels: app: mall-admin-web spec: replicas: 1 selector: matchLabels: app: mall-admin-web template: metadata: labels: app: mall-admin-web spec: containers: - name: mall-admin-web # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-admin-web:v1.0 imagePullPolicy: Always ports: - containerPort: 80 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai --- apiVersion: v1 kind: Service metadata: name: mall-admin-web spec: type: NodePort selector: app: mall-admin-web ports: - name: http protocol: TCP port: 80 targetPort: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: mall-admin-web annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: mall-admin-web.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: mall-admin-web port: number: 80
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-admin labels: app: mall-admin spec: replicas: 1 selector: matchLabels: app: mall-admin template: metadata: labels: app: mall-admin spec: containers: - name: mall-admin # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-admin:1.0-SNAPSHOT ports: - containerPort: 8080 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-admin" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-admin spec: type: ClusterIP selector: app: mall-admin ports: - name: http protocol: TCP port: 8080 targetPort: 8080
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-auth labels: app: mall-auth spec: replicas: 1 selector: matchLabels: app: mall-auth template: metadata: labels: app: mall-auth spec: containers: - name: mall-auth # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-auth:1.0-SNAPSHOT ports: - containerPort: 8401 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-auth" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-auth spec: type: ClusterIP selector: app: mall-auth ports: - name: http protocol: TCP port: 8401 targetPort: 8401
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-monitor labels: app: mall-monitor spec: replicas: 1 selector: matchLabels: app: mall-monitor template: metadata: labels: app: mall-monitor spec: containers: - name: mall-monitor # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-monitor:1.0-SNAPSHOT #imagePullPolicy: Always ports: - containerPort: 8101 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-monitor" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-monitor spec: type: ClusterIP selector: app: mall-monitor ports: - name: http protocol: TCP port: 8101 targetPort: 8101 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: mall-monitor annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: mall-monitor.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: mall-monitor port: number: 8101
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-portal labels: app: mall-portal spec: replicas: 1 selector: matchLabels: app: mall-portal template: metadata: labels: app: mall-portal spec: containers: - name: mall-portal # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-portal:1.0-SNAPSHOT ports: - containerPort: 8085 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-portal" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-portal spec: type: ClusterIP selector: app: mall-portal ports: - name: http protocol: TCP port: 8085 targetPort: 8085
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-search labels: app: mall-search spec: replicas: 1 selector: matchLabels: app: mall-search template: metadata: labels: app: mall-search spec: containers: - name: mall-search # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-search:1.0-SNAPSHOT ports: - containerPort: 8081 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-search" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-search spec: type: ClusterIP selector: app: mall-search ports: - name: http protocol: TCP port: 8081 targetPort: 8081
--- apiVersion: apps/v1 kind: Deployment metadata: name: mall-gateway labels: app: mall-gateway spec: replicas: 1 selector: matchLabels: app: mall-gateway template: metadata: labels: app: mall-gateway spec: containers: - name: mall-gateway # 指定Docker Hub中的镜像地址 image: ikubernetes/mall-gateway:1.0-SNAPSHOT ports: - containerPort: 8201 env: # 指定环境 - name: spring.profiles.active value: prod # 指定时区 - name: TZ value: Asia/Shanghai # 指定Nacos地址 - name: spring.cloud.nacos.discovery.server-addr value: http://nacos.nacos.svc:8848 - name: spring.cloud.nacos.config.server-addr value: http://nacos.nacos.svc:8848 - name: JAVA_OPTS value: |- -javaagent:/skywalking/agent/skywalking-agent.jar -Dskywalking.agent.service_name=mall-gateway -Dskywalking.collector.backend_service=skywalking-oap.tracing.svc:11800 - name: JAVA_TOOL_OPTIONS value: "-javaagent:/skywalking/agent/skywalking-agent.jar" - name: SW_AGENT_NAME value: "mall-gateway" - name: SW_AGENT_NAMESPACE value: "mall" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "skywalking-oap.tracing.svc:11800" volumeMounts: - mountPath: /skywalking name: skywalking-agent initContainers: - name: sw-agent #image: registry.magedu.com/apache/skywalking-java-agent:9.0.0-java8 image: apache/skywalking-java-agent:9.0.0-java8 volumeMounts: - name: skywalking-agent mountPath: /data command: [ "/bin/sh" ] args: [ "-c", "cp -R /skywalking/agent /data/" ] volumes: - name: skywalking-agent emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: mall-gateway spec: type: NodePort selector: app: mall-gateway ports: - name: http protocol: TCP port: 8201 targetPort: 8201 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: mall-gateway annotations: ingress.cilium.io/loadbalancer-mode: 'shared' ingress.cilium.io/service-type: 'Loadbalancer' spec: ingressClassName: cilium rules: - host: mall-gateway.k8s.com http: paths: - path: / pathType: Prefix backend: service: name: mall-gateway port: number: 8201
admin/magedu.com
部署详情
root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# kubectl create namespace mall namespace/mall created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# kubectl apply -f ./ -n mall deployment.apps/mall-admin-web created service/mall-admin-web created ingress.networking.k8s.io/mall-admin-web created deployment.apps/mall-admin created service/mall-admin created deployment.apps/mall-auth created service/mall-auth created deployment.apps/mall-gateway created service/mall-gateway created ingress.networking.k8s.io/mall-gateway created deployment.apps/mall-monitor created service/mall-monitor created ingress.networking.k8s.io/mall-monitor created deployment.apps/mall-portal created service/mall-portal created deployment.apps/mall-search created service/mall-search created root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# kubectl get pod -n mall NAME READY STATUS RESTARTS AGE mall-admin-bddf95d49-j2hpc 1/1 Running 0 39m mall-admin-web-5f9ccc58f7-fkz2x 1/1 Running 0 39m mall-auth-f5c98c8-xhpvs 1/1 Running 0 39m mall-gateway-769c755b88-rf5dm 1/1 Running 0 39m mall-monitor-6d888b55ff-tpmdx 1/1 Running 0 39m mall-portal-5d74bb8f9d-h2mmd 1/1 Running 0 39m mall-search-5ccfc85854-fwdl2 1/1 Running 0 39m root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking# kubectl get ingress -n mall NAME CLASS HOSTS ADDRESS PORTS AGE mall-admin-web cilium mall-admin-web.k8s.com 172.16.88.90 80 53s mall-gateway cilium mall-gateway.k8s.com 172.16.88.90 80 51s mall-monitor cilium mall-monitor.k8s.com 172.16.88.90 80 51s root@k8s-cilium-master-01:~/learning-k8s/Mall-MicroService/mall-and-skywalking#
