云容器集群与日志分析平台应用

一、项目完整拓扑

1、完整拓扑

使用 Nginx 搭建 web 集群，动态页面由 PHP 和 Tomcat 集群解析
使用 NFS 存放网页，将访问日志存放在计算节点的 /var/weblog 目录下
弹性部署 PHP、Tomcat 集群服务

使用ELK集群收集k8s容器日志，进行分析

二、编写资源文件

1、创建nginx，php，filebeat容器，使用test主机的nfs做共享存储卷，使用jumpserver管理资产，按p，查看资产，输入id 5，进入资产

[root@jumpserver ~]# ssh k8s@192.168.1.252 -p2222


[k8s@test ~]$ sudo -s
[root@test k8s]# yum -y install nfs-utils
nginx的共享网页目录
[root@test k8s]# mkdir -m 777 /var/nginxphp
tomcat的共享目录
[root@test k8s]# mkdir -m 777 /var/nginxtomcat
[root@test k8s]# vim /etc/exports
/var/nginxphp *(rw)
/var/nginxtomcat *(rw)
[root@test ~]# systemctl enable --now nfs
[root@test ~]# cd /var/nginxphp
[root@test nginxphp]# scp 192.168.1.252:/root/5/public/info.php ./ #拷贝nginx的动态页面
[root@test nginphp]# echo hello world > info.html 创建nginx静态页面
[root@test nginxphp]# vim /var/nginxtomcat/test.jsp 编写tomcat的共享文件

Now time is: <%=new java.util.Date()%>
[root@test nginxphp]# exit exit [k8s@test ~]$ exit logout [Host]> exit

jumpserver主机创建pv和pvc的资源，共享网页资源
[root@jumpserver ~]# mkdir nginx
[root@jumpserver ~]# cd nginx/
[root@jumpserver nginx]# vim pv-nfs.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-nfs
spec:
volumeMode: Filesystem
capacity:
storage: 30Gi
accessModes:

  • ReadWriteMany
  • ReadOnlyMany
  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    nfs:
    path: /var/nginxphp
    server: 192.168.1.101

[root@jumpserver nginx]# vim pvc-nfs.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-nfs
spec:
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 25Gi

nginx加载pvc,使用nfs共享目录
[root@jumpserver nginx]# vim nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: web-site #新添加pvc
persistentVolumeClaim: #新添加
claimName: pvc-nfs #新添加
containers:
- name: nginx
image: harbor:80/library/myos:nginx
ports:
- protocol: TCP
containerPort: 80
volumeMounts:
- name: web-site #新添加，应用pvc
mountPath: /usr/local/nginx/html #新添加，指定目录
restartPolicy: Always

php加载pvc,使用nfs共享目录
[root@jumpserver nginx]# vim php-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: php-fpm
spec:
selector:
matchLabels:
app: php-fpm
replicas: 1
template:
metadata:
labels:
app: php-fpm
spec:
volumes:
- name: web-site #新添加
persistentVolumeClaim: #新添加
claimName: pvc-nfs #新添加
containers:
- name: php-fpm
image: harbor:80/library/myos:phpfpm
ports:
- protocol: TCP
containerPort: 9000
volumeMounts:
- name: web-site #新添加
mountPath: /usr/local/nginx/html #新添加
restartPolicy: Always

让nginx和php连接，编写php-service.yaml文件
[root@jumpserver nginx]# vim php-service.yaml

apiVersion: v1
kind: Service
metadata:
name: phpfpm-service
spec:
ports:

  • protocol: TCP
    port: 9000
    targetPort: 9000
    selector:
    app: php-fpm #需要和php-deployment.yaml里面labels里面定义的一致
    type: ClusterIP

使用configmap映射nginx配置文件，实现动静分离

[root@jumpserver nginx]# yum -y install docker-ce
[root@jumpserver nginx]# vim /etc/hosts
192.168.1.100 harbor

[root@jumpserver nginx]# vim /etc/docker/daemon.json

{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://08fd0a6fce0026040ffdc0158fe37d60.mirror.swr.myhuaweicloud.com"],
"insecure-registries":["harbor:80"]
}
[root@jumpserver nginx]# systemctl restart docker
[root@jumpserver nginx]# docker run -itd --name nginx harbor:80/library/myos:nginx
[root@jumpserver nginx]# docker cp nginx:/usr/local/nginx/conf/nginx.conf ./
[root@jumpserver nginx]# vim nginx.conf
21 log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
22 '$status $body_bytes_sent "$http_referer" '
23 '"$http_user_agent"'; #更改
24
25 access_log logs/access.log main; #更改，去掉#
...
65 location ~ .php$ {
66 root html;
67 fastcgi_pass phpfpm-service:9000; #更改
68 fastcgi_index index.php;
69 include fastcgi.conf;
70 }

使用configmap进行映射nginx配置文件
[root@jumpserver nginx]# kubectl create configmap nginx-conf --from-file=nginx.conf
[root@jumpserver nginx]# kubectl get configmaps

让nginx加载ConfigMap
[root@jumpserver nginx]# vim nginx-deployment.yaml
...
- name: nginx-php #新添加confgiMap
configMap: #新添加
name: nginx-conf #新添加
containers:
- name: nginx
image: harbor:80/library/myos:nginx
ports:
- protocol: TCP
containerPort: 80
volumeMounts:
- name: web-site
mountPath: /usr/local/nginx/html
- name: nginx-php #引用configMap
mountPath: /usr/local/nginx/conf/nginx.conf #新添加
subPath: nginx.conf #新添加
restartPolicy: Always

更改php配置文件，能够监听所有端口，解析php代码
[root@jumpserver nginx]# docker run -itd --name myphp harbor:80/library/myos:phpfpm
[root@jumpserver nginx]# docker cp myphp:/etc/php-fpm.d/www.conf ./
[root@jumpserver nginx]# vim www.conf
12 listen = 0.0.0.0:9000
24 ; listen.allowed_clients = 127.0.0.1

创建configmap，进行映射php-fpm配置文件
[root@jumpserver nginx]# kubectl create configmap php --from-file=www.conf
[root@jumpserver nginx]# vim php-deployment.yaml
...
- name: php-conf #新添加
configMap: #新添加
name: php #新添加
containers:
- name: php-fpm
image: harbor:80/library/myos:phpfpm
ports:
- protocol: TCP
containerPort: 9000
volumeMounts:
- name: web-site
mountPath: /usr/local/nginx/html
- name: php-conf #新添加
mountPath: /etc/php-fpm.d/www.conf #新添加
subPath: www.conf #新添加
restartPolicy: Always

编写filebeat的configMap配置，使其和nginx共享日志
[root@jumpserver nginx]# docker run -itd --name filebeat harbor:80/library/myos:filebeat
[root@jumpserver nginx]# docker cp filebeat:/etc/filebeat/filebeat.yml ./

修改filebeat的配置文件
[root@jumpserver nginx]# vim filebeat.yml
24 enabled: true #打开收集模块
28 - /var/weblog/access.log #指定filebeat读取的日志文件
45 fields:
46 my_type: nginx_log #新添加自定义标签
149 #output.elasticsearch: #加上注释
151 #hosts: ["localhost:9200"] #加上注释
162 output.logstash: #去掉注释
164 hosts: ["192.168.1.75:5044"] #指定logstash主机IP地址
180 #processors: #加上注释
181 #- add_host_metadata: ~ #加上注释
182 #- add_cloud_metadata: ~ #加上注释
[root@jumpserver nginx]# kubectl create configmap filebeat --from-file=filebeat.yml

编写nginx-deployment.yaml资源清单文件，把filebeat和nginx放到同一个pod中，收集nginx日志，到elk可以进行分析
[root@jumpserver nginx]# vim nginx-deployment.yaml
...
- name: filebeat #新添加
configMap: #新添加
name: filebeat #新添加
- name: log-data #新添加
hostPath: #新添加
path: /var/weblog #新添加
type: DirectoryOrCreate #新添加
containers:
- name: nginx
image: harbor:80/library/myos:nginx
ports:
- protocol: TCP
containerPort: 80
volumeMounts:
- name: web-site
mountPath: /usr/local/nginx/html
- name: nginx-php
mountPath: /usr/local/nginx/conf/nginx.conf
subPath: nginx.conf
- name: log-data #新添加
mountPath: /usr/local/nginx/logs #新添加
- name: filebeat #新添加
image: harbor:80/library/myos:filebeat #新添加
volumeMounts: #新添加
- name: filebeat #新添加
mountPath: /etc/filebeat/filebeat.yml #新添加
subPath: filebeat.yml #新添加
- name: log-data #新添加
mountPath: /var/weblog #新添加
restartPolicy: Always

编写nginx的service服务文件
[root@jumpserver nginx]# vim nginx-service.yaml

apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
ports:

  • protocol: TCP
    port: 80
    targetPort: 80
    selector:
    app: nginx #需要和nginx-deployment.yaml里面labels里面定义的一致
    type: ClusterIP

执行上面创建的资源文件
[root@jumpserver nginx]# kubectl apply -f pv-nfs.yaml
[root@jumpserver nginx]# kubectl apply -f pvc-nfs.yaml
[root@jumpserver nginx]# kubectl apply -f nginx-service.yaml
[root@jumpserver nginx]# kubectl apply -f nginx-deployment.yaml
[root@jumpserver nginx]# kubectl apply -f php-service.yaml
[root@jumpserver nginx]# kubectl apply -f php-deployment.yaml

** 三、动态集群HPA搭建**

1、安装metrics-server插件

安装完成之后，可以看到资源
[root@jumpserver nginx]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.168.1.19 68m 3% 994Mi 47%
192.168.1.34 58m 3% 883Mi 42%
192.168.1.65 64m 3% 935Mi 44%

更改php的资源文件，实现HPA弹性集群自动伸缩，在php-deployment.yaml添加资源度量指标
[root@jumpserver nginx]# vim php-deployment.yaml
...
- name: php-conf
mountPath: /etc/php-fpm.d/www.conf
subPath: www.conf
resources: #新添加
requests: #新添加
cpu: 200m #新添加
restartPolicy: Always

[root@jumpserver nginx]# vim hpa.yaml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: nginxphp-backend
spec:
minReplicas: 1
maxReplicas: 3
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-fpm
targetCPUUtilizationPercentage: 50
[root@jumpserver nginx]# kubectl apply -f php-deployment.yaml
[root@jumpserver nginx]# kubectl apply -f hpa.yaml
[root@jumpserver nginx]# kubectl get hpa #刚开始可能是unknown，等一会就会正常
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginxphp-backend Deployment/php-fpm 0%/50% 1 3 3 15s

2、nginx+tomcat配置

nginx+tomcat配置和php差不多，需要注意如何配置代理，网页在前面已经写好

创建tomcat的pv 和 pvc使用的资源
[root@jumpserver nginx]# mkdir /root/tomcat
[root@jumpserver nginx]# cd /root/tomcat
[root@jumpserver tomcat]# vim pv-tomcat.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-tomcat #名字
spec:
volumeMode: Filesystem
capacity:
storage: 5Gi #可以提供的空间
accessModes:

  • ReadWriteMany
  • ReadOnlyMany
  • ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    nfs:
    path: /var/nginxtomcat/ #指定tomcat的共享目录，之前已经创建
    server: 192.168.1.101

[root@jumpserver tomcat]# vim pvc-tomcat.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-tomcat #名称
spec:
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi #需要的空间

编写关于nginx+tomcat转发的配置文件，配置nginx的ConfigMap，更改日志格式 和 nginx转发规则
[root@jumpserver tomcat]# docker run -itd --name mynginx harbor:80/library/myos:nginx
[root@jumpserver tomcat]# docker cp mynginx:/usr/local/nginx/conf/nginx.conf ./
[root@jumpserver tomcat]# vim nginx.conf
21 log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
22 '$status $body_bytes_sent "$http_referer" '
23 '"$http_user_agent"';
24
25 access_log logs/access.log main;
...
35 server {
36 listen 80;
37 server_name nginx-tomcat; #更改此参数为nginx-service的名字
...
43 location / {
44 root html;
45 index index.html index.htm;
46 proxy_pass http://tomcat-service:8080; #新添加参数，转发后端的tomcat-service
47 }
...
制作名为nginx-tomcat的configMap
[root@jumpserver tomcat]# kubectl create configmap nginx-tomcat --from-file=nginx.conf

[root@jumpserver tomcat]# scp /root/nginx/filebeat.yml ./
[root@jumpserver tomcat]# kubectl create configmap tomcat-filebeat --from-file=filebeat.yml

创建nginx的资源文件
[root@jumpserver tomcat]# vim nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-tomcat
spec:
selector:
matchLabels:
app: nginx-tomcat
replicas: 1
template:
metadata:
labels:
app: nginx-tomcat
spec:
volumes:
- name: nginx-tomcat
configMap:
name: nginx-tomcat
- name: tomcat-filebeat
configMap:
name: tomcat-filebeat
- name: log-data
hostPath:
path: /var/weblog
type: DirectoryOrCreate
containers:
- name: nginx-tomcat
image: harbor:80/library/myos:nginx
volumeMounts:
- name: nginx-tomcat
mountPath: /usr/local/nginx/conf/nginx.conf
subPath: nginx.conf
- name: log-data
mountPath: /usr/local/nginx/logs/
ports:
- protocol: TCP
containerPort: 80
- name: nginx-tomcat-filebeat
image: harbor:80/library/myos:filebeat
volumeMounts:
- name: tomcat-filebeat
mountPath: /etc/filebeat/filebeat.yml
subPath: filebeat.yml
- name: log-data
mountPath: /var/weblog
restartPolicy: Always #定义容器的重启方式

创建tomcat的资源文件
[root@jumpserver tomcat]# vim tomcat.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
spec:
selector:
matchLabels:
app: tomcat
replicas: 1
template:
metadata:
labels:
app: tomcat
spec:
volumes:
- name: web-site
persistentVolumeClaim:
claimName: pvc-tomcat
containers:
- name: tomcat
image: harbor:80/library/myos:tomcat
volumeMounts:
- name: web-site
mountPath: /usr/local/apache-tomcat-9.0.6/webapps/ROOT
ports:
- protocol: TCP
containerPort: 8080
resources:
requests:
cpu: 200m
restartPolicy: Always

创建连接tomcat服务的service资源文件
[root@jumpserver tomcat]# vim tomcat-svr.yaml

apiVersion: v1
kind: Service
metadata:
name: tomcat-service
spec:
ports:

  • protocol: TCP
    port: 8080
    targetPort: 8080
    selector:
    app: tomcat #需要和tomcat.yaml里面labels里面定义的一致
    type: ClusterIP

创建连接nginx服务的service文件
[root@jumpserver tomcat]# vim nginx-svr.yaml

apiVersion: v1
kind: Service
metadata:
name: nginx-tomcat
spec:
ports:

  • protocol: TCP
    port: 80
    targetPort: 80
    selector:
    app: nginx-tomcat #需要和nginx.yaml里面labels里面定义的一致
    type: ClusterIP

创建hpa资源
[root@jumpserver tomcat]# vim hpa.yaml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: tomcat-backend
spec:
minReplicas: 1
maxReplicas: 3
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: tomcat #新更改tomcat资源的名字
targetCPUUtilizationPercentage: 50
[root@jumpserver tomcat]# kubectl apply -f pv-tomcat.yaml
[root@jumpserver tomcat]# kubectl apply -f pvc-tomcat.yaml
[root@jumpserver tomcat]# kubectl apply -f tomcat.yaml
[root@jumpserver tomcat]# kubectl apply -f tomcat-svr.yaml
[root@jumpserver tomcat]# kubectl apply -f nginx.yaml
[root@jumpserver tomcat]# kubectl apply -f nginx-svr.yaml
[root@jumpserver tomcat]# kubectl apply -f hpa.yaml

四、ingress发布服务

1、配置ingress服务

打标签并上传ingress相关镜像
[root@jumpserver ingress]# docker login harbor:80 #登录harbor
Username: admin #用户名admin
Password: #密码Harbor12345
[root@jumpserver tomcat]# cd /root/5/kubernetes/plugins/ingress
[root@jumpserver ingress]# docker load -i ingress.tar.xz
[root@jumpserver ingress]# docker tag k8s.gcr.io/ingress-nginx/controller:v1.1.0 harbor:80/library/controller:v1.1.0
[root@jumpserver ingress]# docker push harbor:80/library/controller:v1.1.0

[root@jumpserver ingress]# docker tag k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 harbor:80/library/kube-webhook-certgen:v1.1.1
[root@jumpserver ingress]# docker push harbor:80/library/kube-webhook-certgen:v1.1.1

[root@jumpserver ingress]# vim deploy.yaml #更改文件
328 image: harbor:80/library/controller:v1.1.0
609 image: harbor:80/library/kube-webhook-certgen:v1.1.1
661 image: harbor:80/library/kube-webhook-certgen:v1.1.1
[root@jumpserver ingress]# kubectl apply -f deploy.yaml

[root@jumpserver ingress]# vim example.yaml #设置访问策略

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myweb
  #namespace: ingress-nginx				#注释
  #annotations:							#注释
    #nginx.ingress.kubernetes.io/rewrite-target: /		#注释
    #kubernetes.io/ingress.class: "nginx"				#注释
spec:
  ingressClassName: nginx		#新添加
  rules:
  - host: foo.bar.com			#使用域名访问
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service	#使用域名访问
            port:
              number: 80
  - host: bar.foo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-tomcat
            port:
              number: 80

[root@jumpserver ingress]# kubectl apply -f example.yaml
[root@jumpserver ingress]# kubectl get ingress -o wide
可以看到服务发布到1.65
NAME CLASS HOSTS ADDRESS PORTS AGE
myweb foo.bar.com,bar.foo.com 192.168.1.65 80 2m26s

[root@jumpserver ingress]# vim /etc/hosts
192.168.1.65 foo.bar.com bar.foo.com #ip地址为ingress发布的地址
[root@jumpserver ingress]# curl foo.bar.com/info.php
[root@jumpserver ingress]# curl bar.foo.com/test.jsp

若Linux没有写hosts文件，需要命令行访问可以curl -H "HOST: foo.bar.com" http://192.168.1.65/info.php

使用ELB发布ingress，配置监听器,监听80端口，高级设置里面选中获取客户端ip，配置后端服务

192.168.1.65为ingress发布的机器

点击提交

通过域名访问在win上面更改hosts文件，公网ip + 域名 即可实现使用域名访问网站

测试弹性集群
[root@jumpserver ingress]# ab -c 1000 -n 10000 foo.bar.com/info.php
[root@jumpserver ingress]# ab -c 1000 -n 10000 bar.foo.com/test.jsp

[root@jumpserver ingress]# kubectl get hpa -w #可以看到结果
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginxphp-backend Deployment/php-backend 0%/50% 1 3 1 16m
nginxphp-backend Deployment/php-backend 231%/50% 1 3 1 19m
tomcat-backend Deployment/php-backend 1%/50% 1 3 1 20m

五、收集日志

1、把数据写入到ES集群

通过堡垒机管理
[root@jumpserver ~]# ssh elk@192.168.1.252 -p2222

安装logstash服务1.75
[elk@logstash ~]$ sudo -s
[root@logstash elk]# yum -y install java-1.8.0-openjdk logstash
[root@logstash elk]# ln -s /etc/logstash /usr/share/logstash/config #logstash安装时配置在 /usr/share/logstash/config，但是红帽安装时放到了/etc/logstash，需要需要做个软连接到/usr/share/logstash/config，不然logstash找不到配置文件
配置logstash
[root@logstash elk]# vim /etc/logstash/conf.d/my.conf

 stdin{ codec => "json" }
 file {
  path => ["/tmp/a.log"]
  start_position => "beginning"
  sincedb_path => "/var/lib/logstash/sincedb"
 }

 beats{
  port => 5044
 }
 }

filter{
  if [fields][my_type] == "nginx_log" {
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  }}
}

output{
  stdout{ codec => "rubydebug" }
  if [fields][my_type] == "nginx_log" {
  elasticsearch {
    hosts => ["es-0001:9200", "es-0002:9200"]
    index => "nginx_log-%{+YYYY.MM.dd}"
  }
 }
}
[root@logstash elk]# /usr/share/logstash/bin/logstash

访问从ELB发布的ingress，http://foo.bar.com/info.php或者 http://bar.foo.com/test.jsp，可以看到数据

六、kibana数据展示

1、数据展示

使用kibana 导入索引数据，绘制图形,可以采用通配符

时间字段选择@timestamp

将前面案例收集的数据，进行可视化展示，单击Discover,可以看到数据

以表格的形式进行展示，同时可选择什么时间段内的数据，以及数据刷新时间

创建可视化图表

点击Dashbard，会有很多仪表板。在搜索框里写入关键词host，会出现和搜索关键词相关的仪表板。可以通过它进行数据展示

点击添加

保存仪表板

访问网站页面，可以看到结果输出

posted @   zky-1  阅读(61)  评论(0)  编辑  收藏  举报
相关博文：
·  云容器集群创建，使用、构建日志分析平台
·  服务器硬件、部署LNMP动态网站
·  （ELK应用之Logstash）Logstash的简介、安装和具体的使用流程
·  ELK环境部署-LogStash数据收集(二)
·  ELK日志收集记录
阅读排行：
· winform 绘制太阳，地球，月球 运作规律
· AI与.NET技术实操系列（五）：向量存储与相似性搜索在 .NET 中的实现
· 超详细：普通电脑也行Windows部署deepseek R1训练数据并当服务器共享给他人
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 上周热点回顾（3.3-3.9）
点击右上角即可分享
微信分享提示
AI IDE Trae