Kubernetes K8S之Helm部署ELK日志分析系统

 

Kubernetes K8S之Helm部署ELK日志分析系统;由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。

主机配置规划

服务器名称(hostname)系统版本配置内网IP外网IP(模拟)
k8s-master CentOS7.7 2C/4G/20G 172.16.1.110 10.0.0.110
k8s-node01 CentOS7.7 2C/4G/20G 172.16.1.111 10.0.0.111
k8s-node02 CentOS7.7 2C/4G/20G 172.16.1.112 10.0.0.112

备注:由于EFK部署比较消耗内存;所以每台机器的内存最好大于等于4G。

 

ELK概述

ELK是Elasticsearch、Logstash、Kibana的简称,这三者是核心套件,但并非全部。

Elasticsearch是实时全文搜索和分析引擎,提供搜集、分析、存储数据三大功能;是一套开放REST和JAVA API等接口,提供高效搜索功能,可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。

Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和JMX,它能够以多种方式输出数据,包括电子邮件、websockets和Elasticsearch。

Kibana是一个基于Web的图形界面,用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据,不仅允许用户创建他们自己数据定制仪表板的视图,还允许他们以特殊的方式查询和过滤数据。

由于Logstash比较消耗资源,因此本次我们使用Fluentd实现日志收集(EFK)。

 

EFK镜像下载

由于镜像都在国外,因此我们在国内下载镜像,然后tag为对应的镜像名称。执行如下脚本【集群所有机器都执行】:

 1 [root@k8s-node02 software]# vim download_efk_image.sh 
 2 #!/bin/sh
 3 
 4 ##### 在 master 节点和 worker 节点都要执行 【所有机器执行】
 5 
 6 # 加载环境变量
 7 . /etc/profile
 8 . /etc/bashrc
 9 
10 # 变量设置
11 elasticsearch_iamge="elasticsearch-oss:6.7.0"
12 busybox_image="busybox:latest"
13 bats_image="bats:0.4.0"
14 fluentd_image="fluentd-elasticsearch:v2.3.2"
15 kibana_image="kibana-oss:6.7.0"
16 
17 # 集群所有机器执行
18 # elasticsearch镜像下载
19 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
20 docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge}
21 docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
22 # busybox镜像下载
23 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
24 docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image}
25 docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
26 # bats镜像下载
27 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
28 docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image}
29 docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
30 # fluentd-elasticsearch镜像下载
31 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
32 docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image}
33 docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
34 # kibana-oss镜像下载
35 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
36 docker tag  registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image}
37 docker rmi  registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}

 

Elasticsearch部署

本次部署EFK,创建一个efk名称空间。

chart下载与配置修改

 1 # 当前目录
 2 [root@k8s-master efk]# pwd
 3 /root/k8s_practice/efk
 4 # 创建 efk 名称空间
 5 [root@k8s-master efk]# kubectl create namespace efk
 6 [root@k8s-master efk]# 
 7 # ES版本查看,本次我们部署chart 1.30.0版本,ES 6.7.0版本
 8 [root@k8s-master efk]# helm search stable/elasticsearch -l
 9 NAME                             CHART VERSION    APP VERSION    DESCRIPTION                                                 
10 stable/elasticsearch             1.32.5           6.8.6          DEPRECATED Flexible and powerful open source, distributed...
11 stable/elasticsearch             1.32.4           6.8.6          Flexible and powerful open source, distributed real-time ...
12 stable/elasticsearch             1.32.3           6.8.6          Flexible and powerful open source, distributed real-time ...
13 ………………
14 [root@k8s-master efk]# 
15 [root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0
16 [root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz
17 # 修改配置文件1
18 [root@k8s-master efk]# vim elasticsearch/values.yaml
19 initImage:
20   repository: "busybox"
21   tag: "latest"
22   pullPolicy: "IfNotPresent"  # 从Always 改为IfNotPresent
23 ………………
24 client:
25   name: client
26   replicas: 1  # 从2改为1,因为是在自己PC机操作的,内存有限
27   serviceType: ClusterIP
28 ………………
29 master:
30   name: master
31   exposeHttp: false
32   replicas: 3   # 不要修改
33   heapSize: "512m"
34   persistence:
35     enabled: false  # 没有多余的PVC,因此从true改为false
36     accessMode: ReadWriteOnce
37 ………………
38 data:
39   name: data
40   exposeHttp: false
41   replicas: 1  # 从2改为1,因为是在自己PC机操作的,内存有限
42   heapSize: "1024m"  # 从1536m改为1024m,因为是在自己PC机操作的,内存有限
43   persistence:
44     enabled: false  # 没有多余的PVC,因此从true改为false
45     accessMode: ReadWriteOnce
46 [root@k8s-master efk]# 
47 # 修改配置文件2
48 [root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml
49 apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
50 kind: Deployment
51 ………………
52 spec:
53   replicas: {{ .Values.client.replicas }}
54   # 添加信息 如下
55   selector:
56     matchLabels:
57       app: {{ template "elasticsearch.name" . }}
58       component: "{{ .Values.client.name }}"
59       release: {{ .Release.Name }}
60   # 添加信息 如上
61 [root@k8s-master efk]# 
62 # 修改配置文件3
63 [root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml
64 apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
65 kind: StatefulSet
66 ………………
67 spec:
68   serviceName: {{ template "elasticsearch.data.fullname" . }}
69   replicas: {{ .Values.data.replicas }}
70   # 添加信息 如下
71   selector:
72     matchLabels:
73       app: {{ template "elasticsearch.name" . }}
74       component: "{{ .Values.data.name }}"
75       release: {{ .Release.Name }}
76       role: data
77   # 添加信息 如上
78 [root@k8s-master efk]# 
79 # 修改配置文件4
80 [root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml 
81 apiVersion: apps/v1  # 从 apps/v1beta1 改为 apps/v1
82 kind: StatefulSet
83 ………………
84 spec:
85   serviceName: {{ template "elasticsearch.master.fullname" . }}
86   replicas: {{ .Values.master.replicas }}
87   # 添加信息 如下
88   selector:
89     matchLabels:
90       app: {{ template "elasticsearch.name" . }}
91       component: "{{ .Values.master.name }}"
92       release: {{ .Release.Name }}
93       role: master
94   # 添加信息 如上
95 [root@k8s-master efk]#

 

Elasticsearch部署

步骤如下:

 1 # 当前目录
 2 [root@k8s-master efk]# pwd
 3 /root/k8s_practice/efk
 4 # 部署ES
 5 [root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/
 6 [root@k8s-master efk]# 
 7 # 状态查看
 8 [root@k8s-master ~]# helm list
 9 NAME              REVISION    UPDATED                     STATUS      CHART                    APP VERSION    NAMESPACE  
10 es01              1           Sat Jul 25 12:18:50 2020    DEPLOYED    elasticsearch-1.30.0     6.7.0          efk 
11 [root@k8s-master efk]# 
12 # 等待一会儿后【估计几分钟】,查看pod状态信息如下
13 [root@k8s-master ~]# kubectl get deploy -n efk
14 NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
15 es01-elasticsearch-client   1/1     1            1           6m13s
16 [root@k8s-master ~]# 
17 [root@k8s-master ~]# kubectl get pod -n efk
18 NAME                                         READY   STATUS    RESTARTS   AGE
19 es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          6m21s
20 es01-elasticsearch-data-0                    1/1     Running   0          6m21s
21 es01-elasticsearch-master-0                  1/1     Running   0          6m21s
22 es01-elasticsearch-master-1                  1/1     Running   0          5m30s
23 es01-elasticsearch-master-2                  1/1     Running   0          5m3s
24 [root@k8s-master efk]# 
25 [root@k8s-master efk]# kubectl get svc -n efk
26 NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
27 es01-elasticsearch-client      ClusterIP   10.100.237.152  <none>        9200/TCP   6m34s
28 es01-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP   6m42s
29 [root@k8s-master efk]# 
30 [root@k8s-master efk]# kubectl get sts -n efk
31 NAME                        READY   AGE
32 es01-elasticsearch-data     1/1     7m4s
33 es01-elasticsearch-master   3/3     7m4s
34 [root@k8s-master efk]#

 

Elasticsearch访问

其中IP来源于ES的svc。

 

 1 [root@k8s-master ~]# curl 10.100.237.152:9200/
 2 {
 3   "name" : "es01-elasticsearch-client-646f8f866d-rt2wp",
 4   "cluster_name" : "elasticsearch",
 5   "cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw",
 6   "version" : {
 7     "number" : "6.7.0",
 8     "build_flavor" : "oss",
 9     "build_type" : "docker",
10     "build_hash" : "8453f77",
11     "build_date" : "2019-03-21T15:32:29.844721Z",
12     "build_snapshot" : false,
13     "lucene_version" : "7.7.0",
14     "minimum_wire_compatibility_version" : "5.6.0",
15     "minimum_index_compatibility_version" : "5.0.0"
16   },
17   "tagline" : "You Know, for Search"
18 }
19 [root@k8s-master ~]# 
20 [root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty
21 {
22   "cluster_name" : "elasticsearch",
23   "status" : "green",   # 可见状态正常
24   "timed_out" : false,
25   "number_of_nodes" : 5,
26   "number_of_data_nodes" : 1,
27   "active_primary_shards" : 0,
28   "active_shards" : 0,
29   "relocating_shards" : 0,
30   "initializing_shards" : 0,
31   "unassigned_shards" : 0,
32   "delayed_unassigned_shards" : 0,
33   "number_of_pending_tasks" : 0,
34   "number_of_in_flight_fetch" : 0,
35   "task_max_waiting_in_queue_millis" : 0,
36   "active_shards_percent_as_number" : 100.0
37 }

 

至此,elasticsearch部署完毕

 

elasticsearch-client域名获取

根据es01-elasticsearch-client的svc信息获取到es01-elasticsearch-client的域名;用于后面的fluentd 和kibana。

启动一个pod

 

 1 [root@k8s-master test]# pwd
 2 /root/k8s_practice/test
 3 [root@k8s-master test]# cat myapp_demo.yaml
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: myapp-demo
 8   namespace: default
 9   labels:
10     k8s-app: myapp
11 spec:
12   containers:
13   - name: myapp
14     image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1
15     imagePullPolicy: IfNotPresent
16     ports:
17     - name: httpd
18       containerPort: 80
19       protocol: TCP
20 [root@k8s-master test]# 
21 [root@k8s-master test]# kubectl apply -f myapp_demo.yaml
22 pod/myapp-demo created
23 [root@k8s-master test]# 
24 [root@k8s-master test]# kubectl get pod -o wide
25 NAME         READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
26 myapp-demo   1/1     Running   0          6s    10.244.2.84   k8s-node02   <none>           <none>

 

进入pod并得到elasticsearch-client域名信息

 1 # 进入一个pod容器
 2 [root@k8s-master test]# kubectl exec -it myapp-demo sh 
 3 ##### 格式 nslookup svc-cluster-ip
 4 / # nslookup 10.100.237.152
 5 nslookup: can't resolve '(null)': Name does not resolve
 6 
 7 Name:      10.100.237.152
 8 Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local
 9 / # 
10 / # 
11 ### 通过ping判断域名是否通畅
12 / # ping es01-elasticsearch-client.efk.svc.cluster.local
13 PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes
14 64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms
15 64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms
16 64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms

由上可得,Service中es01-elasticsearch-client的域名为:es01-elasticsearch-client.efk.svc.cluster.local

Service的域名格式为:$(service name).$(namespace).svc.cluster.local,其中 cluster.local 指定的集群的域名

 

Fluentd部署

chart下载与配置修改

 1 [root@k8s-master efk]# pwd
 2 /root/k8s_practice/efk
 3 [root@k8s-master efk]#
 4 # fluentd版本信息查看
 5 [root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l
 6 NAME                            CHART VERSION    APP VERSION    DESCRIPTION                                                 
 7 stable/fluentd-elasticsearch    2.0.7            2.3.2          DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
 8 stable/fluentd-elasticsearch    2.0.6            2.3.2          A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
 9 stable/fluentd-elasticsearch    2.0.5            2.3.2          A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
10 ………………
11 # 获取fluentd-elasticsearch 并解压
12 [root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7
13 [root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz
14 # 配置修改
15 [root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml
16 ### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的
17 elasticsearch:
18   host: 'es01-elasticsearch-client.efk.svc.cluster.local'   # 修改处,域名获取参见上文
19   port: 9200
20   scheme: 'http'
21 [root@k8s-master efk]#

 

fluentd-elasticsearch部署

步骤如下:

 1 ################ 部署fluentd-elasticsearch
 2 # 当前目录
 3 [root@k8s-master efk]# pwd
 4 /root/k8s_practice/efk
 5 # 部署fluentd-elasticsearch
 6 [root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch
 7 [root@k8s-master efk]# 
 8 # 状态查看
 9 [root@k8s-master efk]#  helm list
10 NAME              REVISION    UPDATED                     STATUS      CHART                          APP VERSION    NAMESPACE  
11 es01              1           Sat Jul 25 12:18:50 2020    DEPLOYED    elasticsearch-1.30.0           6.7.0          efk        
12 fluentd-es01      1           Sat Jul 25 12:36:01 2020    DEPLOYED    fluentd-elasticsearch-2.0.7    2.3.2          efk
13 [root@k8s-master efk]# 
14 #查看pod状态信息如下
15 [root@k8s-master efk]# kubectl get ds -n efk
16 NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
17 fluentd-es01-fluentd-elasticsearch   2         2         2       2            2           <none>          113s
18 [root@k8s-master efk]# 
19 [root@k8s-master efk]# kubectl get pod -n efk -o wide
20 NAME                                         READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
21 es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          17m   10.244.2.57    k8s-node02   <none>           <none>
22 es01-elasticsearch-data-0                    1/1     Running   0          17m   10.244.2.58    k8s-node02   <none>           <none>
23 es01-elasticsearch-master-0                  1/1     Running   0          17m   10.244.4.241   k8s-node01   <none>           <none>
24 es01-elasticsearch-master-1                  1/1     Running   0          17m   10.244.2.59    k8s-node02   <none>           <none>
25 es01-elasticsearch-master-2                  1/1     Running   0          16m   10.244.4.242   k8s-node01   <none>           <none>
26 fluentd-es01-fluentd-elasticsearch-qnmf9     1/1     Running   0          43s   10.244.4.243   k8s-node01   <none>           <none>
27 fluentd-es01-fluentd-elasticsearch-xmw5f     1/1     Running   0          43s   10.244.2.60    k8s-node02   <none>           <none>

 

Kibana部署

kibana的主版本和大版本必须和elasticsearch(ES)一致,小版本可以不同;但两者版本最好一致,这样可以避免一些因版本不同导致的意外情况。

由于elasticsearch(ES)使用的是6.7.0,因此kibana我们也是用该版本。

chart下载与配置修改

 1 [root@k8s-master efk]# pwd
 2 /root/k8s_practice/efk
 3 # 所有版本查看
 4 [root@k8s-master efk]# helm search stable/kibana -l
 5 NAME             CHART VERSION    APP VERSION    DESCRIPTION                                                 
 6 stable/kibana    3.2.7            6.7.0          Kibana is an open source data visualization plugin for El...
 7 stable/kibana    3.2.6            6.7.0          Kibana is an open source data visualization plugin for El...
 8 ………………
 9 # 获取kibana,并解压缩
10 [root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7 
11 [root@k8s-master efk]# tar xf kibana-3.2.7.tgz
12 # 配置修改1
13 [root@k8s-master efk]# vim kibana/values.yaml
14 ### 为什么使用域名而不是IP,因此每次重启ES的svc,对应IP都会改变。而域名是不变的
15 files:
16   kibana.yml:
17     ## Default Kibana configuration from kibana-docker.
18     server.name: kibana
19     server.host: "0"
20     elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200   # 修改处,域名获取参见上文
21 ………………
22 service:
23   type: NodePort  # 修改内容  从ClusterIP改为NodePort
24   externalPort: 443
25   internalPort: 5601
26   nodePort: 30601  # 添加处,Service端口范围:30000-32767
27 [root@k8s-master efk]# 
28 # 配置修改2
29 [root@k8s-master efk]# vim kibana/templates/deployment.yaml 
30 apiVersion: apps/v1   # 从 apps/v1beta1 改为 apps/v1
31 kind: Deployment
32 metadata:
33 ………………
34 spec:
35   replicas: {{ .Values.replicaCount }}
36   revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
37   # 添加信息 如下
38   selector:
39     matchLabels:
40       app: {{ template "kibana.name" . }}
41       release: "{{ .Release.Name }}"
42   # 添加信息 如上

 

kibana部署

步骤如下:

 1 ################ 部署kibana-oss
 2 # 当前目录
 3 [root@k8s-master efk]# pwd
 4 /root/k8s_practice/efk
 5 # 部署kibana-oss
 6 [root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana
 7 [root@k8s-master efk]#
 8 # 状态查看
 9 [root@k8s-master efk]# helm list
10 NAME              REVISION    UPDATED                     STATUS      CHART                          APP VERSION    NAMESPACE  
11 es01              1           Sat Jul 25 12:18:50 2020    DEPLOYED    elasticsearch-1.30.0           6.7.0          efk        
12 fluentd-es01      1           Sat Jul 25 12:36:01 2020    DEPLOYED    fluentd-elasticsearch-2.0.7    2.3.2          efk        
13 kibana01          1           Sat Jul 25 12:38:18 2020    DEPLOYED    kibana-3.2.7                   6.7.0          efk
14 [root@k8s-master efk]# 
15 #查看pod状态信息如下
16 [root@k8s-master efk]# kubectl get deploy -n efk
17 NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
18 es01-elasticsearch-client   1/1     1            1           19m
19 kibana01                    1/1     1            1           27s
20 [root@k8s-master efk]# 
21 [root@k8s-master efk]# kubectl get pod -n efk -o wide
22 NAME                                         READY   STATUS    RESTARTS   AGE     IP             NODE         NOMINATED NODE   READINESS GATES
23 es01-elasticsearch-client-646f8f866d-rt2wp   1/1     Running   0          20m     10.244.2.57    k8s-node02   <none>           <none>
24 es01-elasticsearch-data-0                    1/1     Running   0          20m     10.244.2.58    k8s-node02   <none>           <none>
25 es01-elasticsearch-master-0                  1/1     Running   0          20m     10.244.4.241   k8s-node01   <none>           <none>
26 es01-elasticsearch-master-1                  1/1     Running   0          19m     10.244.2.59    k8s-node02   <none>           <none>
27 es01-elasticsearch-master-2                  1/1     Running   0          19m     10.244.4.242   k8s-node01   <none>           <none>
28 fluentd-es01-fluentd-elasticsearch-qnmf9     1/1     Running   0          3m10s   10.244.4.243   k8s-node01   <none>           <none>
29 fluentd-es01-fluentd-elasticsearch-xmw5f     1/1     Running   0          3m10s   10.244.2.60    k8s-node02   <none>           <none>
30 kibana01-bc479f8c7-kr2n2                     1/1     Running   0          53s     10.244.4.244   k8s-node01   <none>           <none>
31 [root@k8s-master efk]# 
32 # 查看svc信息
33 [root@k8s-master efk]# kubectl get svc -n efk -o wide
34 NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE   SELECTOR
35 es01-elasticsearch-client      ClusterIP   10.100.237.152  <none>        9200/TCP        20m   app=elasticsearch,component=client,release=es01
36 es01-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP        20m   app=elasticsearch,component=master,release=es01
37 kibana01                       NodePort    10.101.200.177  <none>        443:30601/TCP   71s   app=kibana,release=kibana01

 

浏览器访问

http://172.16.1.110:30601/

 

页面访问

 

创建索引

 

 

数据查看

 

相关阅读

1、Kubernetes K8S之Helm部署、使用与示例

完毕!

 

 


———END———
如果觉得不错就关注下呗 (-^O^-) !

 

posted on 2021-01-24 19:53  踏歌行666  阅读(2859)  评论(0编辑  收藏  举报

导航