微服务部署实践
构建微服务体系
- 网络规划
- 基础服务
- 监控服务
- 数据库配比
- 日志收集
- 分布式文件存储
组成
- docker
- docker-compose
- docker swarm(docker集群管理服务)
- portainer.io(docker集群可视化客户端)
- docker registry
- eureka
- zuul
- auth
- spring-cloud
- elasticsearch
- logstash
- kibana
- fluentd
- zookeeper
- kafka
- skywalking
- apollo
- apps
docker
环境:
CentOS 7.5x
安装方式:rpm
$ sudo curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
$ sudo yum makecache
$ sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
$ sudo yum -y install /path/to/package.rpm
$ sudo systemctl start docker
$ sudo systemctl status docker
$ docker --version
$ docker info
...
$ sudo yum remove docker-ce
$ sudo rm -rf /var/lib/docker
docker-compose
环境:
CentOS 7.5x
安装方式:pip
$ sudo yum -y install epel-release
$ sudo yum -y install python-pip
$ sudo pip install --upgrade pip
$ sudo pip --default-timeout=200 install -U docker-compose
$ docker-compose -version
docker registry
$ sudo docker run -d -v /data/registry:/var/lib/registry -p 5000:5000 --restart=always --privileged=true --name registry registry:latest
ps: 注意添加可信息列表
microservice三组件(eureka&zuul&auth)
microservice-stack.yml
version: "3"
services:
auth:
image: 172.16.3.193:5000/microservice/auth:8696
environment:
- ENV=FAT
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 2s
restart_policy:
condition: on-failure
networks:
- microservice-network
ports:
- 7250:7250
logging:
driver: fluentd
options:
tag: app.log
fluentd-address: 172.16.3.185:24224
fluentd-async-connect: "true"
tag: app.log
eureka-peer1:
image: 172.16.3.193:5000/microservice/eureka:8661
environment:
- spring.profiles.active=peer1
- ENV=FAT
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 2s
restart_policy:
condition: on-failure
networks:
- microservice-network
ports:
- 7000:7000
logging:
driver: fluentd
options:
tag: app.log
fluentd-address: 172.16.3.185:24224
fluentd-async-connect: "true"
tag: app.log
eureka-peer2:
image: 172.16.3.193:5000/microservice/eureka:8661
environment:
- spring.profiles.active=peer2
- ENV=FAT
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 2s
restart_policy:
condition: on-failure
networks:
- microservice-network
ports:
- 7010:7010
logging:
driver: fluentd
options:
tag: app.log
fluentd-address: 172.16.3.185:24224
fluentd-async-connect: "true"
tag: app.log
eureka-peer3:
image: 172.16.3.193:5000/microservice/eureka:8661
environment:
- spring.profiles.active=peer3
- ENV=FAT
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 2s
restart_policy:
condition: on-failure
networks:
- microservice-network
ports:
- 7020:7020
logging:
driver: fluentd
options:
tag: app.log
fluentd-address: 172.16.3.185:24224
fluentd-async-connect: "true"
tag: app.log
zuul:
image: 172.16.3.193:5000/microservice/zuul:8699
environment:
- ENV=FAT
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 2s
restart_policy:
condition: on-failure
# zuul要与应用网络互通
networks:
- microservice-network
- app-network
ports:
- 7300:7300
logging:
driver: fluentd
options:
tag: app.log
fluentd-address: 172.16.3.185:24224
fluentd-async-connect: "true"
tag: app.log
networks:
microservice-network:
external: true
app-network:
external: true
elasticsearch
基本介绍
Elasticsearch是一个分布式数据库,一个实例称为node,至少两个节点组成cluster。
Index ≈ 日常所说的某个数据库,名称必须小写
Document ≈ 元组(某条数据),不强制要求同构,但是同构能提升搜索效率
Type ≈ Table(Type概念将会在7.x之后干掉)
Shard ≈ 相当于分库分表(这个是es自动实现的)
Replia = 副本环境:
说明:由于skywalking0.3版本只支持5.x,因此选择安装5.6.10
配置:三台服务器(172.16.2.137, 172.16.2.138 172.16.2.139)
建议使用rpm安装简单快速
使用chrome插件查看Elasticsearch预览图
es和mysql概念比较
elasticsearch.yml
// 137 master
cluster.name: caad-es-dev
node.name: es-node1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: zoo1
http.port: 5020
discovery.zen.ping.unicast.hosts: ["zoo1", "zoo2", "zoo3"]
discovery.zen.minimum_master_nodes: 2
// 138
cluster.name: caad-es-dev
node.name: es-node2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: zoo2
http.port: 5020
discovery.zen.ping.unicast.hosts: ["zoo1", "zoo2", "zoo3"]
discovery.zen.minimum_master_nodes: 2
// 139
cluster.name: caad-es-dev
node.name: es-node3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: zoo3
http.port: 5020
discovery.zen.ping.unicast.hosts: ["zoo1", "zoo2", "zoo3"]
discovery.zen.minimum_master_nodes: 2
// test http://172.16.2.137:5020,http://172.16.2.138:5020,http://172.16.2.139:5020
// chrome extension: Elasticsearch Head
// elasticsearch-plugin install head
// 安装中文分词器 ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.10/elasticsearch-analysis-ik-5.6.10.zip
logstash.conf
input {
file {
path => [
"C:/Workspace/logdemo/logdemo/App_Data/Logs/*.log"
]
# type => "error"
start_position => "beginning"
codec => multiline {
pattern => "^\["
negate => true
# next
what => "previous"
}
}
}
filter {
# 定义数据的格式
grok {
match => {
"message" => "\[(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] \[(?<thread>\d+)\] \[(?<level>\S+\s?)\] (?<message>(.|\n)*)"}
}
# 可能是这个版本有问题或者是插件没有安装,导致解析时间有问题
# date {
# match => {
# "time" => "yyyy-MM-dd HH:mm:ss,SSS"
# }
# locale => "en"
# target => "@timestamp"
# }
}
output {
elasticsearch {
hosts => [ "172.16.2.137:5020", "172.16.2.137:5020", "172.16.2.137:5020" ]
index => "filebeat-%{+yyyy.MM.dd}"
template_overwrite => true
}
# 调试用
# stdout { codec => rubydebug }
}
简单搜索
// 查看所有索引
$ curl -X GET 'http://172.16.2.137:5020/_cat/indices?v',每个节点查询出来的数据应该是一致的
// 创建索引名称为weather的索引
$ curl -X PUT 'http://172.16.2.137:5020/weather'
// 删除索引,支持通配符http://172.16.2.137:5020/*
$ curl -XDELETE "http://172.16.2.137:5020/weather"
// 用中文分词器查询 ik_max_word
$ curl -X PUT 'http://172.16.2.137:5020/accounts' -d '
{
"mappings": {
"person": {
"properties": {
"user": {
"type": "text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_max_word"
},
"title": {
"type": "text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_max_word"
},
"desc": {
"type": "text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_max_word"
}
}
}
}
}'
// 新增一条记录
// 注意,如果没有先创建 Index(这个例子是accounts),直接执行上面的命令,Elastic 也不会报错,而是直接生成指定的 Index。所以,打字的时候要小心,不要写错 Index 的名称
$ curl -X PUT 'http://172.16.2.137:5020/accounts/person/1' -d '
{
"user": "张三",
"title": "工程师",
"desc": "数据库管理"
}'
// 查看记录
$ curl 'http://172.16.2.137:5020/accounts/person/1?pretty=true'
// 删除记录
$ curl -X DELETE 'http://172.16.2.137:5020/accounts/person/1'
// 更新
$ curl -X PUT 'localhost:9200/accounts/person/1' -d '
{
"user" : "张三",
"title" : "工程师",
"desc" : "数据库管理,软件开发"
}'
// 全表查询
$ curl 'localhost:9200/accounts/person/_search'
{
"took":2,
"timed_out":false,
"_shards":{"total":5,"successful":5,"failed":0},
"hits":{
"total":2,
"max_score":1.0,
"hits":[
{
"_index":"accounts",
"_type":"person",
"_id":"AV3qGfrC6jMbsbXb6k1p",
"_score":1.0,
"_source": {
"user": "李四",
"title": "工程师",
"desc": "系统管理"
}
},
{
"_index":"accounts",
"_type":"person",
"_id":"1",
"_score":1.0,
"_source": {
"user" : "张三",
"title" : "工程师",
"desc" : "数据库管理,软件开发"
}
}
]
}
}
// match查询
$ curl 'http://172.16.2.137:5020/accounts/person/_search' -d '
{
// 软件 管理,es理解是or
"query" : { "match" : { "desc" : "软件 管理" }},
"from": 1,(偏移量)
"size": 1(一页多少条,默认是全部)
}'
// and查询
$ curl 'http://172.16.2.137:5020/accounts/person/_search' -d '
{
"query": {
"bool": {
"must": [
{ "match": { "desc": "软件" } },
{ "match": { "desc": "系统" } }
]
}
}
}'
kafka
调试
bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka
$KAFKA_HOME/bin/kafka-topics.sh --create --topic topic \
--partitions 4 --zookeeper 172.25.0.3 --replication-factor 2
$KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper 172.25.0.3
$KAFKA_HOME/bin/kafka-console-producer.sh --topic=topic \
--broker-list=`broker-list.sh`
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic test --from-beginning
skywalking
环境:
配置:172.16.3.141
默认端口:10800 11800 12800
5.0.x架构图
3.2.5+架构(使用这个版本)
部署之后的效果图
// download
$ curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
$ su -c "yum install java-1.8.0-openjdk"
$ systemctl status firewalld
$ systemctl stop firewalld
$ systemctl start firewalld
$ systemctl start elasticsearch
$ sudo yum install -y net-tools
# useradd name
# tar -zxf apache-skywalking-apm-incubating-5.0.0-RC2.tar.gz
# rpm -ih elasticsearch-5.6.10.rpm
# vi config/application.yml
# vi webapp/webapp.yml
# ./bin/startup.sh
// test 172.16.2.141:8080 admin admin
# 删除索引数据,相当于删库
$ curl -XDELETE "http://172.16.2.137:5020/*"
application.yml
naming:
jetty:
#OS real network IP(binding required), for agent to find collector cluster
host: 172.16.2.141
port: 10800
contextPath: /
cache:
caffeine:
remote:
gRPC:
# OS real network IP(binding required), for collector nodes communicate with each other in cluster. collectorN --(gRPC) --> collectorM
host: 172.16.2.141
port: 11800
agent_gRPC:
gRPC:
#OS real network IP(binding required), for agent to uplink data(trace/metrics) to collector. agent--(gRPC)--> collector
host: 172.16.2.141
port: 11800
# Set these two setting to open ssl
#sslCertChainFile: $path
#sslPrivateKeyFile: $path
# Set your own token to active auth
#authentication: xxxxxx
agent_jetty:
jetty:
# OS real network IP(binding required), for agent to uplink data(trace/metrics) to collector through HTTP. agent--(HTTP)--> collector
# SkyWalking native Java/.Net/node.js agents don't use this.
# Open this for other implementor.
host: localhost
port: 12800
contextPath: /
analysis_register:
default:
analysis_jvm:
default:
analysis_segment_parser:
default:
bufferFilePath: ../buffer/
bufferOffsetMaxFileSize: 10M
bufferSegmentMaxFileSize: 500M
bufferFileCleanWhenRestart: true
ui:
jetty:
# Stay in `localhost` if UI starts up in default mode.
# Change it to OS real network IP(binding required), if deploy collector in different machine.
host: localhost
port: 12800
contextPath: /
storage:
elasticsearch:
clusterName: caad-es-dev
clusterTransportSniffer: true
clusterNodes: 172.16.2.137:9300
indexShardsNumber: 2
indexReplicasNumber: 0
highPerformanceMode: true
# Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
bulkActions: 2000 # Execute the bulk every 2000 requests
bulkSize: 20 # flush the bulk every 20mb
flushInterval: 10 # flush the bulk every 10 seconds whatever the number of requests
concurrentRequests: 2 # the number of concurrent requests
# Set a timeout on metric data. After the timeout has expired, the metric data will automatically be deleted.
traceDataTTL: 90 # Unit is minute
minuteMetricDataTTL: 90 # Unit is minute
hourMetricDataTTL: 36 # Unit is hour
dayMetricDataTTL: 45 # Unit is day
monthMetricDataTTL: 18 # Unit is month
configuration:
default:
applicationApdexThreshold: 2000
serviceErrorRateThreshold: 10.00
serviceAverageResponseTimeThreshold: 2000
instanceErrorRateThreshold: 10.00
instanceAverageResponseTimeThreshold: 2000
applicationErrorRateThreshold: 10.00
applicationAverageResponseTimeThreshold: 2000
# thermodynamic
thermodynamicResponseTimeStep: 50
thermodynamicCountOfResponseTimeSteps: 40
# max collection's size of worker cache collection, setting it smaller when collector OutOfMemory crashed.
workerCacheMaxSize: 10000
webapp.yml
server:
port: 8080
collector:
path: /graphql
ribbon:
ReadTimeout: 10000
listOfServers: 172.16.2.141:10800
security:
user:
admin:
password: admin
filebeat
环境:winodws
配置:windows服务
filebeat.prospectors:
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- C:\PublishWeb\Caad.Client.Acl\Log\*.error*
fields:
level: error
appName: Caad.Client.Acl
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- C:\PublishWeb\Caad.Client.Data\Log\*.error*
fields:
level: error
appName: Caad.Client.Data
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- C:\PublishWeb\Caad.Client.Job\Log\*.error*
fields:
level: error
appName: Caad.Client.Job
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- C:\PublishWeb\Caad.Client.Setting\Log\*.error*
fields:
level: error
appName: Caad.Client.Setting
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\PublishWeb\Caad.Service.WebApi\Log\*.error*
fields:
level: error
appName: Caad.Service.WebApi
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
- input_type: log
enabled: true
encoding: utf-8
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\PublishWeb\Viss.Server.Host\Log\*.error*
fields:
level: error
appName: Viss.Server.Host
multiline:
pattern: '^\[\d{4}\-\d{2}\-\d{2}\s\d+\:\d+\:\d+\,\d+\]\s'
negate: true
match: after
max_lines: 500
timeout: 5s
output.elasticsearch:
hosts: ["172.16.2.137:5020","172.16.2.138:5020","172.16.2.139:5020"]
index: "filebeat-%{+yyyy.MM.dd}"
fluentd
环境:
配置:docker中运行
fluentd作用
fluent.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter **>
@type concat
key log
multiline_start_regexp /^\[\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2},\d{3}\]/
flush_interval 5s
</filter>
<match **>
@type copy
<store>
@type elasticsearch
hosts 172.16.2.137:5020,172.16.2.138:5020,172.16.2.139:5020
index_name app-log
type_name app-log
include_timestamp true
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name log
tag_key @log_name
flush_interval 5s
</store>
<store>
@type stdout
</store>
</match>
日志规范
log4net.config
<?xml version="1.0" encoding="utf-8" ?>
<!-- 官方配置参考 http://logging.apache.org/log4net/release/config-examples.html -->
<log4net>
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="[%d] [%t] [%p] %m%n" />
</layout>
<filter type="log4net.Filter.LevelRangeFilter">
<param name="LevelMin" value="DEBUG" />
<param name="LevelMax" value="FATAL" />
</filter>
</appender>
<root>
<appender-ref ref="ConsoleAppender" />
</root>
</log4net>
log4j2.properties
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}%d{yyyyMMdd}.log
appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}%d{yyyyMMdd}.log.%i
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d] [%t] [%p] %m%n
kibana
kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://172.16.2.137:5020"
微服务体系有哪些?
资料
- 大型网站进化
- SOA 百度百科
- SOA
- Docker
- Docker Document
- Docker Hub
- 阿里镜像
- docker-compose 安装
- elk stack
- elasticsearch
- Elasticsearch 中文教程
- Elasticsearch Document
- elasticsearch 查询语法
- elasticsearch ThreadPool配置
- elasticsearch 中文社区
- Courier Fetch: 1 of 610 shards failed
- Courier Fetch: 1 of 610 shards failed
- filebeat
- filebeat 配置说明
- filebeat 运行机制
- ELK常见问题
- fluentd
- skywalking
- 分布式链路调用框架比较
- es模型字段描述
- skywalking中文文档
- apollo中文文档
- zookeeper
- zookeeper 入门简介
- zookeeper Document
- open jdk
- kafka
- kafka 入门简介
- kafka Document
- 微服务配置文件思考
- 微服务整体思考
- ik分词器
- logstash date插件
- logstash grok插件使用