Self Host 使用 Exceptionless 实时监控程序运行日志服务
Exceptionless 是一个可以对 ASP.NET Core, ASP.NET MVC，WebAPI, WebForms, WPF, Console 应用提供系统的日志，错误监控、报表等服务实时日志收集框架 。具体介绍：http://www.cnblogs.com/savorboard/p/exceptionless.html 。
日志分析与程序异常在系统开发与运行中占非常重要的地位，在技术选型的过程中需要考虑一些问题
- 数据分散在多台服务器中，难以查找
- 数据量大，查询速度慢
- 涉及多个系统调用，难以快速定位数据问题
开源日志项目
https://github.com/Graylog2/graylog2-server
https://github.com/exceptionless/Exceptionless
https://github.com/elastic/elasticsearch
https://github.com/getsentry/sentry
https://github.com/dianping/cat
配置 Elasticsearch 集群
最终我们还是选用个了 Exceptionless ，于是就自己搭建了一套本地环境（Elasticsearch 配置官方推荐：three node with two master nodes on Linux）。
- .NET 4.6.1
- IIS 7.5+
- ElasticSearch 5.1. We also have a guide located here.
#配置JAVA运行环境
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install software-properties-common htop
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version
# centos
yum install java(openjdk)
tar -zxvf jdk-8u131-linux-x64.tar.gz
#配置环境变量
vi /etc/profile JAVA_HOME=/opt/jdk1.8.0_131 CLASSPATH=.:$JAVA_HOME/lib PATH=$JAVA_HOME/bin:$PATH export JAVA_HOME CLASSPATH PATH
#生效环境变量
source /etc/profile
#安装elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
tar -xzf elasticsearch-5.5.0.tar.gz
#安装插件
sudo bin/elasticsearch-plugin install mapper-size
#配置elasticsearch-head
Running with docker
https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
for Elasticsearch 5.x open http://localhost:9100/:
docker run -p 9100:9100 mobz/elasticsearch-head:5
#Enable CORS in elasticsearch
http.cors.enabled: true
http.cors.allow-origin: "*"
#创建elastic用户
useradd elastic (userdel elastic)
#修改所有者
chown -R elastic /opt/elasticsearch-5.5.0
#切换到elastic用户启动
su elastic
#以后台方式启动
./bin/elasticsearch –d
Elasticsearch 配置
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # multi node configuration cluster.name: exceptionless action.destructive_requires_name: true action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* #bootstrap.memory_lock: true # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: p2_es_node_03 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: 3 # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true # 启用跨域 #Enable CORS in elasticsearch http.cors.enabled: true http.cors.allow-origin: "*" # 集群配置 # 这个配置限制了单机上可以开启的ES存储实例的个数,当我们需要单机多实例，则需要把这个配置赋值2，或者更高。 node.max_local_storage_nodes: 1 #设置这个集群中节点的数量，默认为2，一旦这N个节点启动，就会立即进行数据恢复。 gateway.expected_nodes: 3 #设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4） discovery.zen.minimum_master_nodes: 2 #设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点。 discovery.zen.ping.unicast.hosts: ["10.255.131.162","10.255.131.163","10.255.131.164"] #network.bind_host: [ '_site_', '_local_' ] #network.publish_host: '_site_'
cluster.name: exceptionless node.name: exceptionless_node_01 action.destructive_requires_name: true action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* #bootstrap.memory_lock: true # Set the bind address to a specific IP (IPv4 or IPv6): network.host: 0.0.0.0 http.port: 9200 #Enable CORS in elasticsearch http.cors.enabled: true http.cors.allow-origin: "*" # --------------------------------- Discovery ---------------------------------- # 这个配置限制了单机上可以开启的ES存储实例的个数,当我们需要单机多实例，则需要把这个配置赋值2，或者更高。 node.max_local_storage_nodes: 1 #设置这个集群中节点的数量，默认为2，一旦这N个节点启动，就会立即进行数据恢复。 gateway.expected_nodes: 3 #设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4）(total number of master-eligible nodes / 2 + 1): discovery.zen.minimum_master_nodes: 2 #设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点。 discovery.zen.ping.unicast.hosts: ["10.255.131.162","10.255.131.163","10.255.131.164"]
JVM 配置
## JVM configuration ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## You should always set the min and max JVM heap ## size to the same value. For example, to set ## the heap to 4 GB, set: ## ## -Xms4g ## -Xmx4g ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information ## ################################################################ # Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms4g -Xmx4g ################################################################ ## Expert settings ################################################################ ## ## All settings below this section are considered ## expert settings. Don't tamper with them unless ## you understand what you are doing ## ################################################################ ## GC configuration -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly ## optimizations # disable calls to System#gc -XX:+DisableExplicitGC # pre-touch memory pages used by the JVM during initialization -XX:+AlwaysPreTouch ## basic # force the server VM (remove on 32-bit client JVMs) -server # explicitly set the stack size (reduce to 320k on 32-bit client JVMs) -Xss1m # set to headless, just in case -Djava.awt.headless=true # ensure UTF-8 encoding by default (e.g. filenames) -Dfile.encoding=UTF-8 # use our provided JNA always versus the system one -Djna.nosys=true # use old-style file permissions on JDK9 -Djdk.io.permissionsUseCanonicalPath=true # flags to configure Netty -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 # log4j 2 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true ## heap dumps # generate a heap dump when an allocation from the Java heap fails # heap dumps are created in the working directory of the JVM -XX:+HeapDumpOnOutOfMemoryError # specify an alternative path for heap dumps # ensure the directory exists and has sufficient space #-XX:HeapDumpPath=${heap.dump.path} ## GC logging #-XX:+PrintGCDetails #-XX:+PrintGCTimeStamps #-XX:+PrintGCDateStamps #-XX:+PrintClassHistogram #-XX:+PrintTenuringDistribution #-XX:+PrintGCApplicationStoppedTime # log GC status to a file with time stamps # ensure the directory exists #-Xloggc:${loggc} # By default, the GC log file will not rotate. # By uncommenting the lines below, the GC log file # will be rotated every 128MB at most 32 times. #-XX:+UseGCLogFileRotation #-XX:NumberOfGCLogFiles=32 #-XX:GCLogFileSize=128M # Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON. # If documents were already indexed with unquoted fields in a previous version # of Elasticsearch, some operations may throw errors. # # WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided # only for migration purposes. #-Delasticsearch.json.allow_unquoted_field_names=true
运行日志
LOG: elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ ls bin config lib LICENSE.txt modules NOTICE.txt plugins README.textile elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ ./bin/elasticsearch [2017-07-12T14:57:36,651][INFO ][o.e.n.Node ] [] initializing ... [2017-07-12T14:57:36,721][INFO ][o.e.e.NodeEnvironment ] [9rZOxk3] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [230.6gb], net total_space [245.9gb], spins? [possibly], types [ext4] [2017-07-12T14:57:36,722][INFO ][o.e.e.NodeEnvironment ] [9rZOxk3] heap size [1.9gb], compressed ordinary object pointers [true] [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] node name [9rZOxk3] derived from node ID [9rZOxk3hS0Wmj6SPvU0ZSg]; set [node.name] to override [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] version[5.5.0], pid[27006], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.4.0-63-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11] [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/elasticsearch-5.5.0] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [aggs-matrix-stats] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [ingest-common] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-expression] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-groovy] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-mustache] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-painless] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [parent-join] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [percolator] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [reindex] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [transport-netty3] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [transport-netty4] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded plugin [mapper-size] [2017-07-12T14:57:38,768][INFO ][o.e.d.DiscoveryModule ] [9rZOxk3] using discovery type [zen] [2017-07-12T14:57:39,323][INFO ][o.e.n.Node ] initialized [2017-07-12T14:57:39,324][INFO ][o.e.n.Node ] [9rZOxk3] starting ... [2017-07-12T14:57:39,463][INFO ][o.e.t.TransportService ] [9rZOxk3] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2017-07-12T14:57:39,475][WARN ][o.e.b.BootstrapChecks ] [9rZOxk3] max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536] [2017-07-12T14:57:39,476][WARN ][o.e.b.BootstrapChecks ] [9rZOxk3] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2017-07-12T14:57:42,521][INFO ][o.e.c.s.ClusterService ] [9rZOxk3] new_master {9rZOxk3}{9rZOxk3hS0Wmj6SPvU0ZSg}{Khs76coWR6qWwC9DjuhyeQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2017-07-12T14:57:42,543][INFO ][o.e.h.n.Netty4HttpServerTransport] [9rZOxk3] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2017-07-12T14:57:42,543][INFO ][o.e.n.Node ] [9rZOxk3] started [2017-07-12T14:57:42,555][INFO ][o.e.g.GatewayService ] [9rZOxk3] recovered [0] indices into cluster_state
问题一：警告提示
[2016-11-06T16:27:21,712][WARN ][o.e.b.JNANatives ] unable to install syscall filter:
Java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:349) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.0.jar:5.0.0]
报了一大串错误，其实只是一个警告。
解决：使用新的linux版本，就不会出现此类问题了。
问题二：ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]
vi /etc/security/limits.conf
添加如下内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
问题三：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
vi /etc/security/limits.d/90-nproc.conf
#修改为
* soft nproc 2048
问题四：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
解决：切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p
然后，重新启动elasticsearch，即可启动成功。
#安装kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
tar -xzf kibana-5.4.0-linux-x86_64.tar.gz
cd kibana/
#查看运行进程
ps -ef |grep java
#kill 进程
kill -9 pid
配置 Exceptionless
- 配置项目文件权限为users
- 修改 app.config.*.js 的 BASE_URL 为本机访问IP
- 修改 Web.config 配置
<configuration> <connectionStrings> <add name="RedisConnectionString" connectionString="" /> <add name="ElasticSearchConnectionString" connectionString="http://10.255.131.162:9200" /> <add name="LdapConnectionString" connectionString="" /> </connectionStrings> <appSettings> <!-- Base url for the ui used to build links in emails and other places. --> <add key="BaseURL" value="http://10.255.130.67/#" /> <!-- Controls whether SSL is required. Only enable this if you have SSL configured. --> <add key="EnableSSL" value="false" /> <!-- Dev: Use this mode when debugging. (Outbound emails will not be sent) QA: Use this mode when deployed to staging. (Outbound emails restricted) Production: Use this mode when deployed to production. --> <add key="WebsiteMode" value="Production" /> <!-- Controls whether users can signup. --> <add key="EnableAccountCreation" value="false" /> <!-- Controls whether daily summary emails are sent --> <add key="EnableDailySummary" value="true" /> <!-- Email Client Settings (Uncomment the section below to change the default email settings) There are three valid SmtpEncryption settings: None, SSL and StartTLS --> <!-- Email --> <add key="SmtpHost" value="smtp.exmail.qq.com" /> <add key="SmtpPort" value="465" /> <add key="SmtpEncryption" value="SSL" /> <add key="SmtpFrom" value="test@test.com" /> <add key="SmtpUser" value="test@test.com" /> <add key="SmtpPassword" value="!test." /> <!-- Folder used to store event post data --> <add key="StorageFolder" value="|DataDirectory|\storage" /> <!-- Runs the jobs in the current website process --> <add key="RunJobsInProcess" value="true" /> <add key="aspnet:UseTaskFriendlySynchronizationContext" value="true" /> <!--保留日志天数 --> <add key="MaximumRetentionDays" value="90" /> <!--分片数量--> <add key="ElasticSearchNumberOfShards" value="3" /> <!--复制分片数量--> <add key="ElasticSearchNumberOfReplicas" value="1" /> </appSettings>
首次访问注册用户后创建项目即可(EnableAccountCreation 需要配置成 true)
检查服务状态
http://ip/api/v2/status
开放API
http://api.exceptionless.com/docs/index
客户端调用示列
https://github.com/exceptionless/Exceptionless.Net/tree/master/samples
QA:
本地部署使用哪个版本?
使用最新的版本即可
Elasticsearch(v5.5.0)
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
Exceptionless(v4.0.2)
https://github.com/exceptionless/Exceptionless/releases
大量记录日志会影响性能吗？
支持 python 或者其他语言收集日志吗?
默认可以使用相关的扩展来实现写入到ES
https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch
https://github.com/jptoto/log4net.ElasticSearch
也可以使用 Exceptionless 的扩展
https://github.com/serilog/serilog-sinks-exceptionless
https://github.com/exceptionless/Exceptionless.Net/tree/master/src/Platforms
如NLLog:
<nlog> <extensions> <add assembly="Exceptionless.NLog"/> </extensions> <targets async="true"> <target name="exceptionless" apiKey="API_KEY_HERE" xsi:type="Exceptionless"> <field name="host" layout="${machinename}" /> <field name="identity" layout="${identity}" /> <field name="windows-identity" layout="${windows-identity:userName=True:domain=False}" /> <field name="process" layout="${processname}" /> </target> </targets> <rules> <logger name="*" minlevel="Trace" writeTo="exceptionless" /> </rules> </nlog>
Elasticsearch 主分片与副本数量设定？
修改分片数
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.number_of_replicas" : "1",
"index.number_of_shards" : "5"
}'
Exceptionless 配置 QQ 邮箱发送不了邮件？
配置 <add key="SmtpFrom" value="test@test.com" /> 节点
怎么配置保留日志的天数？
配置 MaximumRetentionDays 属性即可(https://github.com/exceptionless/Exceptionless/issues/139)
每秒产生大量的日志请求 ，会有性能问题吗？
可以使用使用内存存储(UseInMemoryStorage)，相对会稳定很多; Exceptionless 前端做负载（需要配置Redis）
ExceptionlessClient.Default.Configuration.UseInMemoryStorage();
怎么标识是哪个用户操作发生的事件？
https://github.com/exceptionless/Exceptionless/wiki/User-Sessions
用户标识访问者归属地的 DB 下载失败？
怎么样快速查询日志？
怎么样监控日志服务是否正常？
Redis服务是否是必须配置的？
开放了哪些 API 接口 ？
http://api.exceptionless.com/docs/index
怎么分配 Elasticsearch 中的 JVM 的内存大小？
Exceptionless 日志没有实时显示？
检查 IIS 是否配置了 websocket 协议 ; 检查 SignalR 是否连接成功
ws://xx.xx.xxx.xxx/api/v2/push/connect?transport=webSockets&clientProtocol=1.5&access_token=hkm9YhSalvakZWkJmI3ek2M7rB8WTeuANmy0kp54&connectionToken=YQdrdiaOnprcRq%2BsaMBKfdpRwLnUtc5NPZ0ZExJVfb3h4j5SrHkUDqQOOq8NAgzGx7YADZ3azpJ3JIFFt217sSlE7pJf%2B7848sxFTfzXg%2FmAHvvcHjipHirL0Xl3FvfqaQaqo50Tyo5m77GvBM1YrQ%3D%3D&tid=8
REFER:
https://github.com/medcl/elasticsearch-analysis-ikhttp://blog.takipi.com/how-to-choose-the-right-log-management-tool/
https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-graylog-lessons-learned/
https://exceptionless.com/sending-log-messages-to-exceptionless/
