内存故障检测edac+MCA
重点都在intel EDS中
x86架构—MCA
https://www.codenong.com/cs106476885/
【x86架构】MCA
https://blog.csdn.net/jiangwei0512/article/details/62456226
其他参考 https://blog.csdn.net/jiangwei0512/category_9286944.html
MCA介绍
https://zhuanlan.zhihu.com/p/685525889
MCA机制：硬件错误检测架构
https://blog.csdn.net/chengm8/article/details/53003134
RAS（二）Intel MCA初探
https://cloud.tencent.com/developer/article/2314691
RAS（四）Intel MCA-Uncorrected Recoverable
https://cloud.tencent.com/developer/article/2314691
Machine-check架构 Intel® 64 and IA-32 Architectures Software Developer Manuals的第15、16章
https://huataihuang.gitbooks.io/cloud-atlas/content/os/linux/kernel/cpu/analysis_cpu_mce.html
RAS---->MCA–>Overview
TIP：
名词解释：
MCA：Machine Check Architecture
MSR：Model Specific Register
MCE：Machine Check Error
#MC：Machine Check Exception
CMCI：Corrected Machine Check Error Interrupt
一 前情提要：
RAS可以粗暴的理解为要实现的目的。
实现这个目的的方法之一就是采用（MCA）机制，而这个机制需要通过一定数量的MSR(寄存器)来实现。这两个寄存器分成两部分，一部分用来进行设置，一部分描述硬件错误。
ERROR：
CPU检测MCE时，会对error进行处理。
Error分两种，可纠正的MCE和不可纠正的MCE。处理方法有所不同。
不可纠正的MCE：触发#MC，通常软件会注册相关的函数来处理#MC，在这个函数中会通过读取MSR来收集MCE的错误信息，然后重启系统。当然由于发生的MCE可能是非常致命的，CPU直接重启了，没有办法完成MCE处理函数；甚至有可能在MCE处理函数中又触发了不可纠正的MCE，也会导致系统直接重启。
可纠正的MCE：当可纠正的MCE数量超过一定的阈值时，会触发CMCI（Corrected Machine Check Error Interrupt），此时软件可以捕捉到该中断并进行相应的处理。CMCI是在MCA之后才加入的，算是对MCA的一个增强，在此之前软件只能通过轮询可纠正MCE相关的MSR才能实现相关的操作。
至此，总算是理清MCA MSR MCE CMCI 之间的关系了
linux下提供的MCA SYS 接口
https://blog.csdn.net/melody9040/article/details/129261637
The new handler can be configured at system run time by reading or writing the control files in /sys/devices/system/machinecheck/machinecheck0/ 10 Val are:
• tolerant Tolerance level. The higher this level the more risk the machine check handler takes to keep the machine running.
Valid levels are:
0 always panic on uncorrected errors.
1 panic if deadlock possible
2 try to avoid panic at slight deadlock risk
3 never panic or exit (for testing only)
Specifying oops=panic on the kernel command line implies zero tolerance.
For a cluster setting tolerant to zero may be best, together with panic=10 to
force an reboot.
• check interval Interval in seconds to check for silent machine check events.
Default 5 minutes. 0 disables background checking.
• bank0ctl … bankNctl Binary mask of errors enabled in bank N. Default is to
enable all errors in each bank. An disabled error will be ignored. For details
on the banks and their sub-errors for AMD and Intel CPUs see [opteron]
https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git
v198
https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git/snapshot/mcelog-198.tar.gz
使用edac工具来检测服务器内存故障.
mc06 表示 表示内存控制器0；
CPU_Src_ID#0 表示源CPU0；
Channel#0 表示通道0；
DIMM#0 标示内存槽0；
Corrected Errors 代表已经纠错的次数；
根据前面列出的CPU通道和内存槽对应关系即可给edac-utils 返回的信息进行编号。
即可得出 A1槽 6312 次纠错，B1槽 6459次纠错，B3槽 535次纠错. 3条内存出现潜在故障，接下来联系供应商进行更换即可。
随着虚拟化，Redis,BDB内存数据库等应用的普及，现在越来越多的服务器配置了大容量内存，拿DELL的R620来说在配置双路CPU下，其24个内存插槽，支持的内存高达960GB。对于ECC,REG这些带有纠错功能的内存故障检测是一件很头疼的事情，出现故障，还是可以连续运行几个月甚至几年，但如果运气不好，随时都会挂掉，好在linux中提供了一个edac-utils 内存纠错诊断工具，可以用来检查服务器内存潜在的故障。
下面以CentOS为例，介绍下edac-utils 工具的使用.
在使用edac-utils 工具之前，需要先了解服务器的硬件架构，以DELL R620为例，(其它如HP DL360P G8，IBM X3650 M4 机型都使用了 E5-2600 系列CPU，C600 系列芯片组.大致相同) 其CPU内存控制器对应通道，内存槽关系，如下所示。
处理器0 (对应一个内存控制器)
通道0：内存插槽A1、A5 和A9
通道1：内存插槽A2、A6 和A10
通道2：内存插槽A3、A7 和A11
通道3：内存插槽A4、A8 和A12
处理器1 (对应一个内存控制器)
通道0：内存插槽B1、B5 和B9
通道1：内存插槽B2、B6 和B10
通道2：内存插槽B3、B7 和B11
通道3：内存插槽B4、B8 和B12
1.安装 edac-utils 工具
yum install -y libsysfs edac-utils
2.执行检测命令，可查看纠错提示如下
edac-util -v
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#0_DIMM#0: A1
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#1_DIMM#0: A2
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#2_DIMM#0: A3
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#3_DIMM#0: A4
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#0_DIMM#1: A5
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#1_DIMM#1: A6
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#2_DIMM#1: A7
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#3_DIMM#1: A8
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#0_DIMM#2: A9
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#1_DIMM#2: A10
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#2_DIMM#2: A11
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#3_DIMM#2: A12
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#0_DIMM#0: B1
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#1_DIMM#0: B2
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#2_DIMM#0: B3
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#3_DIMM#0: B4
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#0_DIMM#1: B5
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#1_DIMM#1: B6
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#2_DIMM#1: B7
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#3_DIMM#1: B8
mc1: csrow2: CPU_SrcID#1_Ha#0_Chan#0_DIMM#1: B9
mc1: csrow2: CPU_SrcID#1_Ha#0_Chan#1_DIMM#1: B10
mc1: csrow2: CPU_SrcID#1_Ha#0_Chan#2_DIMM#1: B11
mc1: csrow2: CPU_SrcID#1_Ha#0_Chan#3_DIMM#1: B12
其中 mc0 表示 表示内存控制器0, CPU_Src_ID#0表示源CPU0 , Channel#0 表示通道0
DIMM#0 标示内存槽0，Corrected Errors 代表已经纠错的次数，根据前面列出的CPU通
道和内存槽对应关系即可给edac-utils 返回的信息进行编号。
即可得出 A1槽 6312 次纠错，B1槽 6459次纠错，B3槽 535次纠错. 3条内存出现潜在故障，接下来联系供应商进行更换即可。
12条内存的对应关系
mc0: csrow0: CPU#0Channel#0_DIMM#0: A1
mc0: csrow0: CPU#0Channel#1_DIMM#0: A2
mc0: csrow0: CPU#0Channel#2_DIMM#0: A3
mc0: csrow1: CPU#0Channel#0_DIMM#1: A4
mc0: csrow1: CPU#0Channel#1_DIMM#1: A5
mc0: csrow1: CPU#0Channel#2_DIMM#1: A6
mc1: csrow0: CPU#1Channel#0_DIMM#0: B1
mc1: csrow0: CPU#1Channel#1_DIMM#0: B2
mc1: csrow0: CPU#1Channel#2_DIMM#0: B3
mc1: csrow1: CPU#1Channel#0_DIMM#1: B4
mc1: csrow1: CPU#1Channel#1_DIMM#1: B5
mc1: csrow1: CPU#1Channel#2_DIMM#1: B6
20条内存的对应关系
mc0: 0 Uncorrected Errors with no DIMM info
mc0: 0 Corrected Errors with no DIMM info
mc0: csrow0: 0 Uncorrected Errors
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#0_DIMM#0: 0 Corrected Errors A1
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#1_DIMM#0: 0 Corrected Errors B1
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#2_DIMM#0: 0 Corrected Errors C1
mc0: csrow0: CPU_SrcID#0_Ha#0_Chan#3_DIMM#0: 0 Corrected Errors D1
mc0: csrow1: 0 Uncorrected Errors
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#0_DIMM#1: 0 Corrected Errors A2
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#1_DIMM#1: 0 Corrected Errors B2
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#2_DIMM#1: 0 Corrected Errors C2
mc0: csrow1: CPU_SrcID#0_Ha#0_Chan#3_DIMM#1: 0 Corrected Errors D2
mc0: csrow2: 0 Uncorrected Errors
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#0_DIMM#2: 0 Corrected Errors A3
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#1_DIMM#2: 11 Corrected Errors B3
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#2_DIMM#2: 0 Corrected Errors C3
mc0: csrow2: CPU_SrcID#0_Ha#0_Chan#3_DIMM#2: 0 Corrected Errors D3
mc1: 0 Uncorrected Errors with no DIMM info
mc1: 0 Corrected Errors with no DIMM info
mc1: csrow0: 0 Uncorrected Errors
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#0_DIMM#0: 0 Corrected Errors
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#1_DIMM#0: 0 Corrected Errors
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#2_DIMM#0: 0 Corrected Errors
mc1: csrow0: CPU_SrcID#1_Ha#0_Chan#3_DIMM#0: 0 Corrected Errors
mc1: csrow1: 0 Uncorrected Errors
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#0_DIMM#1: 0 Corrected Errors
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#1_DIMM#1: 0 Corrected Errors
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#2_DIMM#1: 0 Corrected Errors
mc1: csrow1: CPU_SrcID#1_Ha#0_Chan#3_DIMM#1: 0 Corrected Errors
4x16关系
mc0: csrow0: CPU#0Channel#0_DIMM#0: 0 Corrected Errors 8a
mc0: csrow0: CPU#0Channel#1_DIMM#0: 0 Corrected Errors 5b
mc0: csrow0: CPU#0Channel#2_DIMM#0: 0 Corrected Errors 2c
mc0: csrow1: 0 Uncorrected Errors
mc0: csrow1: CPU#0Channel#0_DIMM#1: 1 Corrected Errors 7d
mc0: csrow1: CPU#0Channel#1_DIMM#1: 0 Corrected Errors 4e
mc0: csrow1: CPU#0Channel#2_DIMM#1: 0 Corrected Errors 1f
mc0: csrow2: 0 Uncorrected Errors
mc0: csrow2: CPU#0Channel#0_DIMM#2: 0 Corrected Errors 6G
mc0: csrow2: CPU#0Channel#1_DIMM#2: 0 Corrected Errors 3h
