prometheus 服务器重启后之"Error on ingesting samples that are too old or are too far into the future"问题解决
背景描述：aws中的prometheus实例状态检查出现异常，重启实例后实例正常，此时发现grafana中主机为空，经查看日志发现如下：
level=warn ts=2021-08-16T03:20:04.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78 level=warn ts=2021-08-16T03:20:04.961Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds" level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1507 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=491 level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1203 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Appending scrape report failed" err="out of bounds" level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=959 level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Appending scrape report failed" err="out of bounds" level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=941 level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Appending scrape report failed" err="out of bounds" level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78 level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds" level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1507 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=558 level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1203 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Appending scrape report failed" err="out of bounds"
解决方法：
备份prometheus数据库，新建data目录，重启prometheus和grafana服务 [root@prometheus /opt/prometheus]# mv data/ data_bak [root@prometheus /opt/prometheus]# mkdir -pv data && chown -R prometheus.prometheus data [root@prometheus /opt/prometheus]# systemctl restart prometheus [root@prometheus /opt/prometheus]# systemctl restart grafana-server
web界面访问prometheus正常，grafana访问正常
