prometheus 报错 Error on ingesting samples that are too old or are too far into the future

level=warn ts=2021-08-16T03:20:04.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:04.961Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1507 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=491
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1203 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=959
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=941
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1507 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=558
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1203 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Appending scrape report failed" err="out of bounds"

 

原因：

？

解决：

备份prometheus数据库，新建data目录，重启prometheus和grafana服务，我的Prometheus数据保存在influxdb中，所以数据没丢