Prometheus dropping collectd stats

I am trying to get collectd and Prometheus + grafana. However, I am seeing all stats are being dropped in Prometheus. Initially. doubted the time..but now I have made both servers to UTC and the time match.

Still dropping stats

Apr 29 05:36:03 centos77 prometheus[20377]: ts=2022-04-29T05:36:03.043Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:36:18 centos77 prometheus[20377]: ts=2022-04-29T05:36:18.042Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:36:33 centos77 prometheus[20377]: ts=2022-04-29T05:36:33.042Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:36:48 centos77 prometheus[20377]: ts=2022-04-29T05:36:48.042Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:37:03 centos77 prometheus[20377]: ts=2022-04-29T05:37:03.042Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:37:18 centos77 prometheus[20377]: ts=2022-04-29T05:37:18.042Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:37:33 centos77 prometheus[20377]: ts=2022-04-29T05:37:33.041Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529
Apr 29 05:37:48 centos77 prometheus[20377]: ts=2022-04-29T05:37:48.043Z caller=scrape.go:1580 level=warn component="scrape manager" scrape_pool=collectd target=http://10.239.41.113:9103/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=529

Any idea on this?

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum