minikube consul service mesh with prometheus

I'm using terraform and helm to deploy a kubernetes cluster on minikube. I've deployed consul to act as a service mesh. ATM, when I want to access the services inside minikube's cluster I use kubectl's port-forward command.

Now I want to activate prometheus to run together with consul's services and sidecars. I'm trying to set the values of the consul chart according to what it says in the tutorial but for what some reason when I go into the UI to the topology section, what I see for my services is that its trying to load the metrics but its not working:

enter image description here

The requests that fail on the browser are:

enter image description here

When I try to see what prometheus got from the pods I see:

enter image description here

From searching google for the problem it seems like its an issue with the way the pods export their metrics.

Adding the configuration for the consul chart:

resource "helm_release" "consul" {
  name       = "consul"
  repository = "https://helm.releases.hashicorp.com"
  chart      = "consul"
  version    = "0.39.0"
  namespace  = "consul"

  set {
    name  = "server.replicas"
    value = 1
  }
  set {
    name  = "connectInject.enabled"
    value = true
  }
  set {
    name  = "connectInject.default"
    value = true
  }
  set {
    name  = "connectInject.metrics.defaultEnabled"
    value = true
  }
  set {
    name  = "connectInject.metrics.defaultEnableMerging"
    value = true
  }
  set {
    name  = "connectInject.metrics.defaultPrometheusScrapePort"
    value = 20200
  }
  set {
    name  = "connectInject.metrics.defaultPrometheusScrapePath"
    value = "/metrics"
  }
  set {
    name  = "connectInject.metrics.defaultMergedMetricsPort"
    value = 20100
  }
  set {
    name  = "global.metrics.enableAgentMetrics"
    value = true
  }
  set {
    name  = "global.metrics.agentMetricsRetentionTime"
    value = "1m"
  }
  set {
    name  = "global.metrics.enabled"
    value = true
  }
  set {
    name  = "controller.enabled"
    value = true
  }
  set { # this configuration is for deploying prometheus for non production use cases
    name  = "prometheus.enabled"
    value = true
  }
  set {
    name  = "ui.enabled"
    value = true
  }
  set {
    name  = "ui.metrics.enabled"
    value = true
  }
  set {
    name  = "ui.metrics.provider"
    value = "prometheus"
  }
  set {
    name  = "ui.metrics.baseUrl"
    value = "http://prometheus-server"
  }
}

Does someone have any idea what might be the problem here, how can I make sure the UI has access to those metrics?

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum