Prometheus query to alert if something happens x times within a time period?

See also questions close to this topic

• What does duration [] mean when used in Prometheus UI Graph

  In Prometheus when the below query would mean fetching all samples observed during 5m period and then perform the rate calculation with those samples and the the duration.

  rate(prometheus_http_request_duration_seconds_sum{handler="/-/reload", cluster="k8s.util.ue1.plaid.io"}[5m])
  

  When i run this in Table mode in Prometheus UI I get to select a time and then the interval is then determined based my duration and the timestamp i chose. This makes sense.

  But what happens when i switch to Graph tab? It accepts a duration in the UI along with the timestamp. Say I select 1 hour duration with the same query above the resulting graph covers 1 hour. How does the graph work for 1 hour while my query has explicitly selected 5 mins as duration for rate calculation.

  Thanks in advance for your help.
• PromQL in /graph returning no result for any metric

  Whenever I query any basic metric like http_requests_total on prometheus's /graph page, I get zero results in return. I checked the /targets page. All the endpoints are shown as UP there and on clicking the endpoints, the pages do contain metrics.

  Why am I not able to get any result while querying? Help, please.

  Attaching screenshots here.
• jobs prometheus endpoint multi

  I have prometheus with blackbox exporter and I want add in my job different endpoint etc.

  /health
  /q/health/live 
  /health/live
  

  How add multiple endpoint in __tmp_health_url

  prometheus jobs:

      - job_name: health-check
    metrics_path: /probe
    scheme: http
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    kubernetes_sd_configs:
    - role: service
    relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_healthcheck]
          regex: true
          action: keep
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__tmp_health_url]
          replacement: 'health'
          target_label: __tmp_health_url
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_healthcheckurl]
          regex: (.+)
          target_label: __tmp_health_url
        - source_labels: [__address__,__tmp_health_url]
          separator: '/'
          target_label: __param_target
        - target_label: __address__
          replacement: grafana-prometheus-blackbox-exporter:9115
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_service_name]
          target_label: service
  
• Adding two metrics together with a different number of series each

  I have two custom metrics:

  api_success, api_failure

  api_success has a label for "endpoint", and api_failure has a label for "endpoint" and a label for "error_code"

  I'm having difficulty writing a query that will give us the total number of requests made per endpoint, by adding these metrics together.

  The obvious answer is something like this:

  sum by (endpoint) (rate(api_success[5m])) + sum by (endpoint) (rate(api_failure[5m]))
  

  however, this doesn't actually work. In some cases, an endpoint could have no failures, so prometheus is unable to match the vector from the left side to the right side, and the endpoint simply doesn't show up as a series at all. I have read this page: https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching front and back looking for a solution to coerce the right side to have all of the series that the left side does, and use zeroes for all missing series.

  This seems like such a basic use case, I can't believe I haven't found an answer.