7,022 questions
Best practices
0
votes
0
replies
16
views
Build histogram from pre-existing data
I have a question regarding histograms, not built from data collected in real time.
Our codebase deals with processes, all of which have a duration. I want to expose these durations as a metric. We ...
0
votes
1
answer
53
views
Get accurate numbers for counter increases
Assume that I have this counter, with the server scraping these metrics every 15 seconds:
vector_component_sent_events_total{component_id=~"sink.*"}
If the counter increases by 30. As shown ...
0
votes
0
answers
24
views
Promtail Configuration replace expressions in the pipeline stages are not working
My Promtail configuration where the replace expressions in the pipeline stages are not working as expected.
I’m trying to replace certain keywords in logs from the testing namespace:
elevenlabs to ...
0
votes
0
answers
23
views
Jmx Prometheus agent - alternating and inconsistent values issue
Setup explanation:
I am using jmx prometheus agent, which starts with the VM as vm argument in HDFS setup.
Version: 0.20.0
There is a jmx http port displaying the Mbeans as JSON.
Problem:
When I view ...
0
votes
1
answer
32
views
Defining thresholds as constants in rules
Apologies as I'm very new to Prometheus
I wish to define a threshold as a constant in a rule as follows
groups:
- name: UI Service
rules:
- alert: HighErrorRate
expr: calculation > {{ ...
1
vote
0
answers
40
views
How to use Prometheus OpenMetrics backfill correctly?
I have a rudimentary Prometheus test environment for backfill testing, and, unfortunately I cannot seem to get it to work; all remarks are appreciated as I am sure I am making some fundamental mistake....
-1
votes
1
answer
60
views
Grafana graph plot data only once instead of again and again metric java
I have implementation on java and Grafana/prometheus
From our java service we are ingesting payload to grafana and it plots
I have defined a Metric using java
public class MyPayloadService {
...
0
votes
0
answers
65
views
Prometheus scraping wrong url
I have a set of micro services with Spring Boot running on containers,
I'm using Eureka discovery and Spring Cloud API Gateway,
I'm trying to set Prometheus to monitor them, so my prometheus.yml looks ...
1
vote
1
answer
73
views
Expose custom health group components into Prometheus metrics in Spring Boot
We have two custom health groups named extended and readiness:
extended: custom dependency-heavy HealthIndicators provided explicitly via application property
readiness: everything except those ...
0
votes
1
answer
96
views
Flux with SOPS doesn't decrypt stringData [closed]
I'm trying to use fluxcd on my kubernetes cluster, which is working perfectly fine.
Now I wanted to add prometheus/grafana stack and use sops to decrypt the basic_auth part in the scrapingConfig from ...
0
votes
0
answers
53
views
Terraform - AWS ECS - Communication between containers in single task definition using awsvpc network (ADOT collector for prometheus )
OK so I have created a Terraform ECS cluster and In a single task definition, i have 3 containers.
Backend spring boot - which exposes actuator endpoints health, info and prometheus on port 8080
...
0
votes
0
answers
28
views
Select first result per instance in Prometheus
I am trying to create a query that will show me the last time Windows servers were patched.
count(((time() - windows_update_patch_state_time) / 86400) > 0 < 31)
The above query would show how ...
0
votes
1
answer
62
views
Decode label value in Grafana panel [closed]
I have a Grafana panel that display server states:
I need to give the servers human recognizable names like: "London", "Paris", "Milan", "Berlin", "...
-2
votes
1
answer
135
views
Reusing a regexp match across 2 metrics/labels in prometheus alerts [closed]
For a Prometheus alert I have the following expression:
node_hwmon_tempcelsius{sensor="nvme_nvme0"} >= node_hwmon_temp_crit_celsius{sensor="nvme_nvme0"}
repeated for nvme 0-5. ...
3
votes
1
answer
120
views
Prometheus/Grafana: increase() spikes during Kubernetes deploys with multi-pod counters (dynamic labels)
Context
Java service in Kubernetes, multiple pods.
Metrics exposed via Micrometer + Prometheus.
Grafana dashboards use increase(...) and sum by (...) to count events in a time range.
Counters are ...
1
vote
1
answer
96
views
Why python prometheus client collectors create metric object every time when collect method is invoked
https://github.com/prometheus/client_python/blob/master/prometheus_client/gc_collector.py
import gc
import platform
from typing import Iterable
from .metrics_core import CounterMetricFamily, Metric
...
0
votes
1
answer
46
views
Counter type/label not getting displayed in prometheus web console [closed]
I use micrometer + spring boot + prometheus to create, register and monitor the counters from the application, apart from counters we have timer and gauge. counter created inside application which has ...
-2
votes
1
answer
426
views
How to send metrics to Prometheus using Spring Boot 3.5, Micrometer, using Pushgateway, i.e. without scraping
I have a Spring Boot application that needs to send metrics to Prometheus, but I can't use the traditional scraping approach because my application runs in an environment where Prometheus can't reach ...
1
vote
1
answer
228
views
NoClassDefFoundError: io/micrometer/prometheus/PrometheusMeterRegistry
I have two dependency in my pom but I keep getting the error for: java.lang.NoClassDefFoundError: io/micrometer/prometheus/PrometheusMeterRegistry
I made sure the dependency for prometheus exists in ...
2
votes
0
answers
70
views
push Histogram metrics through pushgateway, but only found +lnf bucket
What did you do?
I use the following code to generate some metrics data, and then push it to Prometheus through pushgateway. I use Histogram type data and design multiple buckets. However, after ...
2
votes
1
answer
131
views
Cortex/Prometheus custom recording rule not being created
I have custom recording rules set up in our cortex cluster as defined below. The source data for all of these recording rules exists for the last several months. All of the recording rules generate ...
0
votes
1
answer
71
views
How to implement OR logic for Grafana variables?
I have 2 variables in my Grafana: namespace and service.
namespace is defined as: foo: app1|app2|app3
service is defined as: list of app4, app5, app6
I've got a metric named http_request_count with ...
0
votes
0
answers
99
views
Count unique users in the last 30 days - Promtail, Loki, and Grafana
I have a Kubernetes cluster with Promtail, Loki, Grafana, and Prometheus installed. I have an nginx-ingress that generates logs in JSON. Promtail extract the fields, creates a label for http_host, and ...
0
votes
1
answer
52
views
cloudwatch_exporter and CWAgent metrics
My cloudwatch_exporter happily exports metrics in AWS/* namespaces, but seems to ignore metrics in the CWAgent namespace. From looking through bug reports, it seems like this is a known but ignored ...
1
vote
0
answers
54
views
Most appropriate PromQL function and corresponding panel type on grafana for methods with @Timed
I am using @Timed from micrometer-core on a few methods to capture the response time of methods in a very simple way like this - @Timed(percentiles = {0.50, 0.95, 0.99, 0.999})
It is a spring-boot ...
0
votes
1
answer
57
views
ActiveMQ Artemis metrics plugin: no aggregate totalMessageCount if metrics disabled for some addresses
I enabled the Prometheus metrics plugin for an ActiveMQ Artemis broker instance.
Metrics are disabled for some addresses like so:
<address-settings>
<address-setting match="test.#&...
0
votes
0
answers
117
views
Add custom label based on endpoint with prometheus_fastapi_instrumentator
I am using prometheus_fastapi_instrumentator for my service. Now I am using Instrumentator().instrument(app).expose(app, include_in_schema=False) to get the basic metircs. But I want to add some ...
1
vote
0
answers
43
views
Grafana settings
I am creating a Grafana dashboard to display some prometheus metrics. I was able to create a Variables in Settings, which is a list of endpoint. So in the dashboard, each graph could display specific ...
0
votes
0
answers
49
views
topk 10 function in PromQL query
I was able to use the below query to list the top 10 CPU utilization in grafana, which works fine. However, in an alerting system, there is no alert generated.
topk(10, 100 - (rate(...
0
votes
1
answer
190
views
prometheus and cAdvisor: Error scraping target: label_value_length_limit exceeded
I'm using Prometheus 3.4.1, set up as described here. I run it and cAdvisor this way:
docker run -d --name prometheus -p 9090:9090 --group-add 988 \
-v "/etc/$p_conf":"/etc/prometheus/$...
0
votes
1
answer
45
views
Dealing with a series of prometheus gauges that are actually events
We are using the https://github.com/Labbs/github-actions-exporter at my workplace and I'm having a hard time accomplishing a query that can alert us when a particular workflow in a particular repo has ...
0
votes
1
answer
276
views
How to create Grafana dashboard with variable of type map
I have got grafana dashboard with panel , outputting the following metric:
rate(http_requests_total{service=~"${$service}"}[5m])
where $service is a variable. One can select service from ...
0
votes
1
answer
336
views
Grafana Prometheus legend add custom column field from a label
Is it possible to put a value of a label on a Legend?
I used Legend option in my query to format Name to {{__name__}} - {{env}} but I would rather have a separate column for {{env}} instead. Is it ...
2
votes
1
answer
141
views
Displaying multiple OpenShift clusters version within grafana
I want to make a pie chart where I display the current versions of available OpenShift clusters. The clusters uses remote_write to a central Thanos instance, from where we can use promql to query ...
0
votes
0
answers
157
views
Prometheus not scraping metrics from .NET 9 web application
I'm trying to set up Prometheus with my application, following this guide from Microsoft, but data is not arriving in Prometheus.
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/...
0
votes
0
answers
49
views
How to properly graph a counter?
My code is increasing the counter app_test_metric_counter_total every 1 minute by 1.
Attaching a label color with 3 possible values (green, yellow, red)
I have these metrics being exposed:
# HELP ...
0
votes
1
answer
165
views
prometheus.exporter.azure not sending metrics
hi i'm creating an observability stack to collect metrics of azure resources using alloy as agent and prometheus-mimir as datasource to store the metrics.
for authentication i have created a service ...
0
votes
0
answers
45
views
How to scrape metrics from composite data in jmx exporter
Suppose a composite data is like as following
HealthStateJMX = {
Component = ServerRuntime;
HealthState = HEALTH_OK;
IsCritical = true;
MBean = server1;
Partition = null;
ReasonCode = [ ...
0
votes
0
answers
48
views
How to Accurately Calculate Per-Service Trace Durations and P95 Using PromQL or TraceQL
I'm using Tempo's metrics generator to extract spanmetrics and calculate the duration of each trace.
However, when I use the following PromQL expression, the results differ significantly from the ...
0
votes
1
answer
299
views
Prometheus Alert Manager Integration with OAuth2 Credentials Not Working
Snippet of my alertmanager.yml file:
global:
smtp_smarthost: "smtp.office365.com:587"
smtp_from: "[email protected]"
http_config:
oauth2:
client_id: "...
0
votes
0
answers
81
views
Alertmanager alerts not being sent to both receivers
I have a range of alerts marked with a 'critical' label such as
- alert: query
expr: sgn(sum(rate(query{kubernetes_namespace=~"ns"}[1m])) by (kubernetes_namespace)) == 0
for: 1m
labels:...
0
votes
0
answers
102
views
Configuring Prometheus Alert manager notifications based on labels?
I want to Configure alerts in Prometheus Alertmanager based on the group label.
I have 12 servers with node-exporter installed. Each server has different project and I want to configure alerts for it ...
0
votes
0
answers
36
views
Instrumenting spring boot scheduler
I've created a Spring boot using scheduled. I configured to prometheus collect metrics of this api, but when I try to access the /actuator url, I got error like:
Error scraping target: Get
"http:...
1
vote
0
answers
34
views
Node Memory SReclaimable Bytes Equivalent on FreeBSD
I am on a FreeBSD machine, and am trying to get some basic metrics from prometheus about my memory usage. I'm new to this however and cannot find great references online. I was the equivalent metric ...
0
votes
1
answer
91
views
How to add different bearer tokens to Prometheus jobs?
I want to add multiple targets to a job, but they use different tokens.
I don't want to configure multiple jobs.
I try to do the following configuration but it doesn't work.
- job_name: minio-job-...
0
votes
1
answer
201
views
micrometer-registry-prometheus not showing exemplars
I would like to send metrics with exemplars using Spring Boot and Micrometer.
Part of the pom.xml:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId&...
0
votes
0
answers
75
views
Adding a new label in a prometheus metric
Currently running Prometheus 3.3.0
So i have the following metrics
jvm_memory_used_bytes{area="heap", environment="prod", id="PS Survivor Space", instance="my....
0
votes
0
answers
79
views
Customizing Grafana Histogram graph Axes for Latency Metrics
I'm using multiple Prometheus histogram metrics following the pattern *_duration_seconds_bucket that represent latency breakdown across my pipeline. These metrics track iterations typically taking ~...
0
votes
0
answers
210
views
Loki v3 Migration: Memberlist TCP Connection Issues
I'm migrating from Loki v2 (distributed) to Loki v3 in a Kubernetes environment. After deployment, I'm seeing two major issues:
. Massive amounts of gRPC client cancellation errors:
level=warn ts=2025-...
1
vote
2
answers
483
views
How to set alertmanager in prometheus-community/prometheus helm chart?
I'm using this chart:
https://artifacthub.io/packages/helm/prometheus-community/prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install my-...