I'm trying to answer a question of how many pods were started/scheduled/whatever per namespace per day. I have not found any useful counter-type metric that would be counting that, just related gauges like kube_pod_status_scheduled_time.
So far best thing I was able to come up with is this:
topk(3,
sum_over_time((
delta((
count by (namespace) (
count by (pod, namespace) (kube_pod_status_scheduled_time{namespace=~".*-tenant"})
)
)[5m:]) > 0
)[1d:])
)
But first this seems to be too heavy/inefficient to finish before request timeout and second I'm very skeptical it is actually correct.
Would you have any ideas please?