Prometheus's UI is good for prototyping and exploration, but you probably want Grafana for more permanent use.

a single symptom isn't enough for a diagnosis. Metrics can show you how something is failing, but not necessarily why.

even if they can't solve the problem, metrics can guide you in the right direction when debugging an issue

metrics are only useful in context. API server uptime is interesting, but is there one API server? many? what's the request pattern like?

the kubernetes control plane is a lot of components that can't directly talk to each other - they have to be mediated by the apiserver

the kubelet is responisble for liveness probes. pods can specify an endpoint that the kubelet can probe to make sure a pod is still up. Pods can also specify metrics endpoints (prometheus!) and emit logs.

the apiserver is a good source of metrics, because unlike most other components, all of the components have to talk to the apiserver. IT exposes /metrics, health endpoints (/healthz, /livez and /readyz as of 1.16)

audit-logs are ostensibly a security feature, but they're also a great source of queryable structured logs from the apiserver

etcd is the database that backs the apiserver. so it's another juice metrics target.

etcd has a /metrics and a /health, but there are also tools like auger and etcdctl

auger is specifically for kubernetes, so unlike etcdctl it can decode kubernetes protobufs in etcd.

etcd isn't a kubernetes component, so things like the health endpoint and logs are in different formats.

time for an example! problem: a node is down. how do we know? you can look for all nodes for whom promothesus scrapes are failing

but things aren't always so neat in production. a grey failure could just be a very slow, as indicated by a long scrape duration

this doesn't need any innate knowledge of the node, perfect for black-box debugging!

another problem: a crash-looping apiserver, detected by either monitoring healthcheck endpoints, or by master kubelet's metrics for liveness or readiness.

is the process crashing immediately, or is it getting killed by the kubelet (e.g. because the liveness check failed) ?

healthz?verbose can help us here, if we can get a curl off before a crashloop. in this example, we can see etcd is down. And because it's down, apiserver fails its own healthcheck and crashloops.

the default storage limit for etcd is 2gb - so etcd_object_count can be a clue to an overwhelmed etcd

etcd objects aren't all the same size. a watch mechanism means etcd has to store historical versions of an object. So a very frequently updated object can take up a disproportionate amount of space on etcd!

etcd compacts every 5 minutes, so you can easily go over storage limit sizes. you can use kubectl for this if apiserver is up, but if it's not this is where auger shines!

probably don't store images in etcd, btw. it's not meant for that

last example: slow API server! let's take a look at request latency. it turned out these metrics were useless, because the bucket sizes were too coarse.

bad metrics make it difficult for cluster operators. but updating is hard! Changing metrics could cause false positives or negatives in people's monitoring

bad metrics can't be disabled, so you need a full upgrade. sig-instrumentation did an overhaul of broken metrics in 1.14.
- labels didn't match
- wrong data types
- units weren't standardized, and many were even outright incorrect!

how do we deprecate bad metrics? sig-instrumentation is going to treat metrics as a proper api. we'll have stability levels for all metrics going forward, so we can mark metrics as deprecated

what metrics are stable? we need criteria and promotion still. and we need runtime flags for disabling individual metrics

distributed tracing and structured logs are (hopefully) coming soon too!

(that's all for today. Thanks @ehashdn and @LogicalHan!) #KubeCon