etcd compacts every 5 minutes, so you can easily go over storage limit sizes. you can use kubectl for this if apiserver is up, but if it's not this is where auger shines!
Nov 19, 2019, 4:02:18 PM [original]-
Beta@beta@gayhorse.clubPublic
probably don't store images in etcd, btw. it's not meant for that
Nov 19, 2019, 4:02:19 PM [original] -
Beta@beta@gayhorse.clubPublic
last example: slow API server! let's take a look at request latency. it turned out these metrics were useless, because the bucket sizes were too coarse.
Nov 19, 2019, 4:02:19 PM [original] -
Beta@beta@gayhorse.clubPublic
bad metrics make it difficult for cluster operators. but updating is hard! Changing metrics could cause false positives or negatives in people's monitoring
Nov 19, 2019, 4:06:29 PM [original] -
Beta@beta@gayhorse.clubPublic
bad metrics can't be disabled, so you need a full upgrade. sig-instrumentation did an overhaul of broken metrics in 1.14.
Nov 19, 2019, 4:06:29 PM [original]
- labels didn't match
- wrong data types
- units weren't standardized, and many were even outright incorrect! -
Beta@beta@gayhorse.clubPublic
how do we deprecate bad metrics? sig-instrumentation is going to treat metrics as a proper api. we'll have stability levels for all metrics going forward, so we can mark metrics as deprecated
Nov 19, 2019, 4:06:30 PM [original] -
Beta@beta@gayhorse.clubPublic
what metrics are stable? we need criteria and promotion still. and we need runtime flags for disabling individual metrics
Nov 19, 2019, 4:06:31 PM [original] -
Beta@beta@gayhorse.clubPublic
distributed tracing and structured logs are (hopefully) coming soon too!
Nov 19, 2019, 4:06:31 PM [original] -
Beta@beta@gayhorse.clubPublic
(that's all for today. Thanks @ehashdn and @LogicalHan!) #KubeCon
Nov 19, 2019, 4:14:23 PM [original]