> Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately!
Because it's probably an impossible to answer question hinging on each person's pet definition of "stable".
In one example of an unstable cluster, pods are "regularly" crashing — but at a really low frequency, like, each pod crashes onces every few days. They're OOM'ing — a bug in the workload, not really the cluster's fault — but the control loop pattern still ensures the failure is corrected in the sense that the pod will either be restarted or evicted and re-created.
(But yes, you absolutely want monitoring to pick up on that. In the specific example here, I'd say it's "just monitor pod failures", and then you get not only OOMs, but all the other ways a workload might crash too.)
I've actually had more problems when the control loops get stuck. (Which they can, almost always because you've asked for the impossible. But sometimes it is hard for the various pieces to say more than "no solution". Why no solution is usually a human-level deduction.)
Because it's probably an impossible to answer question hinging on each person's pet definition of "stable".
In one example of an unstable cluster, pods are "regularly" crashing — but at a really low frequency, like, each pod crashes onces every few days. They're OOM'ing — a bug in the workload, not really the cluster's fault — but the control loop pattern still ensures the failure is corrected in the sense that the pod will either be restarted or evicted and re-created.
(But yes, you absolutely want monitoring to pick up on that. In the specific example here, I'd say it's "just monitor pod failures", and then you get not only OOMs, but all the other ways a workload might crash too.)
I've actually had more problems when the control loops get stuck. (Which they can, almost always because you've asked for the impossible. But sometimes it is hard for the various pieces to say more than "no solution". Why no solution is usually a human-level deduction.)