Not sure if this has changed, but there have been companies admitting to simply nuking Kubernetes clusters if they fail, because it does happens. The argument, which I completely believe, is that it's faster to build a brand new cluster than debugging a failed one.
I had this happen on a small scale and it scared me a lot. It felt like your executable suddenly falling apart and you now need to fix it in assembly. My takeaway was that the k8s abstraction is way leakier than it is made out to be
