I actually had to attach to init(1) with gdb and reap a zombie process.  Supposed to be a "this should never happen" scenario with init, but ...
http://serverfault.com/questions/575162/remove-a-zombie-process-from-the-process-table
It ended up working :)
Interesting. And here I thought a reboot was the only way to reap zombies.
(Trusty serverfault/stackoverflow - Initial question is "Yes I know the usual reasons *lists them off*" and someone responds with one of those reasons. Morons.)
Yeah I never knew there was an alternative to reboot
I think kvr18 just crashed
I'm inclined to agree :(
Description: No report from Monit for 182 seconds: the network, the host or Monit is down. Last report was 19 Jan 2017 15:45:53 -0500
brycec: i see everything up now; are you still having problems?
my vps hasn't phoned-home yet and isn't pinging. 1s while I check console
manual fsck >:(
(thanks for the prodding)
:)
up_the_irons: thoughts/clues on what happened to kvr18?
(just curious)
brycec: no idea; i was doing that zombie reaping.  worked at first, but then didn't subsequent times, and there was a VPS stuck in shutdown.  I've actually never seen it like that before, ever.  Maybe something was up with the host, b/c eventually the whole thing froze.
Huh, weird.
i'm not sure zombie reaping is helpful
mercutio: VM wouldn't start until I reaped the zombie; then it started fine
it used to be when things started zombieing that it was due to memory management etc bugs iirc
I have no idea why all it's resources (tap, disk , etc...) were still held by the zombie
hmm
was that the same vps that got stuck?
yup, got stuck 3 times
weird.  t's still strange.  why that host, why that vm
exactly
it may be some weird race condition that hardly ever happens
that broke something that made it break again
yeah
whats a zombie?
when a process state says Z in top
it's basically a stuck process
It's a child process that is not properly detached, right?
https://en.wikipedia.org/wiki/Zombie_process
Zombie process :: On Unix and Unix-like computer operating systems, a zombie process or defunct process is a process that has completed execution (via the exit system call) but still has an entry in the process table: it is a process in the "Terminated state". This occurs for child processes, where the entry is still needed to allow the parent process to read its child's exit status: once the exit status is read via the wait system call...
Yeah, that.