I killed the process and the drain still hung: a grandchild held the pipe

A program of mine hung for forty minutes. Not spinning at a thousand loops a second: at zero percent CPU . It wasn't doing too much work; it wasn't doing any work at all. And still it wouldn't finish. The program does something common: it orchestrates external command-line tools. It launches one, reads what it writes to standard output, and moves on to the next when it's done. So it doesn't get stuck when a tool drags, each one has a timeout: when it fires, the process is killed and we carry on. That's the part that failed, and it failed where no one looks: after killing the process. Killing the process doesn't close the pipe When you read a subprocess's output, you read from a pipe : one end writes (the subprocess), the other reads (you). Your reader doesn't finish when the subprocess dies. It finishes when EOF arrives, and a pipe's EOF arrives only when the last write end is closed. Almost always they coincide: the subprocess is the only writer, it dies, its end closes, EOF arrives, your reader finishes. All in microseconds. But "almost always" isn't "always". The tool I launched launched another one in turn —a grandchild—. And that grandchild inherited the pipe's write end, because on Unix a child inherits its parent's open descriptors unless told otherwise. So when the timeout fired, I killed the child. Its end closed. But the grandchild was still alive , with its copy of the descriptor open. The last write end hadn't closed. EOF never came. And my reader sat waiting for an EOF that would never arrive —at zero percent CPU, blocked in a read() , indistinguishable from slow work—. The symptom that deceives What makes this failure so hard to see is that it doesn't look like a failure . An infinite-loop hang burns CPU: you see it in top instantly. This one spends nothing. The thread is asleep in the kernel waiting for data that isn't coming. In the process list it looks healthy. In the metrics it looks like it's "taking a while". The only way to tell "hung forever"