At work, we often need to run computation heavy workloads which involve using multi processes to distribute computations on different cores of a single machine (and also a cluster). When the parent process sometimes dies, its children are kept alive running and we want to ensure they also exit as fast as possible to prevent using too much resources.
For example, when using pytorch on multiple cores (with pytorch.distributed.run or pytorch.distributed.launch), children are created by the library and not by our own program. We don’t have the direct knowledge about which processes were started.
Setup
Examples below are in Go, but can be easily ported to Python using the subprocess and threading modules.
Imagine the following program (build with go build ${filename.go}):
package main
import (
"log"
"os"
"os/exec"
)
func main() {
log.Printf("parent: PID: %v\n", os.Getpid())
cmd := exec.Command("child")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}package main
import (
"log"
"os"
"time"
)
func main() {
for {
log.Printf("child: PID: %v alive, parent PID is %v\n", os.Getpid(), os.Getppid())
time.Sleep(2 * time.Second) // Sleep to fake the running process
}
}When running the parent, respective process IDs will be printed:
$ PATH=. ./parent
parent: PID: 20212
child: PID: 20213 alive, parent PID is 20212
child: PID: 20213 alive, parent PID is 20212
If the parent isn’t killed by the shell, its child is then attached to init (1) and continues to run. Using pkill parent effectively kills the parent, but the child is still running and now prints:
child: PID: 20238 alive, parent PID is 1
The child now needs to be killed with pkill child.
Ways around
There seems to be no magic bullet to kill the child in such situation, to free resources as fast as possible. The child process must somehow watch the parent.
Watch stdin
When the parent process dies, its associated file descriptors are closed. If the parent process stdin (or a pipe) is passed to the child stdin, and the child tries to read from it, an error will occur when the parent process dies.
package main
import (
"log"
"os"
"os/exec"
)
func main() {
log.Printf("parent: PID: %v\n", os.Getpid())
cmd := exec.Command("child")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Stdin = os.Stdin
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}package main
import (
"bufio"
"fmt"
"log"
"os"
"time"
)
// watchStdin reads from stdin, and exits if an error occurred, it means the parent process died.
// Depending on what the child is doing, we may also want to cleanup opened resources
func watchStdin() {
in := bufio.NewReader(os.Stdin)
if _, err := in.ReadByte(); err != nil {
log.Println(fmt.Errorf("parent died, exiting: %w", err))
os.Exit(1)
}
}
func main() {
go watchStdin()
for {
log.Printf("child: PID: %v alive, parent PID is %v\n", os.Getpid(), os.Getppid())
time.Sleep(5 * time.Second)
}
}PATH=. ./parent and killing it, the child will have a read error on stdin and exit.
$ PATH=. ./parent
parent: PID: 22083
child: PID: 22084 alive, parent PID is 22083
[1] 22083 terminated PATH=. ./parent
parent died, exiting: read /dev/stdin: input/output error
The child cmd.Stdin can also be relaced by a pipe (cmd.StdinPipe()) instead of the parent’s os.Stdin.
This solution works well if we have control over the parent process or do not need stdin, but does not when we do not have access to the process creation (like with Pytorch).
Watch the parent PID
When the parent process dies, the child is attached to init (1), the child can just watch its parent PID to check if it has changed and is now 1. child.go can be modified as follow:
package main
import (
"log"
"os"
"time"
)
// watchParent checks every `delay` seconds if the parent PID has changed to 1.
// If so, brutally exits the current process as the parent has died.
// Depending on what the child is doing, we may also want to cleanup opened resources
func watchParent(delay int64) {
for {
if os.Getppid() == 1 {
log.Printf("parent died, exiting")
os.Exit(0)
}
time.Sleep(time.Duration(delay) * time.Second)
}
}
func main() {
go watchParent(2)
for {
log.Printf("child: PID: %v alive, parent PID is %v\n", os.Getpid(), os.Getppid())
time.Sleep(2 * time.Second)
}
}When running the parent again PATH=. ./parent, and killing it, the child will detect within 2 seconds if the parents died and exit:
$ PATH=. ./parent
parent: PID: 21742
child: PID: 21743 alive, parent PID is 21742
[1] 21742 terminated PATH=. ./parent
parent died, exiting%
Note: when running in a container, the entry point or command process usually has a PID of 1, so this does not work if the parent is the one started by default.
Using a process group to destroy everything
By mixing one of the above technique with a process group, resources can be freed (violently, as processes are killed). This is how a shell kills all the child processes with a CTRL-C (SIGINT), because they are part of the same process group. In that case it can kill a whole process hierarchy if they belong to the same process group.
In the example below, the parent creates 2 processes, one which watches the parent, and one which does not.
package main
import (
"fmt"
"log"
"os"
"os/exec"
"syscall"
)
func runChild(watch bool) {
var cmd *exec.Cmd
if watch {
cmd = exec.Command("child", "watch")
} else {
cmd = exec.Command("child")
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Stdin = os.Stdin
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
func main() {
if err := syscall.Setpgid(0, 0); err != nil {
log.Fatal(fmt.Errorf("could not setpgid: %w", err))
}
pgid, err := syscall.Getpgid(os.Getpid())
if err != nil {
log.Fatal(fmt.Errorf("could not get pgid: %w", err))
}
log.Printf("parent: PID: %v, pgid: %v\n", os.Getpid(), pgid)
go runChild(true) // watch the parent
runChild(false) // do not watch
}package main
import (
"bufio"
"fmt"
"log"
"os"
"syscall"
"time"
)
func getpgid() int {
pgid, err := syscall.Getpgid(os.Getpid())
if err != nil {
log.Fatal(fmt.Errorf("could not get pgid: %w", err))
}
return pgid
}
// watchStdin reads from stdin, and exits if an error occurred, it means the parent process died.
// Depending on what the child is doing, we may also want to cleanup opened resources
func watchStdin() {
in := bufio.NewReader(os.Stdin)
if _, err := in.ReadByte(); err != nil {
pgid := getpgid()
log.Printf("parent died, killing process group: %v\n", pgid)
syscall.Kill(-pgid, syscall.SIGKILL) // Kill the process group with using a negative number
}
}
func main() {
if len(os.Args) == 2 {
go watchStdin()
}
for {
log.Printf("child: PID: %v alive, parent PID is %v, process group is %v\n", os.Getpid(), os.Getppid(), getpgid())
time.Sleep(5 * time.Second)
}
}When the watching child detects its parent died, it kills the whole process group including the other child that wasn’t watching.
Conclusion
The above tricks are just for child processes to terminate themselves once the parent died, and exit violently. Of course it is also the parent responsibility to correctly cleanup its children if one of them has a failure, but in our scenario, we wanted the parent to fail unexpectedly.