A 'fork & exec' surprise

See my gist for a self-contained C++ example of the problem code:
https://gist.github.com/ludamad/5579602

So recently during my work at Red Hat I was looking into an issue with Java on Linux (with IcedTea7+ and Oracle's JDK).

The straight-forward seeming code

FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
writer.close();
giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
Runtime.getRuntime().exec(someScriptName);

This code snippet creates a new text file, sets executable permissions, and then tries to execute it.

This code runs just fine when executed in a single thread. Introduce multiple threads (also running this spawning code) and every-so-often you will get an IOException with the message 'error=26 Text file busy'. This IOException means that Linux is refusing to execute the file because it is open for writing.

However, this is clearly the only place we ever open the file - and we promptly close it. So what's going on ?

An aside about executable text files

On Linux (and originally, UNIX), we can execute a text file by looking up the 'shebang' line, eg at the start of the file:

    #!/bin/bash

This tells the kernel which program to use to execute the file. You get 'text file busy' basically if you try to execute a file that is being written to. Normally this would be a text file (you can get this error with binary files as well, though).

Digging deeper

Unfortunately, you can stare at the Java code endlessly; it won't tell you anything. Luckily, we have OpenJDK. A quick foray into UNIXProcess_md.c shows the native implementation of Java's Runtime#exec method. It is implemented using the system calls 'fork' and 'exec'. (Actually, it uses 'vfork', in OpenJDK7, but it's not important.)

Fork, if you're not familiar, essentially creates a new copy of the process, with enough copy-on-write magic to do it efficiently. Worth noting, this child process gets a copy of all the file descriptors opened by the program.

To spawn the program we actually want, we then use 'exec'. This replaces the current process with the specified executable. However, we continue to inherit whatever file descriptors were already open in the new process.

Don't copy my file descriptors, please

Normal programs will take care that the file descriptors copies are closed upon 'exec'. This prevents unwanted information from leaking. Indeed, this is the case in Java-land, where all open file descriptors are marked with FD_CLOEXEC.

This solves a very important problem -- that file descriptors can leak into the child process. However, this does not prevent the copy from occurring in the first place!

Back to the code

Is this knowledge enough to pose an explanation of what's going on ? In fact, it is.

FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
// Say another thread executes 'Runtime.getRuntime().exec(someOtherScriptName);'
// this forked process will have a copy of the file descriptor we are about to close!
writer.close();
giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
// It is very possible that the forked process has not released the copy here!
Runtime.getRuntime().exec(someScriptName);

Unfortunately, the fork from another thread can copy the file-descriptor, suddenly be set aside to let other threads execute, and the file will remain open!

See my gist for a self contained example: https://gist.github.com/ludamad/5579602

Avoiding problems

So what's the takeaway ? If you're going to fork & exec in a multi-threaded program, you cannot do any operations that require file-descriptors to be closed predictably. This is true even if only one of the threads does the process spawning!