Tuesday, 14 May 2013

A 'fork & exec' surprise

See my gist for a self-contained C++ example of the problem code:

So recently during my work at Red Hat I was looking into an issue with Java on Linux (with IcedTea7+ and Oracle's JDK).

The straight-forward seeming code

FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
This code snippet creates a new text file, sets executable permissions, and then tries to execute it.

This code runs just fine when executed in a single thread. Introduce multiple threads (also running this spawning code) and every-so-often you will get an IOException with the message 'error=26 Text file busy'. This IOException means that Linux is refusing to execute the file because it is open for writing.

However, this is clearly the only place we ever open the file - and we promptly close it. So what's going on ?

An aside about executable text files

On Linux (and originally, UNIX), we can execute a text file by looking up the 'shebang' line, eg at the start of the file:


This tells the kernel which program to use to execute the file. You get 'text file busy' basically if you try to execute a file that is being written to. Normally this would be a text file (you can get this error with binary files as well, though).

Digging deeper

Unfortunately, you can stare at the Java code endlessly; it won't tell you anything. Luckily, we have OpenJDK. A quick foray into UNIXProcess_md.c shows the native implementation of Java's Runtime#exec method. It is implemented using the system calls 'fork' and 'exec'. (Actually, it uses 'vfork', in OpenJDK7, but it's not important.)

Fork, if you're not familiar, essentially creates a new copy of the process, with enough copy-on-write magic to do it efficiently. Worth noting, this child process gets a copy of all the file descriptors opened by the program.

To spawn the program we actually want, we then use 'exec'. This replaces the current process with the specified executable. However, we continue to inherit whatever file descriptors were already open in the new process.

Don't copy my file descriptors, please

Normal programs will take care that the file descriptors copies are closed upon 'exec'. This prevents unwanted information from leaking. Indeed, this is the case in Java-land, where all open file descriptors are marked with FD_CLOEXEC.

This solves a very important problem -- that file descriptors can leak into the child process. However, this does not prevent the copy from occurring in the first place!

Back to the code

Is this knowledge enough to pose an explanation of what's going on ? In fact, it is.
FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
// Say another thread executes 'Runtime.getRuntime().exec(someOtherScriptName);'
// this forked process will have a copy of the file descriptor we are about to close!

giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
// It is very possible that the forked process has not released the copy here!


Unfortunately, the fork from another thread can copy the file-descriptor, suddenly be set aside to let other threads execute, and the file will remain open!

See my gist for a self contained example: https://gist.github.com/ludamad/5579602

Avoiding problems

So what's the takeaway ? If you're going to fork & exec in a multi-threaded program, you cannot do any operations that require file-descriptors to be closed predictably. This is true even if only one of the threads does the process spawning!

icedtea-web 1.4 Released!

icedtea-web 1.4 is here! Lots of fixes and enhancements in this release. This release also fixes known regressions that occurred from 1.2->1.3.

Release announcement by Jiri Vanek:

Hi all!

After long and furious development, I'm finally proud to announce release of icedtea-web 1.4.

97c1d5f02f9a4ab19812a216f39e401a  icedtea-web-1.4.tar.gz

How we were dealing and what we plan to do, can be  checked on wiki:


New in IcedTea-Web 1.4

* Numerous improvements and enhancements in core and system of classloaders
* Added cs localization
* Added de localization
* Added pl localization
* Splash screen for javaws and plugin
* Better error reporting for plugin via Error-splash-screen
* All IcedTea-Web dialogues are centered to middle of active screen
* Download indicator made compact for more then one jar
* User can select its own JVM via itw-settings and deploy.properties.
* Added extended applets security settings and dialogue
* Security updates
  - CVE-2013-1926, RH916774: Class-loader incorrectly shared for applets with same relative-path.
  - CVE-2013-1927, RH884705: fixed gifar vulnerabilit
  - CVE-2012-3422, RH840592: Potential read from an uninitialized memory location
  - CVE-2012-3423, RH841345: Incorrect handling of not 0-terminated strings
* NetX
  - PR1027: DownloadService is not supported by IcedTea-Web
  - PR725: JNLP applications will prompt for creating desktop shortcuts every time they are run
  - PR1292: Javaws does not resolve versioned jar names with periods correctly
* Plugin
  - PR1106: Buffer overflow in plugin table-
  - PR1166: Embedded JNLP File is not supported in applet tag
  - PR1217: Add command line arguments for plugins
  - PR1189: Icedtea-plugin requires code attribute when using jnlp_href
  - PR1198: JSObject is not passed to javascript correctly
  - PR1260: IcedTea-Web should not rely on GTK
  - PR1157: Applets can hang browser after fatal exception
  - PR580: http://www.horaoficial.cl/ loads improperly
* Common
  - PR1049: Extension jnlp's signed jar with the content of only META-INF/* is considered
  - PR955: regression: SweetHome3D fails to run
  - PR1145: IcedTea-Web can cause ClassCircularityError
  - PR1161: X509VariableTrustManager does not work correctly with OpenJDK7
  - PR822: Applets fail to load if jars have different signers
  - PR1186: System.getProperty("deployment.user.security.trusted.cacerts") is null
  - PR909: The Java applet at http://de.gosupermodel.com/games/wardrobegame.jsp fails
  - PR1299: WebStart doesn't read socket proxy settings from firefox correctly

People who helped with this release (If I forgot somebody, please let me know!):

Deepak Bhole <dbhole@redhat.com>
Danesh Dadachanji <ddadacha@redhat.com>
Adam Domurad <adomurad@redhat.com>
Jana Fabrikova <jfabriko@redhat.co>
Peter Hatina <phatina@redhat.com>
Andrew John Hughes <ahughes@redhat.com>
Matthias Klose <doko@ubuntu.com>
Alexandr Kolouch <skolnag@gmail.com>
Jan Kmetko <jan.kmetko.ml@gmail.com>
Omair Majid <omajid@redhat.com>
Thomas Meyer <thomas@m3y3r.de>
Saad Mohammad <smohammad@redhat.com>
Martin Olsson  <martin@minimum.se>
Pavel Tisnovsky <ptisnov@redhat.com>
Jiri Vanek <jvanek@redhat.com>
Jacob Wisor  <gitne@excite.co.jp>

Special thanks to:

 * Adam Domurad - for deep investigations and  fixes in core, and in numerous classloaders or otherwise complicated bugs
 * Jan Kmetko - for initial design of splashscreen
 * Deepak Bhole and Omair Majid - for ever keeping an watchful eye on patches
 * to community - namely:
   -  Jacob Wisor and Alexandr Kolouch  - who voluntary offered and delivered Pl+De and Cz translation

And not finally to all who tested the pre and final versions for regressions

Best regards

Happy hacking,