Java Web Start connection problem

Anyone who has deployed any large application using Java Web Start 1.5 or 1.6 over a slow WAN or through some kind of SSL proxy has noticed the extreme number of connections that can be initiated during application startup. The first deploy of the application works as expected and all the JAR files is downloaded. But the sencond time the user clicks to start the application, Java Web Start will flood the network with connections to the server.

Investigation the problem revelas an interesting piece of code in the class com.sun.javaw.LaunchDownload and the method updateCheck, this function is called once for every JAR file in the JNLP description file. The code does the update check by starting a new thread and doing a HTTP request to the web server to check if the JAR file has been updated.
This means that if we have a JNLP file with 200 JAR files, the code will start 200 threads, creating 200 individual connections to the server!

The solution can be found in the same function as well, as the first lines of the function reads:

// no update check for versioned resource
if (version != null) return;


This means that if we put version attributes on the JAR files we will not invoke the badly written code. To add the version attribute is simple, just add it to the JAR element of the JNLP file:

<jar href="/application/foo.jar" version="”1.0” />


When Java Web Start finds this attribute on a JAR element in the JNLP file it will send this version string along with the GET request to the server. Like:

GET /application/foo.jar?version-id=1.0

To handle this on the server we have two options, if we are running a J2EE environment the JDK contains a JNLP servlet that can respond to this request and also return the required x-java-jnlp-version-id custom header, or we could implement the same functionality in some other kind of server side language.

I choose to implement the JAR server in PHP/Apache since we have noticed that the J2EE container we are using is not very good at serving large amounts of data.

To have Apache invoke my script for every JAR file requested from the server, I added the following lines to the httpd.conf file.

AddType application/java-archive .jar

Action application/java-archive /cgi-bin/jar_send.php

The script is very simple; we have chosen to store the versioned JAR archives
as /application/foo_x_y.jar for version x.y. This makes the script very simple to implement, and make deploying very simple as well. Another strategy could be to store a complete version of the application under /x.y/application/foo.jar and have all JAR archives in the JNLP file reference the same version.

Here is the code for the jar_send.php script:

<?

 // Make sure we have got a version-id argument
 if (isset($_GET["version-id"])) {
   $version = $_GET["version-id"];
 } else {
   $version = null;
 }
  
 // Retreive the requested file
 $file = $_SERVER["PATH_TRANSLATED"];
 
 // If not version is requested, or version is 1.0, send foo.jar
 if ($version == null || $version == "1.0") {
   $path = $file;
 } else {
   // If version 1.1 is requested, send foo_1_1.jar
   $x = strrpos($file,".jar");
   $path = substr($file,0,$x);
   $path .= "_";
   $path .= str_replace(".","_",$version);
   $path .= ".jar";
 }
 
 // Make sure the file exists
 if (!is_file($path)) {
   header("HTTP/1.0 404 NOT FOUND");
   print $path;
   die;
 }
  
 // Open the file
 $f = fopen($path, 'rb');
 if ($f == null) {
   header("HTTP/1.0 404 NOT FOUND");
   print $path;
   die;
 }

 // Send the JNLP custom header
 header("x-java-jnlp-version-id: $version");

 // Inform Apache about how much data we are going to send
 header("Content-Length: ".(string)(filesize($path)));
 
 // Send the data, 8K blocks
 while(!feof($f) &&(connection_status()==0)) {
   print(fread($f, 1024*8));
   flush();
 }

 // Close the file
 fclose($f);

?>

Java SSL client authentication with intermediate CA

Last week I helped a customer with a problem doing client authentication against a web-service. The connection did not work becuse the client software, implemented using JBoss was not sending in its client certificate during the handshake. The client certificate was issued by Verisign, witch uses an intermediate CA to issue the certificates.

The server side was using a Microsoft ISA 2004 server. We started with checking the list of trusted certificate issuers the server was sending out with the help of the openssl toolkit.

openssl s_client -host secure.customer.com -port 443

Look at the list after Acceptable client certificate CA names.
We concluded that the server was only sending a list of self-signed root CA certificates. The server had a lot more CA certificates installed but only a subset was presented as acceptable client authentication CA's.

In Microsoft ISA 2004 you don't have any means of configuring the list of acceptable CA's so we had to look at the other side in the communication.

The client software is implemented using JBoss 4.0.5.GA and JBoss remoting 1.4.3.GA but the as we found out, the problem was a general java problem.

When requested to present a client certificate the Java SSL classes uses an interface called KeyManager to locate the correct credential to use. The default implementation SunX509KeyManagerImpl does a search through all PrivateKeyEntry elements found in the configured keystore. Then doing the search the class will match all certificates in the entry against the list of acceptable CA's.

To get client authentication to work with a client certificate issued by an intermediate CA we need to have the complete chain present in the PrivateKeyEntry in the keystore.

To update a keystore with the correct certificate chain, we can use the openssl toolkit again. First get hold of all certificates in PEM format. To convert a DER encoded certificate to PEM format use:

openssl x509 -in file.der -inform DER -out file.pem

Then create a PKCS#7 encoded certificate chain with:

openssl crl2pkcs7 -nocrl -out chain.pem -certfile root.pem -certfile intermediate.pem -certfile clientcert.pem

How this complete certificate chain can be imported into the java keystore using the standard keytool program. But first we need to know the alias of the key we are going to update. We can list the content of the keystore with:

keytool -list -keystore keystore-file

During the import of the certificate chain, keytool will check that the certificates match the private key, so it's should not be possible to corrupt the keystore. But as always backup the file if your production depends on it...

To import the chain, use:

keytool -import -keystore keystore-file -alias alias -file chain.pem

Keytool will ask if you trust the certificate issuer, just answer Yes.

We can now check the content of the keystore again and make sure that the complete chain is present with:

keytool -list -v -keystore keystore-file

This command will do a verbose listing of the keystore, with all certificates.