Java problem with mutual TLS authentication when using incoming and outgoing connections simultaneously

In most enterprise environments some form of secure communication (e.g. TLS or SSL) is used in connections between applications. In some environments mutual (two-way) authentication is also a non-functional requirement. This is sometimes referred to as two-way SSL or mutual TLS authentication. So as well as the server presenting it’s certificate, it requests that the client send it’s certificate so that it can then be used to authenticate the caller.

A partner of my current client has been developing a server which receives data over MQTT and because the data is quite sensitive the customer decided that the data should be secured using mutual TLS authentication. Additionally, the customer requires that when the aggregated data which this server collects is posted to further downstream services, it is also done using mutual TLS authentication. This server needs to present a server certificate to its callers so that they can verify the hostname and identity, but additionally it must present a client certificate with a valid user ID to the downstream server when requested to do so during the SSL handshake.

The initial idea was to implement this using the standard JVM system properties for configuring a keystore: "…", i.e. putting both client and server certificates into the single keystore. We soon realised however that this doesn’t work, and tracing the SSL debug logs showed that the server was presenting the wrong certificate, either during the incoming SSL handshake or the outgoing SSL handshake. During the incoming handshake it should present its server certificate. During the outgoing handshake it should present its client certificate.

The following log extracts have been annotated and show the problems:

Upon further investigation it became clear that the problem is related to the default key manager implementation in the JVM. The SunX509KeyManagerImpl class is used for selecting the certificate which the JVM should present during the handshake, and for both client certificate and server certificate selection, the code simply takes the first certificate it finds:

    String[] aliases = getXYZAliases(keyTypes[i], issuers);
    if ((aliases != null) && (aliases.length > 0)) {
        return aliases[0];  <========== NEEDS TO BE MORE SELECTIVE

The aliases returned by the method on the first line simply match key types (e.g. DSA) and optional issuers. So in the case where the keystore contains two or more certificates, this isn’t selective enough. Furthermore, the order of the list is based on iterating over a HashMap entry set, so the order is not say alphabetical, but it is deterministic and constant. So while searching for the server certificate, the algorithm might return the client certificate. If however that part works, the algorithm will then fail when the server makes the downstream connection and needs to present its client certificate, as again the first certificate will be presented, namely the server certificate. As such, because it is impossible to create concurrent incoming and outgoing two-way SSL connections, I have filed a bug with Oracle (internal review ID 9052786 reported to Oracle on 20180225, now officially JDK-8199440).

One solution is to use two keystores, one for each certificate as demonstrated here.

A possible patch for the JVM would be to make the algorithm more selective by using the "extended key usage" certificate extensions. Basically the above code could be enhanced to additionally check the extended key usage and make a more informed decision during alias selection, for example:

String[] aliases = getXYZAliases(keyTypes[i], issuers);
if ((aliases != null) && (aliases.length > 0)) {
    String alias = selectAliasBasedOnExtendedKeyUsage(aliases, "");  //TODO replace with constant
    if (alias != null) return alias;

    //default as implemented in openjdk
    return aliases[0];

The method to select the alias would then be as follows:

private String selectAliasBasedOnExtendedKeyUsage(String[] aliases, String targetExtendedKeyUsage) {
    for(String alias : aliases){
        //assume cert in index 0 is the lowest one in the chain, and check its EKU
        X509Certificate certificate = this.credentialsMap.get(alias).certificates[0];
        List ekus = certificate.getExtendedKeyUsage();         for (String eku : ekus) {             if(eku.equals(targetExtendedKeyUsage)){                 return alias;             }         }     }     return null; } 

More details including a fully running example and unit tests are available here.

Copyright ©2018, Ant Kutschera