The Internet is safe & secure, if you use encryption. At least that’s what we believed, before all these attacks on ciphers and their implementation. And since attacks like “Poodle” and “LogJam”, it’s time to upgrade to the most recent TLS protocols.
When you’re running a SUSE Linux Enterprise Server 11 box, even at the latest patch level (SLES 11 SP3), you may notice that you cannot get TLS 1.2 running with the shipped Apache httpd, openssl libraries and “mod_ssl”.
There’s an article on the “SUSE Conversations” site explaining some of the background and stating SUSE’s strategy to handle the issue – which basically means switching to mod_nss.
Installing and configuring apache2-mod_nss
Installing the module to your SLES server is a piece of cake, and migrating your mod_ssl site configuration to a mod_nss variant is assisted by a helper script. All steps necessary are listed in apache2-mod_nss’s README-SUSE.txt file.
Two things worth mentioning come to mind, though:
- When running the helper script “/usr/sbin/mod_nss_migrate.pl” as user root, you’ll end up with an NSS certificate database that is owned by and only readable for “root”. If you run Apache’s httpd as user “wwwrun” (which is the default and should usually not be changed), mod_nss will not be able to access that database and bail out. The results depend on the patch level of your system – it may either be aborts or just messages in httpd’s error log (“NSS_Initialize failed. Certificate database: /etc/apache2/mod_nss.d.”), but either way, you’ll end up with a server not supporting HTTPS at all.
You can resolve this issue by re-owning the NSS files to group “www” and adjusting the permissions:root@server # chgrp www /etc/apache2/mod_nss.d/*.db ; chmod g+r /etc/apache2/mod_nss.d/*.db
- Some versions of the module are shipped with a configuration file (“/etc/apache2/conf.d/mod_nss.conf”) containing references to unsupported ciphers. If httpd does not start because of this (have a look at the SSL site’s error log, for lines like “Unknown cipher ecdhe_rsa_aes_256_sha256”), you’ll have to remove references to the reported cipher from mod_nss.conf manually.
The fun part: FakeBasicAuth
So you have your SSL site up & running again, using mod_nss instead of mod_ssl. It’s late at night, you try to get some sleep, but much too soon your phone rings: The users aren’t able to connect to the site. They’re using client-side certificates, as they did yesterday, but today they’re being told that they’re no longer allowed to access.
Well, what’s documented nowhere (as far as I could see) is a major difference in DN handling between mod_ssl and mod_nss. So if you’re using client-side certificated in conjunction with FakeBasicAuth, the old password and group files no longer work.
Technical Background
For those not too much into the subject, here’s a tiny bit of background information:
“FakeBasicAuth” enables you to handle certificate-based authentication just like regular user names (“BasicAuth”) inside you site configuration. You can thus keep your configuration simple and understandable, and need no external authentication mechanism. In your SSL site config file, you can i.e. add the following statements:
<Location /somewhere> NSSOptions +StrictRequire +ExportCertData +FakeBasicAuth AuthName "Restricted Area" AuthType Basic AuthUserFile /etc/apache2/vhosts.d/passwd AuthGroupFile /etc/apache2/vhosts.d/groups Require group ourSpecialUsers </Location>
This directs httpd to take the client’s user name, look at the given password file for an entry with a matching password, and then to check the named groups file for a group “ourSpecialUsers” with an entry for our current user name.
But wait, it’s about certificates, that’s a tiny bit different: The SSL module checks the validity of the certificate presented by the client (by checking the signature chain) and then reports the certificate’s “distinguished name” (DN, also known as the “certificate subject”) to httpd as the user name of the current client. What makes it a “fake basic auth” is that httpd won’t ever actually ask for a password, but uses an imaginary password with the value “password”. So to make the remainders of the BasicAuth mechanism happy, you’ll have to have entries in your password file with a pattern of “<cert dn>:xxj31ZMTZzkVA”, like
/E=user@company.com,CN=the user,O=Company,L=City,ST=State,C=DE:xxj31ZMTZzkVA
“xxj31ZMTZzkVA” is simply the htpasswd2-encrypted string “password”.
Accordingly, the groups file is a text file with on line per group, with the pattern “<group name>: <DN> [<DN> …]”
ourSpecialUsers: "/E=user@company.com,CN=the user,O=Company,L=City,ST=State,C=DE"
The quotation marks are required if the DN contains blanks, so that httpd will know where one DN ends and the next one starts.
CA certificates
The first step in the process described above is the validation of your client’s certificates, which will fail unless you include the proper CA certificates to the NSS certificate store. The CA certificate does not get migrated automagically by the helper script mentioned above, even if you had the according “SSLCACertificateFile” statement in your former (mod_ssl) site configuration – you’ll have to do this yourself, manually. The according command is referenced in the README_SUSE.txt file, here’s the info for the impatient:
certutil -A -n "<certificate nickname>" -t "CT,," -a -i yourCAcertificate.pem -d /etc/apache2/mod_nss.d
Of course, you need to replace the nickname and the certificate’s PEM file name with values applicable to your environment. We use the certificate’s DN as a nick, which is a bit longish, but works nevertheless. If you’re using a whole directory of CA certs, or a compound file with multiple CA certificates, you’ll better prepare to import them all.
You can have a look at the content of the NSS certificate store via the following command:
root@server # certutil -d /etc/apache2/mod_nss.d -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
C=DE,ST=State,L=City,O=Company,OU=CA Department,CN=The CA,emailAddress=ca@company.com CT,,
Server-Cert u,u,u
You should see at least your server’s certificate and the CA certificate(s) required to validate your client certificates in the resulting list.
Certificate DNs reported to httpd
If you have been using FakeBasicAuth with mod_ssl before, you’ll probably have rubbed your eyes when looking at above examples for the passwd and groups file: Yes indeed, mod_nss reports the DN in RFC 2253 order, as opposed mod_ssl’s “oneline” style:
user@server > openssl x509 -nameopt RFC2253 -subject -noout -in user.cert.pem subject= emailAddress=user@company.com,CN=the user,O=Company,L=City,ST=State,C=DE
user@server > openssl x509 -subject -noout -in user.cert.pem subject= /C=DE/ST=State/L=City/O=Company/CN=the user/emailAddress=user@company.com
So you will have to convert all your entries in your password and groups file(s) to the new format. One of the advantages of (client-side) certificates is that you do not need the certificates stored at the server, which now turns into a disadvantage: You cannot simply mass-generate the “new” DNs from the certificate files, but will likely have to manually edit all entries in all files.
It’s not only much work if you have a number of users, and probably different groups for them as well. It’s also very error-prone, too.
Certificates with UTF-8 characters in the DNs
To make matters worse, some other detail has changed, too: The handling of UTF-8 characters in the certificate DN.
Many names carry non-ASCII characters, and in user-friendly certificates, these reflect in the “common name” (CN) part of the certificate DN. Let’s create a sample:
"/C=DE/ST=State/L=City/O=Company/OU=Department/CN=testuserUmlaut Ü Ue/emailAddress=testuserumlaut@company.com"
(that’s “capital umlaut U” between “testuserUmlaut ” and ” Ue”)
Taking above two versions of the “openssl x509” command, the following subjects are reported:
user@server > openssl x509 -nameopt RFC2253 -subject -noout -in testuserUmlaut.pem subject= emailAddress=testuserumlaut@company.com,CN=testuserUmlaut \C3\9C Ue,OU=Department,O=Company,L=City,ST=State,C=DE
user@server > openssl x509 -subject -noout -in testuserUmlaut.pem subject= /C=DE/ST=State/L=City/O=Company/OU=Department/CN=testuserUmlaut \xDC Ue/emailAddress=testuserumlaut@company.com
You can see that the RFC2253 outputs the Umlaut as “\C3\9C”, while the “oneline” format uses “\xDC”. And while you’ll see the “oneline” format in your site’s error log with mod_ssl, there’s a third variant output by mod_nss:
user /E=testuserumlaut@company.com,CN=testuserUmlaut \xc3\x9c Ue,OU=Department,O=Company,L=City,ST=State,C=DE not found: /
The error message reports “\xc3\x9c” instead of the Umlaut.
So, what do we put into the passwd and groups files? The answer is as easy as it makes one feel uneasy: None of these three.
With mod_ssl, things were simple: You just went ahead and copied that string from the error file to your passwd and groups file, creating the proper entries, and once the files were saved, the user could work with your site.
Not so with mod_nss. Breaking the typical admin work flow, it required you to enter the actual Umlaut into the passwd and groups file:
/E=testuserumlaut@company.com,CN=testuserUmlaut Ü Ue,OU=Department,O=Company,L=City,ST=State,C=DE:xxj31ZMTZzkVA
For the developer of mod_nss, this may have been the most natural thing to do, since the string does indeed look like what any usual application will display as the subject of the certificate. But both for admins doing manual administrations, as well as those trying to automate passwd/groups file maintenance, this is a big hassle.
Conclusion
Switching to mod_nss for SSL sites is fairly easy – unless you have a larger number of users authenticated via client-side certificates. And comparing the test results from Qualys SSL Labs with mod_ssl and those with mod_nss enabled does indeed show the desired improvement.
When it comes to automatically managing web server certificates or client-side certificate DNs, things look entirely different. Changing from mod_ssl to mod_nss will impose a major work impact on the administrator – while it can be done, I rather tend to suggest looking into other options, like upgrading to a newer version of SLES that comes with “openssl” with support for TLS 1.2.