It’s been quite a while since I’ve updated this blog, but not because we’ve run out of problems – most of the issues were either too complex to document them in a simple article, too confidential to be allowed to be documented here, or both. And I’ve been pretty busy fighting bugs, too, leaving close to no time for getting updates into this blog. So to set a new starting point, here’s a smaller recent issue.
While migrating a PHP application to a newly set up server platform, an issue with non-working connections to a back-end LDAP server turned up. The application just reported some generic problem, while the log revealed a bit more information:
Unable to start TLS and unable to fetch rootDSE entry to see if TLS is supported: Can't contact LDAP server
Fortunately, the LDAP server wasn’t down nor malfunctioning, and fortunately invoking “ldapsearch” to verify the state of the LDAP server (of course invoked on the server running the PHP application) gave an immediate hint at the root cause:
# ldapsearch -Wx -ZZ -h ldap.example.com -b "dc=example,dc=com" "(objectClass=*)" ldap_start_tls: Connect error (-11) additional info: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain) #
Indeed, the LDAP server was configured to use a certificate that was issued by an internal Certification Authority, the latter using a self-signed certificate. But that’s nothing new and the CA certificate was available on the server where we called “ldapsearch”, so what’s the deal?
The server in question (the one running the PHP application) is based on openSUSE Leap 42.2, installed with PHP7 and Apache httpd 2.4. The internal CA’s certificate was stored in /etc/pki/trust/anchors and “update-ca-certificates” was run after installing the CA certificate.
But it seems that this procedure wasn’t enough to let the openssl stack know where to spot the CA certificate store. Running “ldapsearch” under “strace” control revealed that /etc/openssl/ldap.conf was read, but none of the available CA stores were queried.
# grep tls /etc/openldap/ldap.conf ssl start_tls tls_checkpeer no #
As a work-around, /etc/openldap/ldap.conf was extended to list a pointer to the directory-based CA certificate store:
# grep tls /etc/openldap/ldap.conf ssl start_tls tls_checkpeer yes tls_cacertdir /etc/ssl/certs #
Once that was done, “ldapsearch” was able to properly start TLS when connecting to the LDAP server.
But wait: With that statement, we would have expected to see files opened in /etc/ssl/certs. Interestingly, that was not the case. “strace” revealed that instead, /var/lib/ca-certificates/ca-bundle.pem was queried. (That file got updated with the internal CA’s certificate when invoking “update-ca-certificates”). Only after removing the specific internal CA’s entry from that file, we could see that after (unsuccessfully) checking ca-bundle.pem, the actual certificate file in the configured directory was accessed. While the man page for ldap.conf does mention this (“The TLS_CACERT is always used before TLS_CACERTDIR.”), it’s still unclear why TLS_CACERT is only checked once TLS_CACERTDIR is set.
Update: As I’ve been told by SUSE, an according patch was already sent upstream . Creating a distro-specific patch was not considered an option, as this might break setups that rely on the current behavior of the upstream code. From what I read in OpenLDAP’s ITS, OpenLDAP version 2.4.45 will have that fix included.
With OpenSUSE Leap 42.2 current OpenLDAP version 2.4.44, all that was left was to re-generate the CA bundle (which we had “reduced” manually) and to restart Apache httpd, so that the new configuration was actually picked up and the PHP LDAP code was able to connect to the configured LDAP server via TLS.