RHEL 6.3 – LDAP Series – Part 4 : Troubleshooting

 
RHEL 6.3 - LDAP Troubleshooting
 
This is  my fourth post in RHEL 6.3 LDAP implementation Series. And the purpose of this post is to give extra extra muscle to troubleshoot the issues that you encounter during or after the LDAP implementation. In this post i am documenting the troubleshooting tips that i used to solve various questions that I encountered during the LDAP configuration.
 
For successful LDAP encryption configuration, the following command from the client should show the server’s configuration without any errors. 
 
# ldapsearch -x  -b ‘dc=gurkulindia,dc=com’
 
Sometimes, we see that test fails with different errors . And in many cases the command hangs without any error. Troubleshooting at this level is very difficult because we will have no related logs neither at the server nor at the client. This Section I will be explaining the procedure to troubleshoot the connection issues between ldap client and server. 
 
 
 
 

My LAB Setup Information

 
Certification Authority ( CA) SERVER    : gurkulrhelca 
LDAP SERVER  and Self LDAP Client       : gurkulrhel1  – alias ldapserver
LDAP CLIENT                             : gurkulrhel2  
 

Certification location and their description  for Encrypted Communication. 

 
CA Certificate   :  /etc/pki/tls/certs/cacert.pem
LDAP SERVER Private Key File   :  /etc/pki/tls/certs/slapdkey.pem
LDAP SERVER Certificate Request File :  /etc/pki/tls/certs/slapdcert.pem
 

#Note 1 : RHEL 6.3  uses “/etc/openldap/certs”  default path for the signed certificates and keys. But in my configuration i have placed them in /etc/pki/tls/certs. And I have removed the below entry from the file /etc/openldap/ldap.conf file. But that is my mistake actually, I will show how I did i corrected my mistake in the later post.

 
            TLS_CACERTDIR /etc/openldap/certs
 

Content of /etc/pki/tls/certs

 
[root@gurkulrhel1 certs]# ls -l /etc/pki/tls/certs
-rw-r–r–. 1 ldap ldap 571410 Sep  2  2011 ca-bundle.crt
-rw-r–r–. 1 root root 651043 Sep  2  2011 ca-bundle.trust.crt
-rw-r–r–  1 ldap ldap   1505 Mar 29 20:00 cacert.pem   
-rw-r–r–  1 root root   2242 Mar  5 06:12 Makefile
-rw-r–r–  1 ldap ldap   4726 Mar 29 21:30 slapdcert.pem     
-rw-r–r–  1 ldap ldap   1704 Mar 29 21:29 slapdkey.pem      
 
 

LDAP CLIENT CONFIGURATION FILE

 
[root@gurkulrhel1 certs]# cat /etc/openldap/ldap.conf
URI ldaps://gurkulrhel1
BASE dc=gurkulindia,dc=com
[root@gurkulrhel1 certs]#
 

LDAP NSCLD CONFIGURATION FILE

[root@gurkulrhel1 certs]# cat  /etc/nslcd.conf
uid nslcd
gid ldap
uri ldaps://gurkulrhel1
base dc=gurkulindia,dc=com
ssl start_tls
tls_reqcert allow
[root@gurkulrhel1 certs]#
 

Check that nslcd ( nss-pam-ldapd daemon) running fine.

 
[root@gurkulrhel1 certs]# service nslcd status
nslcd (pid  5458) is running…
[root@gurkulrhel1 certs]#
 

Now testing the connection

 
[root@gurkulrhel1 certs]# ldapsearch -x  -b ‘dc=gurkulindia,dc=com’
ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)
[root@gurkulrhel1 certs]#
 

Troubleshooting Procedure 

 

Troubleshooting Stage 1: Just run the ldapsearch with debug level 3  and you will see following kind of output:

 
[root@gurkulrhel1 certs]# ldapsearch -x  -d3 -b ‘dc=gurkulindia,dc=com’
ldap_create
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP gurkulrhel1:636
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 192.168.1.31:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
tls_write: want=70, written=70
  0000:  16 03 01 00 41 01 00 00  3d 03 01 51 56 7e 84 7b   ….A…=..QV~.{
  0010:  b6 ef 9e f6 19 22 b8 21  49 52 4b 88 cb dc 35 c8   …..”.!IRK…5.
  0020:  ea 95 7e fd 9e f2 ae f6  16 91 42 00 00 16 00 ff   ..~…….B…..
  0030:  00 35 00 04 00 05 00 2f  00 0a 00 09 00 64 00 62   .5…../…..d.b
  0040:  00 03 00 06 01 00                                  ……
tls_read: want=5, got=5
  0000:  16 03 01 08 c9                                     …..
tls_read: want=2249, got=2249
  0000:  02 00 00 4d 03 01 51 56  7e 84 a1 75 51 83 fe b2   …M..QV~..uQ…
  08b0:  b2 03 dc b0 49 f9 3d 17  d3 8f 7c 44 26 17 e6 eb   ….I.=…|D&…
  08c0:  db 33 ae ef 3c 0e 00 00  00                        .3..<….
TLS: certificate [E=root@gurkulrhelca,CN=gurkulrhelca,OU=Gurkulindia,O=Gurkulindia Company Ltd,L=Singapore City,ST=Singapore,C=SG] is not valid – error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
tls_write: want=7, written=7
  0000:  15 03 01 00 02 02 30                               ……0
TLS: error: connect – force handshake failure: errno 21 – moznss error -8172
TLS: can’t connect: TLS error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
ldap_err2string
ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)
[root@gurkulrhel1 certs]#
 
#Note 2 : At this level we know the error is related to certificate. Not the connection with ldap server. Since we can see that the connection was established already in the top with below messages
 
ldap_connect_to_host: Trying 192.168.1.31:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
 
#Note 3 : All is complaining here is the CA which issued the current certificate is not trustable. But we know we have copied right CA certificate in the location /etc/pki/tls/certs/cacert.pem.  
 
TLS: certificate [E=root@gurkulrhelca,CN=gurkulrhelca,OU=Gurkulindia,O=Gurkulindia Company Ltd,L=Singapore City,ST=Singapore,C=SG] is not valid – error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
TLS: error: connect – force handshake failure: errno 21 – moznss error -8172
 

Question 1: all we want to confirm is that CA certificate right one?

Observation :  The desription of the certificate “[E=root@gurkulrhelca,CN=gurkulrhelca,OU=Gurkulindia,O=Gurkulindia Company Ltd,L=Singapore City,ST=Singapore,C=SG]” says that the tls looking at right certification. 

 

Question 2: Is that ldap using default tls/ssl to decrypt the certificate.

Observation : No, the error “TLS: error: connect – force handshake failure: errno 21 – moznss error -8172” says, ldap using moznss authentication. 

 

Question 3 : What is moznss?

Observation: Starting from “openldap-clients-2.4.23-26” version, ldap’s openssl by default uses mozilla nss insteald of TLS/SSL. And you can check this from the rpm change log. Mozilla NSS uses the NAA Certificate/Key database located in “/etc/pki/nssdb” instead of the pki files. To switch to  Mozilla NSS Certificate/Key database instead of flat PEM files we need to create a Mozilla Database and specify the path of NSS db in TLS_* directives. 
 
[root@gurkulrhel1 certs]# rpm -qa|grep openldap-clients
openldap-clients-2.4.23-26.el6_3.2.x86_64
 
[root@gurkulrhel1 certs]# rpm -q –changelog openldap-clients | grep -e ‘^*’ -e ‘uses Mozilla’ | grep -B1 ‘uses Mozilla’
* Sat Dec 04 2010 Jan Vcelak <jvcelak@redhat.com> 2.4.23-4
– uses Mozilla NSS instead of OpenSSL for TLS/SSL
 
Solution: My observation, we don’t have any NSS database created, all we have is the .pem files which should work properly with tls/ssl authentication. Just to force the authentication to TLS/SSL, i added  the “TLS_CACERTDIR /etc/pki/tls/certs” to “/etc/openldap/ldap.conf”, to refer the path where i have kept my key files ( if see my #Note 1, you can notice that I have removed this entry purposefully for different reason)
 
[root@gurkulrhel1 ]# grep -v ‘^#’ /etc/openldap/ldap.conf
TLS_CACERTDIR /etc/pki/tls/certs
URI ldaps://gurkulrhel1
BASE dc=gurkulindia,dc=com
[root@gurkulrhel1]#
 
————————————————————————————
man refernce ldap.conf  .. TLS_CACERTDIR
 
TLS_CACERTDIR <path>
 Specifies the path of a directory that contains Certificate  Authority  certificates  in  separate  individual
 files.  The  TLS_CACERT is always used before TLS_CACERTDIR.  The specified directory must be managed with the
 OpenSSL c_rehash utility.  This parameter is ignored with GNUtls.
 
 When using Mozilla NSS, <path> may contain a Mozilla NSS cert/key database.  If <path> contains a Mozilla  NSS
 cert/key  database  and  CA  cert  files,  OpenLDAP will use the cert/key database and will ignore the CA cert
 files.
————————————————————————————-
 
 

Troubleshooting Stage 2: Now test the connection

 
[root@gurkulrhel1 certs]# ldapsearch -x -d3 -b ‘dc=gurkulindia,dc=com’
ldap_create
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP gurkulrhel1:636
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 192.168.1.31:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
TLS: file cacert.pem does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file ca-bundle.trust.crt does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file make-dummy-cert does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file slapdkey.pem does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file client.pem does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file localhost.crt does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file Makefile does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file ca-bundle.crt does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file slapdcert.pem does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
TLS: file dump does not end in [.0] – does not appear to be a CA certificate directory file with a properly hashed file name – skipping.
tls_write: want=70, written=70
 0000:  16 03 01 00 41 01 00 00  3d 03 01 51 56 99 81 82   ….A…=..QV…
 0010:  3b 95 3b 7c b0 e3 be fa  65 7e d0 25 8f 11 c6 f3   ;.;|….e~.%….
 0020:  41 25 b9 a2 84 dc 1e 2f  94 12 6f 00 00 16 00 ff   A%…../..o…..
 
 ::: Snip the output :::::
 
 08a0:  05 df 97 19 a7 18 03 28  2b 18 4d 72 16 8d 2b 16   …….(+.Mr..+.
 08b0:  b2 03 dc b0 49 f9 3d 17  d3 8f 7c 44 26 17 e6 eb   ….I.=…|D&…
 08c0:  db 33 ae ef 3c 0e 00 00  00                        .3..<….
TLS: certificate [E=root@gurkulrhelca,CN=gurkulrhelca,OU=Gurkulindia,O=Gurkulindia Company Ltd,L=Singapore City,ST=Singapore,C=SG] is not valid – error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
tls_write: want=7, written=7
0000:  15 03 01 00 02 02 30                               ……0
TLS: error: connect – force handshake failure: errno 0 – moznss error -8172
TLS: can’t connect: TLS error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
ldap_err2string
ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)
[root@gurkulrhel1 certs]#
 
 

Question : There is an error for every “.pem” file that we placed in the “etc/pki/tls/certs” directory, complaining that the file name doesn’t have the proper file format and  also says that files doesn’t appear like hashed files. what is hashed files?

Answer   : OpenSSL looks up certificates by their hash. We need to Generate hash for the CA certificate. We have two ways to create a hashed file:
 
Method  1   :  create Hashed file for each .pem as below . e.g. cacert.pem
 
[root@gurkulrhel1#  HASH=$( openssl x509 -noout -hash -in /etc/pki/tls/certs/cacert.pem )
[root@gurkulrhel1#  ln -s /etc/pki/tls/certs/cacert.pem /etc/pki/tls/certs/${HASH}.0
 
Method  2   : Create hashed file for all the .pem files from the directory using the tool “cacertdir_rehash”
 
[root@gurkulrhel1 certs]# cacertdir_rehash /etc/pki/tls/certs
[root@gurkulrhel1 certs]# ls -l /etc/pki/tls/certs
 
total 1232
lrwxrwxrwx  1 root root     13 Mar 30 16:28 381ce4dd.0 -> ca-bundle.crt
lrwxrwxrwx  1 root root     19 Mar 30 16:28 381ce4dd.1 -> ca-bundle.trust.crt
lrwxrwxrwx  1 root root     10 Mar 30 16:28 539a37f4.0 -> cacert.pem
lrwxrwxrwx  1 root root     10 Mar 30 16:28 8e89bed9.0 -> client.pem
lrwxrwxrwx  1 root root     13 Mar 30 16:28 8e89bed9.1 -> slapdcert.pem
-rw-r–r–. 1 ldap ldap 571410 Sep  2  2011 ca-bundle.crt
-rw-r–r–. 1 root root 651043 Sep  2  2011 ca-bundle.trust.crt
-rw-r–r–  1 ldap ldap   1505 Mar 29 20:00 cacert.pem
-rw-r–r–  1 ldap ldap   1517 Mar 30 01:57 client.pem
lrwxrwxrwx  1 root root     13 Mar 30 16:28 da4d55fe.0 -> localhost.crt
-rw——-. 1 ldap ldap   1188 Sep 27  2012 localhost.crt
-rwxr-xr-x  1 root root    610 Mar  5 06:12 make-dummy-cert
-rw-r–r–  1 root root   2242 Mar  5 06:12 Makefile
-rw-r–r–  1 ldap ldap   4726 Mar 29 21:30 slapdcert.pem
-rw-r–r–  1 ldap ldap   1704 Mar 29 21:29 slapdkey.pem
[root@gurkulrhel1 certs]#
 
 
 

Troubleshooting Stage 3 : Now retest the connection. . The results says our configuration test successful

 
[root@gurkulrhel1 certs]# ldapsearch -x  -b ‘dc=gurkulindia,dc=com’
# extended LDIF
#
# LDAPv3
# base <dc=gurkulindia,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
 
# gurkulindia.com
dn: dc=gurkulindia,dc=com
objectClass: top
objectClass: domain
dc: gurkulindia
 
# Groups, gurkulindia.com
dn: ou=Groups,dc=gurkulindia,dc=com
objectClass: top
objectClass: organizationalUnit
ou: Groups
 
# People, gurkulindia.com
dn: ou=People,dc=gurkulindia,dc=com
objectClass: top
objectClass: organizationalUnit
ou: People
 
# gurkuluser, People, gurkulindia.com
dn: uid=gurkuluser,ou=People,dc=gurkulindia,dc=com
givenName: ldap
sn: user1
loginShell: /bin/bash
uidNumber: 1250
gidNumber: 1500
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
uid: gurkuluser
cn: ldap user1
homeDirectory: /home/gurkuluser
userPassword:: e1NTSEF9NWtPL0sxS0J6SjB3RWFLQkFHaklEWTZNRzZUR3pnOVE=
 
# redhat, Groups, gurkulindia.com
dn: cn=redhat,ou=Groups,dc=gurkulindia,dc=com
objectClass: posixGroup
objectClass: top
cn: redhat
gidNumber: 1500
 
# search result
search: 2
result: 0 Success
 
# numResponses: 6
# numEntries: 5
[root@gurkulrhel1 certs]#
 
 
 
 

 

How to Stay Connected to Us ?

You can simply subscribe for our free email posts from here

 

You can always stay close to us by connecting in  Facebook,  LinkedIn twitter and Google + social networks.   And We have very active Facebook’s just-UNIX-no-noise group and   Linked in  Enterprise UNIX administration group, for active discussions.

We always love to hear your comments and feedback. 

 

 

 

 

 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

8 Responses

  1. Pawan says:

    Very nice tutorial.

  2. Vishal says:

    ssh ldapuser1@client
    ldapuser1@client’s password:
    Last login: Sat May  4 21:22:15 2013 from 
    Could not chdir to home directory /home/ldapuser1: No such file or directory
    -bash-4.1$
    =========================================
    Home directory is not getting created .. I am new to ldap 

  3. Un Garvin says:

    Nice post. I was checking constantly this weblog and I’m impressed!
    Very useful info particularly the closing phase :
    ) I care for such info a lot. I used to be seeking this certain info for a long time.
    Thank you and good luck.

  4. Yogesh Raheja says:

    Thanks Un Garvin.

  5. Giedrius says:

    I still get that problem, even after rehashing certs dir


    TLS: error: connect – force handshake failure: errno 21 – moznss error -8172
    TLS: can’t connect: TLS error -8172:Peer’s certificate issuer has been marked as not trusted by the user..
    ldap_err2string
    ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)

    [root@server certs]# certutil -d . -L
    Certificate Nickname Trust Attributes
    SSL,S/MIME,JAR/XPI
    ServerCert u,u,u
    Company Root CA CTu,u,u

  6. João says:

    Very nice! Helped me a lot…

  7. aaron says:

    really really helpful! thanks idol!

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us