Solaris Troubleshooting : Sendmail Troubleshooting – 2

This post  is continuation to the Sendmail Troubleshooting – 1


Step 5. Verify the sendmail configuration.
In Solaris 9 and above, sendmail uses a 2 queue paradigm structure (/var/spool/mqueue and /var/spool/clientmqueue). This is to make sendmail more secure by not giving out setuid for root in the binary, unlike Solaris 8 and below which has one queue (/var/spool/mqueue). 

In Solaris 9 and above, there are 2 sendmail daemons for the 2 queues: 

“/usr/lib/sendmail -Ac -q15m” which is owned by smmsp and the other “/usr/lib/sendmail -bd -q15m” owned by root. The former uses the /var/spool/clientmqueue and the latter uses /var/spool/mqueue. 

i ) verify the /etc/mail directory exist and sendmail.cf and submit.cf exist.  

The main sendmail daemon which listen on port 25 uses the 

sendmail.cf, while the secondary one uses the submit.cf. 

# ls -ail /etc/mail 

total 924 

1445 drwxr-xr-x   3 root     mail        1024 Mar 13 13:48 . 

1403 drwxr-xr-x  76 root     sys         4608 Mar 27 11:59 .. 

19564 -rw-r–r–   1 root     bin          163 Oct 29 12:40 Mail.rc 

3964 -rw-r–r–   1 root     bin         1423 Oct 29 12:24 aliases 

2405 -rw-r—–   1 root     smmsp      40960 Nov  6 14:45 aliases.db 

22855 drwxr-xr-x   9 root     mail         512 Nov  6 14:35 cf 

21951 -rw-r–r–   1 root     bin         5449 Dec 22  2006 helpfile 

4055 -rw-r–r–   1 root     bin            9 Nov 30 10:43 local-host-names 

2977 -r–r–r–   1 root     bin        39953 Dec 22  2006 local.cf 

1865 -rw-r–r–   1 root     bin         1839 Oct 29 12:16 mailx.rc 

4048 lrwxrwxrwx   1 root     root          11 Oct 29 12:24 main.cf -> sendmail.cf 

42821 -rw-r–r–   1 root     root          50 Nov 30 11:09 relay-domains 

42667 -r–r–r–   1 root     root       40551 Mar 14 14:58 sendmail.cf 

41752 -r–r–r–   1 root     other      39900 Nov  6 14:35 sendmail.cf.old      

4331 -rw-r–r–   1 root     root       40032 Nov 29 11:35 sendmail.cf_cust 

21801 -r–r–r–   1 root     root       39875 Mar 13 13:48 sendmail.cf_orig 

4054 -r–r–r–   1 root     bin        39895 Nov 28 10:39 sendmail.cf_save 

4049 lrwxrwxrwx   1 root     root           8 Oct 29 12:24 sendmail.hf -> helpfile 

21832 -rw-r–r–   1 root     bin        41448 Mar 14 15:01 submit.cf 

41761 -r–r–r–   1 root     other      40241 Nov  6 14:35 submit.cf.old 

21818 -rw-r–r–   1 root     root       40216 Mar 13 13:48 submit.cf_orig 

4056 -r–r–r–   1 root     bin        40220 Nov  9 12:26 submit.cf_save 

4050 lrwxrwxrwx   1 root     root          11 Oct 29 12:24 subsidiary.cf -> sendmail.cf 

42853 -rw-r–r–   1 root     root           5 Nov 30 11:10 trusted-users 

4058 -rw-r–r–   1 root     bin            0 Nov 30 10:47 trusted-users_save 

ii)  Verify the sendmail.cf has the correct mailhost (this may differs from unique site, see summary) setup. 

In Solaris 9 and above : 

# grep DS sendmail.cf 

DS<blank>                                      

# grep Fallback sendmail.cf 

O FallbackSmartHost=mailhost$?m.$m$. 

Basically the default behaviour for sendmail is mail exchanger, it will use dns servicces to query for the MX host entry of the domainname used in the addresses for forwarding the messages. If this failed for some reasons, eg. the dns does not know the internet domains, it will fallback to use mailhost as defined in the 

FallbackSmartHost.m4 macros:  define(`confFALLBACK_SMARTHOST’, `mailhost$?m.$m$.’) 

In Solaris 8 and below : 

# grep DS sendmail.cf 

DSmailhost$?m.$m$ 

sendmail by default is set to forward all messages to “mailhost” the smarthost. 

m4 macros: 

define(`SMART_HOST’, `mailhost$?m.$m$.’) 

iii) Verify the submit.cf  

# grep MTAHost submit.cf 

D{MTAHost}[127.0.0.1] 

This is to route the local messages to the localhost port 25. 

# grep DS submit.cf 

DS<blank> 

iv)  Verify that the port 25 is configured 

Enabling Access to Remote Clients   On an  unmodified  system,  access  to  sendmail  by  remote   clients  is enabled and disabled through the service management facility. In particular, remote access is   determined by the value of the local_only SMF property:  

 svc:/network/smtp:sendmail/config/local_only = true 

 A setting of true, as above, disallows remote access;  false   allows remote access. The default value is true. 

 The following example shows the  sequence  of  SMF  commands   used to enable sendmail to allow access to remote systems: 

 # svccfg -s svc:/network/smtp:sendmail setprop config/local_only = false 

 # svcadm refresh svc:/network/smtp:sendmail 

 # svcadm restart svc:/network/smtp:sendmail 

 In Solaris 9 and above : 

 # grep Port sendmail.cf 

  O DaemonPortOptions=Name=MTA-v4, Family=inet 

  O DaemonPortOptions=Name=MTA-v6, Family=inet6 

  O DaemonPortOptions=Port=587,, M=E 

 These are the default configuration which has port 25 setup by the above and port 587 has been setup to listen as well. 

 In Solaris 8 and below : 

# grep Port sendmail.cf 

O DaemonPortOptions=Name=MTA-IPv4, Family=inet 

O DaemonPortOptions=Name=MTA-IPv6, Family=inet6 

O DaemonPortOptions=Port=587,, M=E 

m4 macros : 

DAEMON_OPTIONS(`NAME=MSA, Port=27, Addr=127.0.0.1, M=E’) 

V)  For receiving mail, ensure the local-host-names has populated with the localhost name  

eg.  # more local-host-names 

v4u-x1c 

   

Step 6. Verify the messages logged by sendmail trough syslog. 

 i) Verify where syslogd will log sendmail syslog records to. 

After a Solaris install you will find following entries in /etc/syslogd.conf: 


*.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages

mail.debug                      ifdef(`LOGHOST’, /var/log/syslog, @loghost)
…This has the effect that:
– syslog will log critical sendmail messages to /var/adm/messages
–  if loghost can be resolved, syslogd will log sendmail syslog record to the
/var/log/syslog file of that host. 

  

Resolve the ‘loghost’ hostname:
$ getent hosts loghost
172.16.1.1      e450 loghost

If loghost points to the same system where the sendmail service your want to troubleshoot is running on, you will find the sendmail syslog records in /var/log/syslog.
If loghost points to a different system, login into this system and verify the contents of /var/log/syslog. If neither is the case, verify the contents of syslod.conf and verify to which host and/or file mail.debug records points to. 

ii) Verify the contents of /var/adm/messages.

Syslog messages with priotity ‘mail.crit’ will by default logged to /var/adm/messages. 

example: 

The following will be logged when the writing permissions to /var/spool/clientmqueue are not sufficient. 


May 20 04:01:12 db7 sendmail[1872]: [ID 801593 mail.crit] NOQUEUE:SYSERR(oracle): can not write to queue directory /var/spool/clientmqueue/(RunAsGid=0, required=1): Permission denied
May 20 05:00:01 db7 sendmail[1961]: [ID 801593 mail.crit] NOQUEUE:SYSERR(sys): can not write to queue directory /var/spool/clientmqueue/(RunAsGid=0, required=1): Permission denied
… 

 iii)  Verify the contents of the /var/log/syslog file or equivalent file. 

Whenever the delivery status of a mail message changes, sendmail will record this event via syslog. 

Example:

Apr 16 10:58:56 mymailhost sendmail[24234]: [ID 801593 mail.info] m3G8wu8g024234: to=<joe.foe@extdomain.com>, ctladdr=<john@mymailhost.mydomain.com> (22960/117), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30920, relay=mailhost.extdomain.com. [172.17.1.4], dsn=2.0.0, stat=Sent (m3G8wug1009950 Message accepted for delivery)

 Each line of information logged looks like this: 

 date host sendmail[pid]: qid: field1=value,field2=value,field3=value, … 

– The date is the month, day, and time that the line of information was logged.
– The host is the name of the host that produced this information.
  This can be different from the name of the host on which the logfiles are kept (see above).
 – The pid (process id) of the sendmail processes that produced the output.
 – The qid (queue identifier) that uniquely identifies each message on a given host.
– The remainder is a list of fields which define values of who the sender or the recipient is
  and whether delivery succeeded, failed, or was deferred.
With the above list you can identify which record(s) points to a local delivery attempt, a relay attempt, a forwarding attempt, etc… 

The following is a list of some of these fields. 

to=             The final recipient
from=           The envelope sender
ctladdr=        The controlling user
delay=          Total time to deliver
xdelay=         Transaction delay for this address only
mailer=         The delivery agent used
pri=            The initial priority
relay=          The host that sent or accepted the message
dsn=            The DSN status code     
stat=           The status of delivery
size=           The size of the message
ntries=         The number of delivery attempts 

iv) Verify the stat= field of this record. 

This can be in the state queued when the recipient’s mailhost was not accessible. After the same mailhost is accessible again, and after re-queuing of mail messages, the same  message can be in the state Sent as mail was successful delivered. In the case of bounced mail the stat= field will show the reason for the failure. 

v) Verify the dsn= field of this record (when available). 

When a mail server for some reason bounces a message, it may notify the envelope sender of the problem with a DSN error code. The meaning of the DSN error codes are  documented in RFC1893 (Enhanced Mail System Status Codes). 

example:

Apr 11 10:25:29 e450 sendmail[16860]: [ID 801593 mail.info] m3B8PSVM016855: to=<joe@wrongdomain.org>, ctladdr=<john@mydomain.org> (2031/2001), delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=150549, relay=wrongdomain.org.be, dsn=5.1.2, stat=Host unknown (Name server: wrongdomain.org: host not found)
…Here you can see that an attempt was made by john@mydomain.org to send out a mail to a user joe on a non-resolvable domainname. 

  

dsn=5.1.2 is decipherd according to RFC1893

    5.X.X   Permanent Failure 

A permanent failure is one which is not likely to be resolved by resending the message in the current form.  Some change to the message or the destination must be made for successful delivery.
                    … 

 X.1.2  Bad destination system address 

The destination system specified in the address does not exist or is incapable of accepting mail.  For Internet mail  names, this means the address portion to the right of the           “@” is invalid for mail.  This codes is only useful for  permanent failures.
… 

 vi)  Verify the reject= field of this record (when available).

When a mail message is processed by sendmail it will go through some check_rules used of Anti-spam control. (see the  ANTI-SPAM CONFIGURATION CONTROL section of the /usr/lib/mail/README file) If one of these rules rejects a mail message, sendmail will log this.
Example:

Apr 16 15:38:34 e450 sendmail[17707]: [ID 801593 mail.notice] m3GDcYW8017707: ru leset=check_mail, arg1=<nekkipfa1981@54sales.com>, relay=to1-84-91-48-146.netvis ao.pt [84.91.48.146], reject=553 5.1.8 <nekkipfa1981@54sales.com>… Domain of sender address nekkipfa1981@54sales.com does not exist

and

Apr 16 16:50:54 e450 sendmail[18581]: [ID 801593 mail.notice] m3GEorFQ018581: ru leset=check_rcpt, arg1=<a286e4184@yahoo.com.tw>, relay=203-73-236-153.adsl.dynam ic.seed.net.tw [203.73.236.153], reject=550 5.7.1 <a286e4184@yahoo.com.tw>… Relayin Solaris 9 Operating SystemSolaris 8 Operating System 

Step 7. Verify the permissions of sendmail’s mail queues.
 

In Solaris Operating System 9 and above, the sendmail releases uses the 2 queue paradigm: 

                   /var/spool/mqueue
                   /var/spool/clientmqueue 

                   Unlike Solaris Operating System 8 and below which has one queue (/var/spool/mqueue). This paradigm allows us not to give away setuid for root in the binary unlike S8 and below. 

i)Verify the /var/spool/mqueue exist, also has the following permissions and ownerships  

# ls -ail /var/spool/mqueue 

For S9 and above : 

drwxr-x—   2 root     bin         /var/spool/mqueue 

#/usr/bin/chmod 750 /var/spool/mqueue 

#/usr/bin/chown root:bin /var/spool/mqueue 

For S8 and below : 

drwxr-x—   2 root     bin         /var/spool/mqueue 

#/usr/bin/chmod 750 /var/spool/mqueue 

#/usr/bin/chown root:bin  /var/spool/mqueue 

 ii) For S9 and above, verify the /var/spool/clientmqueue exist, also has the following permission and ownership  

 # ls -ail /var/spool/clientmqueue 

drwxrwx—   2 smmsp    smmsp               /var/spool/clientmqueue 

#/usr/bin/chmod 770 /var/spool/clientmqueue 

#/usr/bin/chown smmsp:smmsp  /var/spool/client/mqueue 

                    One can use the -d44.5 switch on sendmail to debug permission problems: 

# /usr/lib/sendmail -v -d44.5 someone@somewhere.com < /etc/hosts 

For further details see   1001806.1 “Debugging Sendmail Permission Issues”. 

Step 8. Check if your DNS infrastructure is  working properly. 

 Is system configured as a DNS client:

Solaris 8, 9 and 10: 

/etc/nsswitch.conf  “hosts:” line contains “dns” keyword
/etc/resolv.conf exists with a minimum configuration of at least 2 (3 preferred) nameserver IP addresses 

To check for a mail exchanger (MX) record for a domain:

  

using nslookup 

  

# nslookup (type this command and return)
Default Server: nameserver.somedomain.COM
Address: 129.168.1.2
> set (type this command and return)
> yahoo.com (type in the domain in question, results follow)
Server: nameserver.somedomain.COM
Address: 129.168.1.2using dig
(“ANSWER” greater than 0 indicates # of records found)
 

Non-authoritative answer: yahoo.com preference = 5, mail exchanger = mx1.yahoo.com
yahoo.com preference = 5, mail exchanger = mx2.yahoo.com
yahoo.com preference = 5, mail exchanger = mx3.yahoo.com
yahoo.com preference = 5, mail exchanger = mx4.yahoo.com 

Authoritative answers can be found from:
yahoo.com nameserver = ns1.yh.net
yahoo.com nameserver = ns2.yh.net
yahoo.com nameserver = ns3.yh.net
yahoo.com nameserver = ns4.yh.net
yahoo.com nameserver = ns5.yh.net
mx1.yahoo.com internet address = 65.54.244.8
mx1.yahoo.com internet address = 65.54.245.8
mx1.yahoo.com internet address = 65.54.244.136
mx2.yahoo.com internet address = 65.54.244.168
mx2.yahoo.com internet address = 65.54.244.40
mx2.yahoo.com internet address = 65.54.245.40
mx3.yahoo.com internet address = 65.54.244.200
mx3.yahoo.com internet address = 65.54.244.72
mx3.yahoo.com internet address = 65.54.245.72
mx4.yahoo.com internet address = 65.54.245.104
mx4.yahoo.com internet address = 65.54.244.104
mx4.yahoo.com internet address = 65.54.244.232
ns1.yh.net internet address = 207.68.160.190
ns2.yh.net internet address = 65.54.240.126
ns3.yh.net internet address = 213.199.161.77
ns4.yh.net internet address = 207.46.66.126
ns5.yh.net internet address = 65.55.238.126 

  

# dig -t mx yahoo.com 

; <<>> DiG 8.3 <<>> -t yahoo.com
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 5, ADDITIONAL: 17
;; QUERY SECTION: 

;; yahoo.com, type = MX, class = IN 

;; ANSWER SECTION: 

yahoo.com. 47m34s IN MX 5 mx2.yahoo.com.
yahoo.com. 47m34s IN MX 5 mx3.yahoo.com.
yahoo.com. 47m34s IN MX 5 mx4.yahoo.com.
yahoo.com. 47m34s IN MX 5 mx1.yahoo.com. 

;; AUTHORITY SECTION: 

yahoo.com. 6h4m35s IN NS ns1.yh.net.
yahoo.com. 6h4m35s IN NS ns2.yh.net.
yahoo.com. 6h4m35s IN NS ns3.yh.net.
yahoo.com. 6h4m35s IN NS ns4.yh.net.
yahoo.com. 6h4m35s IN NS ns5.yh.net. 

;; ADDITIONAL SECTION: 

mx2.yahoo.com. 47m34s IN A 65.54.244.168
mx2.yahoo.com. 47m34s IN A 65.54.244.40
mx2.yahoo.com. 47m34s IN A 65.54.245.40
mx3.yahoo.com. 47m34s IN A 65.54.244.200
mx3.yahoo.com. 47m34s IN A 65.54.244.72
mx3.yahoo.com. 47m34s IN A 65.54.245.72
mx4.yahoo.com. 47m34s IN A 65.54.245.104
mx4.yahoo.com. 47m34s IN A 65.54.244.104
mx4.yahoo.com. 47m34s IN A 65.54.244.232
mx1.yahoo.com. 47m34s IN A 65.54.244.8
mx1.yahoo.com. 47m34s IN A 65.54.245.8
mx1.yahoo.com. 47m34s IN A 65.54.244.136
ns1.yh.net. 1d1h41m34s IN A 207.68.160.190
ns2.yh.net. 7m13s IN A 65.54.240.126
ns3.yh.net. 7m13s IN A 213.199.161.77
ns4.yh.net. 7m13s IN A 207.46.66.126
ns5.yh.net. 7m13s IN A 65.55.238.126 

;; Total query time: 62 msec 

;; FROM: solarishost to SERVER: default — 129.168.1.2
;; WHEN: Thu Apr 3 15:34:41 2008
;; MSG SIZE sent: 29 rcvd: 479< 

If an MX record is not found: 

Doublecheck with DNS adminstrator if one should exist and if so, query it directly using the hostname and address supplied by your DNS admin and use the command: 

nslookup <hostname or IP address> 

dig <hostname or IP address> 

getent hosts <hostname or IP address> 

using getent
(when using getent, a database must be supplied i.e. hosts for name resolution queries. Since getent does not query the DNS name servers directly, use the argument “mailhost”)

root# getent hosts mailhost
129.148.9.192 mailhost.foo.com
129.148.13.5 mailhost.foo.com 

if getent cannot find a match for mailhost it will return the prompt without output: 

root# getent hosts mailhost
root# 

using /usr/lib/sendmail -d 

to output sendmail version; compiled with; OS Defines; Kernel symbols; Conf file; Pid File; Canonical name; UUCP nodename; alias names; system identity and total requests in /var/spool/mqueue: 

 root# /usr/lib/sendmail -d0.11 -bp 

Version 8.13.8+Sun 

Compiled with: DNSMAP LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB NIS NISPLUS PIPELINING SCANF STARTTLS TCPWRAPPERS USERDB USE_LDAP_INIT XDEBUG 

OS Defines: HASCLOSEFROM HASFCHOWN HASFCHMOD HASFDWALK HASGETUSERSHELL HASINITGROUPS HASLDAPGETALIASBYNAME HASLSTAT HASNICE HASRANDOM HASRRESVPORT HASSETREGID HASSETREUID HASSETRLIMIT HASSETSID HASSETVBUF HASURANDOMDEV HASSTRERROR HASULIMIT HASUNAME HASUNSETENV HASWAITPID IDENTPROTO IP_SRCROUTE SAFENFSPATHCONF SYS5SETPGRP SYSTEM5 USE_DOUBLE_FORK USE_SA_SIGACTION USE_SIGLONGJMP USESETEUID 

Kernel symbols: /dev/ksyms
Conf file: /etc/mail/submit.cf (default for MSP)
Conf file: /etc/mail/sendmail.cf (default for MTA)
Pid file: /var/run/sendmail.pid (default)
Canonical name: solarishost.nisplus.com
UUCP nodename: solarishost
a.k.a.: solarishost.nisplus.com
a.k.a.: [10.10.11.88]
a.k.a.: [127.0.0.1]
a.k.a.: loghost
Conf file: /etc/mail/sendmail.cf (selected)
Pid file: /var/run/sendmail.pid (selected) 

============ SYSTEM IDENTITY (after readcf) ============ 

(short domain name) $w = solarishost
(canonical domain name) $j = solarishost.nisplus.com
(subdomain name) $m = nisplus.com
(node name) $k = solarishost 

======================================================== 

/var/spool/mqueue is empty
Total requests: 0 

using /usr/lib/sendmail -bt test mode: 

To query a domain for mailserver (MX) records: 

root@netlab88# /usr/lib/sendmail -bt 

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked) 

Enter <ruleset> <address> 

> /mx yahoo.com [type in /mx domain2query.com and return] 

getmxrr(yahoo.com) returns 7 value(s): 

c.mx.mail.yahoo.com.
f.mx.mail.yahoo.com.
b.mx.mail.yahoo.com.
a.mx.mail.yahoo.com.
e.mx.mail.yahoo.com.
d.mx.mail.yahoo.com.
g.mx.mail.yahoo.com. 

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

2 Responses

  1. Venkat says:

    good explanation Ramdev.. Thank u very much 

  1. September 16, 2015

    […] Read – Troubleshooting Guide – 2 […]

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us