Quick Reference about Console Login Issues – Solaris 10 and Solaris 11

Solaris Console login on Solaris is mainly based on:

SMF service: svc:/system/console-login
process: /usr/lib/saf/ttymon

Basic Checks for Problem Diagnosis

1.Check the SMF service and logfiles

// Example for Solaris 10

# svcs -l ‘*console-login*’
fmri svc:/system/console-login:default
name Console login
enabled true
state online
next_state none
state_time 10 August 2013 21:20:51 MEST
logfile /var/svc/log/system-console-login:default.log
restarter svc:/system/svc/restarter:default
contract_id 160
dependency require_all/none svc:/system/filesystem/minimal (online)
dependency require_all/none svc:/system/identity:node (online)
dependency require_all/none svc:/system/utmp:default (online)
dependency require_all/none svc:/milestone/sysconfig (online)
dependency optional_all/none svc:/system/auditd (disabled)

// Example for Solaris 11

# svcs ‘*console-login*’
STATE STIME FMRI
disabled 5:48:33 svc:/system/console-login:terma
disabled 5:48:33 svc:/system/console-login:termb
disabled 5:50:09 svc:/system/console-login:vt2
disabled 5:50:09 svc:/system/console-login:vt3
disabled 5:50:09 svc:/system/console-login:vt4
disabled 5:50:09 svc:/system/console-login:vt5
disabled 5:50:09 svc:/system/console-login:vt6
online 6:14:38 svc:/system/console-login:default

# svcs -l svc:/system/console-login:default
fmri svc:/system/console-login:default
name Console login
enabled true
state online
next_state none
state_time October 18, 2013 06:14:38 AM BST
logfile /var/svc/log/system-console-login:default.log
restarter svc:/system/svc/restarter:default
contract_id 129
manifest /etc/svc/profile/site/sc_profile.xml
manifest /lib/svc/manifest/system/console-login.xml
manifest /lib/svc/manifest/milestone/config.xml
manifest /lib/svc/manifest/system/auditd.xml
manifest /lib/svc/manifest/system/console-login-termb.xml
manifest /lib/svc/manifest/system/console-login-vts.xml
manifest /lib/svc/manifest/system/console-login-terma.xml
dependency optional_all/none svc:/milestone/config (online)
dependency optional_all/none svc:/system/auditd (online)
dependency require_all/none svc:/system/filesystem/minimal (online)
dependency optional_all/none svc:/system/manifest-import (online)
dependency require_all/none svc:/system/identity:node (online)
dependency require_all/none svc:/system/utmp:default (online)
dependency require_all/none svc:/system/config-user:default (online)
dependency require_all/none svc:/milestone/config (online)
dependency require_all/none svc:/milestone/self-assembly-complete (online)

2.Check the runlevel / SMF milestone of the system

Ensure the system is at SMF milestone ‘all’

3.In case nobody is logged in on console, process /usr/lib/saf/ttymon is expected to run

# ps -ef | egrep tty
root 5672 9 0 11:14:49 console 0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p v490

4.Check for orphaned login processes

// example of an orphaned process on console or hanging console login process

$ ps -ef | egrep ” console “
root 10620 9 0 Oct 01 console 0:00 /usr/bin/login
root 809 10620 0 Nov 11 console 0:00 -sh

// expected process on device ‘console’ in case nobody is logged onto console

# ps -ef | egrep ” console “
root 1261 11 0 22:02:00 console 0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p mera

// expected process on device ‘console’ after successful login onto console

# ps -ef | egrep ” console “
root 1261 11 0 22:02:00 console 0:00 -sh

# ptree 1261
11 /lib/svc/bin/svc.startd
1261 -sh

5.Check logfiles of SMF service system-console-login

# svcs -xv console-login
svc:/system/console-login:default (Console login)
State: online since Thu May 09 15:40:08 2013
See: man -M /usr/share/man -s 1M ttymon
See: /var/svc/log/system-console-login:default.log
Impact: None.

# tail /var/svc/log/system-console-login:default.log
[ May 9 15:29:27 Enabled. ]
[ May 9 15:29:48 Executing start method (“/lib/svc/method/console-login”) ]
[ May 9 15:34:56 Stopping because all processes in service exited. ]
[ May 9 15:34:56 Executing start method (“/lib/svc/method/console-login”) ]
[ May 9 15:40:08 Stopping because all processes in service exited. ]
[ May 9 15:40:08 Executing start method (“/lib/svc/method/console-login”) ]

6.Check /etc/pam.conf for required entries.

# login service (explicit because of pam_dial_auth)
#
login auth requisite pam_authtok_get.so.1
login auth required pam_dhkeys.so.1
login auth required pam_unix_cred.so.1
login auth binding pam_unix_auth.so.1 server_policy
login auth required pam_ldap.so.1 use_first_pass
login auth required pam_dial_auth.so.1
See also:
Logins via SSH or on the console fail with “pam_unix_cred: cannot set user audit Bad address” (Doc ID 1566541.1)

7. Check if a ‘sulogin’ process is running

sulogin – access single-user mode
DESCRIPTION : The sulogin utility is automatically invoked by init when the system is first started.
# ps -ef | grep sulogin
root 5329 7 0 17:00:08 console 0:00 sulogin
root 5410 5114 0 17:23:14 pts/1 0:00 grep sulogin
# pkill sulogin
# svcs -a | grep console
online 17:27:39 svc:/system/console-login:default
#
# ps -ef | grep ttymon | grep console
root 5418 7 0 17:27:39 console 0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -T sun -m ldterm,ttcompat -h

8. Reset the terminal or putty session and try again

Check for console on ILOM (SP) or RSC

1.Check eeprom settings for console on ILOM (SP)

# eeprom | egrep consol
output-device=virtual-console
input-device=virtual-console

2.Check eeprom settings for console on RSC

# eeprom | egrep console
output-device=rsc-console
input-device=rsc-console
diag-out-console=false

3.Check for input/output problems of console on ILOM/RSC

1) return to ILOM or RSC
2) check input/output on ILOM or RSC
3) in case input/output on ILOM or RSC is working fine, stop and restart console

 

In depth Troubleshooting

 

1.Check /usr/lib/saf/ttymon process with truss

# ps -ef | egrep tty
root 5672 9 0 11:14:49 console 0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p v490

# truss -afeld -vall -o /var/tmp/ttymon.truss -p 5672

Base time stamp: 1381400113.2363 [ Thu Oct 10 11:15:13 BST 2013 ]
5672/1: psargs: /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttco
5672/1: pollsys(0xFFBFFC48, 1, 0x00000000, 0x00000000) (sleeping…)
5672/1: fd=0 ev=POLLIN rev=POLLPRI
5672/1: 7.9188 pollsys(0xFFBFFC48, 1, 0x00000000, 0x00000000) = 1
5672/1: fd=0 ev=POLLIN rev=POLLIN
5672/1: 7.9190 ioctl(0, I_PEEK, 0x00023840) = 1
5672/1: ctl: maxlen=1 len=-1 buf=0x0002385C
5672/1: dat: maxlen=255 len=5 buf=0x0002385C: ” r o o t\n”
5672/1: flags=0
5672/1: 7.9191 ioctl(0, STGET, 0xFFBFFC04) Err#22 EINVAL
5672/1: 7.9192 ioctl(0, TCGETS, 0xFFBFFC14) = 0
5672/1: iflag=0022402 oflag=0014005 cflag=03206275 lflag=0105073
5672/1: cc: 003 034 177 025 004 000 000 000
5672/1: 021 023 032 031 022 017 027 026 000 000 000
5672/1: 7.9193 ioctl(0, TCGETX, 0xFFBFFBF4) Err#22 EINVAL
5672/1: 7.9194 ioctl(0, TIOCGWINSZ, 0xFFBFFBEC) = 0
5672/1: row=0 col=0 xpixel=0 ypixel=0
5672/1: 7.9195 ioctl(0, TCSETSW, 0xFFBFFC14) = 0

— lines omitted —

2.Check system with DTrace

Reasonable scripts from DTraceToolkit are execsnoop and opensnoop. On a healthy system, the output looks like below:

DTraceToolkit-0.99/execsnoop -a
TIME STRTIME ZONE PROJ UID PID PPID ARGS
2550301297 2013 Oct 10 10:49:57 global 0 0 5423 9 /usr/bin/login
2553472587 2013 Oct 10 10:50:00 global 1 0 5439 5423 /bin/i386
2553474732 2013 Oct 10 10:50:00 global 1 0 5440 5423 /usr/sbin/quota
2553483601 2013 Oct 10 10:50:00 global 1 0 5441 5423 /bin/cat -s /etc/motd
2553485581 2013 Oct 10 10:50:00 global 1 0 5442 5423 /bin/mail -E
2553470374 2013 Oct 10 10:50:00 global 1 0 5423 9 -sh
#DTraceToolkit-0.99/opensnoop -a
TIME STRTIME UID PID FD ERR PATH ARGS

2052256250 2013 Oct 10 10:41:39 0 4775 5 0 /etc/shadow /usr/bin/login\0
2052256419 2013 Oct 10 10:41:39 0 4775 5 0 /dev/tty /usr/bin/login\0
2055420207 2013 Oct 10 10:41:42 0 4775 5 0 /etc/shadow /usr/bin/login\0
2055420262 2013 Oct 10 10:41:42 0 4775 5 0 /etc/passwd /usr/bin/login\0
2055420299 2013 Oct 10 10:41:42 0 4775 5 0 /etc/shadow /usr/bin/login\0
2055420651 2013 Oct 10 10:41:42 0 4775 -1 2 /etc/dialups /usr/bin/login\0
2055420729 2013 Oct 10 10:41:42 0 4775 5 0 /var/adm/lastlog /usr/bin/login\0
2055420827 2013 Oct 10 10:41:42 0 4775 5 0 /usr/lib/security/pam_roles.so.1 /usr/bin/login\0
2055421077 2013 Oct 10 10:41:42 0 4775 5 0 /usr/lib/security/pam_unix_account.so.1 /usr/bin/login\0
2055421496 2013 Oct 10 10:41:42 0 4775 5 0 /etc/passwd /usr/bin/login\0
2055421526 2013 Oct 10 10:41:42 0 4775 5 0 /etc/shadow /usr/bin/login\0
2055421560 2013 Oct 10 10:41:42 0 4775 5 0 /etc/shadow /usr/bin/login\0
2055421591 2013 Oct 10 10:41:42 0 4775 5 0 /var/adm/lastlog /usr/bin/login\0
2055421636 2013 Oct 10 10:41:42 0 4775 5 0 /etc/default/login /usr/bin/login\0
2055421739 2013 Oct 10 10:41:42 0 4775 5 0 /usr/lib/security/pam_unix_session.so.1 /usr/bin/login\0
2055421930 2013 Oct 10 10:41:42 0 4775 5 0 /var/adm/lastlog /usr/bin/login\0
2055428169 2013 Oct 10 10:41:42 0 4775 5 0 /var/adm/utmpx /usr/bin/login\0
2055428183 2013 Oct 10 10:41:42 0 4775 6 0 /var/adm/utmpx /usr/bin/login\0
2055428292 2013 Oct 10 10:41:42 0 4775 7 0 /etc/utmppipe /usr/bin/login\0
2055428333 2013 Oct 10 10:41:42 0 4775 5 0 /var/adm/wtmpx /usr/bin/login\0
2055428800 2013 Oct 10 10:41:42 0 4775 5 0 /etc/logindevperm /usr/bin/login\0
2055429410 2013 Oct 10 10:41:42 0 4775 8 0 /devices/pseudo/devinfo@0:devinfo /usr/bin/login\0
2055431058 2013 Oct 10 10:41:42 0 4775 8 0 /devices/pseudo/devinfo@0:devinfo /usr/bin/login\0
2055436208 2013 Oct 10 10:41:42 0 4775 5 0 /etc/default/nss /usr/bin/login\0
2055436342 2013 Oct 10 10:41:42 0 175 8 0 /etc/group /usr/sbin/nscd\0
2055436863 2013 Oct 10 10:41:42 0 4775 5 0 /dev/pool /usr/bin/login\0
2055437025 2013 Oct 10 10:41:42 0 4775 5 0 /etc/security/policy.conf /usr/bin/login\0
2055437921 2013 Oct 10 10:41:42 0 4775 5 0 /var/run/syslog_door /usr/bin/login\0
2055439007 2013 Oct 10 10:41:42 0 4775 -1 2 /var/ld/ld.config -sh\0
2055439037 2013 Oct 10 10:41:42 0 4775 3 0 /lib/libc.so.1 -sh\0
2055439841 2013 Oct 10 10:41:42 0 4775 3 0 /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 -sh\0
2055439963 2013 Oct 10 10:41:42 0 4775 3 0 /etc/profile -sh\0
2055442312 2013 Oct 10 10:41:42 0 4775 3 0 /lib/libgen.so.1 -sh\0
2055456824 2013 Oct 10 10:41:42 0 4775 -1 2 //.profile -sh\0
2055428405 2013 Oct 10 10:41:42 0 559 5 0 /proc/4775/psinfo /usr/lib/utmpd\0
2055441334 2013 Oct 10 10:41:42 0 4819 -1 2 /var/ld/ld.config /bin/i386\0
2055441366 2013 Oct 10 10:41:42 0 4819 3 0 /lib/libc.so.1 /bin/i386\0
2055454410 2013 Oct 10 10:41:42 0 4822 -1 2 /var/ld/ld.config /bin/mail -E\0
2055454446 2013 Oct 10 10:41:42 0 4822 3 0 /usr/lib/libmail.so.1 /bin/mail -E\0
2055454561 2013 Oct 10 10:41:42 0 4822 3 0 /lib/libsocket.so.1 /bin/mail -E\0
2055454659 2013 Oct 10 10:41:42 0 4822 3 0 /lib/libc.so.1 /bin/mail -E\0
2055454838 2013 Oct 10 10:41:42 0 4822 3 0 /lib/libnsl.so.1 /bin/mail -E\0
2055455764 2013 Oct 10 10:41:42 0 4822 3 0 /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 /bin/mail -E\0
2055456094 2013 Oct 10 10:41:42 0 4822 3 0 /proc/4822/psinfo /bin/mail -E\0
2055456168 2013 Oct 10 10:41:42 0 4822 3 0 /var/run/name_service_door /bin/mail -E\0
2055456411 2013 Oct 10 10:41:42 0 4822 -1 2 /var/mail/root /bin/mail -E\0
2055443383 2013 Oct 10 10:41:42 0 4820 -1 2 /var/ld/ld.config /usr/sbin/quota\0
2055443414 2013 Oct 10 10:41:42 0 4820 3 0 /lib/libnsl.so.1 /usr/sbin/quota\0
2055443572 2013 Oct 10 10:41:42 0 4820 3 0 /lib/libc.so.1 /usr/sbin/quota\0
2055444528 2013 Oct 10 10:41:42 0 4820 3 0 /platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1 /usr/sbin/quota\0
2055444624 2013 Oct 10 10:41:42 0 4820 3 0 /lib/libzfs.so /usr/sbin/quota\0
2055444775 2013 Oct 10 10:41:42 0 4820 3 0 /lib/libm.so.2 /usr/sbin/quota\0

3.Debug console-login start method

Add ‘set-x’ to second line or console-login start method. NOTE: !! please make a backup of the file before modifying it !!

# head /lib/svc/method/console-login
#!/sbin/sh
set -x
— lines omitted

Restart the console-login service

# svcadm restart svc:/system/console-login:default

Check the console-login svc logfile *before* and *after* login attempt to the console

# tail -100 /var/svc/log/system-console-login:default.log

[ Oct 10 11:03:45 Stopping because all processes in service exited. ]
[ Oct 10 11:03:45 Executing start method (“/lib/svc/method/console-login”) ]
[ Oct 10 11:09:02 Stopping because all processes in service exited. ]
[ Oct 10 11:09:02 Executing start method (“/lib/svc/method/console-login”) ]
FMRI=svc:/system/console-login
args=-g
+ svcprop -p ttymon/device svc:/system/console-login
val=/dev/console
+ [ -z /dev/console ]
args=-g -d /dev/console
+ getproparg -l ttymon/label
+ svcprop -p ttymon/label svc:/system/console-login
val=console
+ [ -n console ]
+ [ console != “” ]
+ echo -l console
args=-g -d /dev/console -l console
+ getproparg -T ttymon/terminal_type
+ svcprop -p ttymon/terminal_type svc:/system/console-login
val=””
+ [ -n “” ]
+ [ “” != “” ]
args=-g -d /dev/console -l console
+ getproparg -m ttymon/modules
+ svcprop -p ttymon/modules svc:/system/console-login
val=ldterm,ttcompat
+ [ -n ldterm,ttcompat ]
+ [ ldterm,ttcompat != “” ]
+ echo -m ldterm,ttcompat
args=-g -d /dev/console -l console -m ldterm,ttcompat
+ svcprop -p ttymon/nohangup svc:/system/console-login
val=true
+ [ true = true ]
args=-g -d /dev/console -l console -m ldterm,ttcompat -h
+ svcprop -p ttymon/timeout svc:/system/console-login
val=0
+ [ -n 0 -a 0 != 0 ]
+ svcprop -p ttymon/prompt svc:/system/console-login
val=\`uname\ -n\`\ console\ login:
+ [ -n \`uname\ -n\`\ console\ login: ]
+ eval echo \`uname\ -n\`\ console\ login:
+ echo `uname -n` console login:
prompt=`uname -n` console login:
+ eval echo `uname -n` console login:
+ uname -n
+ echo v490 console login:
+ exec /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p v490 console login:

Ramdev

Ramdev

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com. You can connect me at - https://www.linkedin.com/in/unixadminschool/

What is in your mind, about this post ? Leave a Reply

Close
  Our next learning article is ready, subscribe it in your email

What is your Learning Goal for Next Six Months ? Talk to us