Solaris Troubleshooting : Effective usage of ” Snoop ” utility
snoop” data is often required to analyze network issues. The snoop command captures packets from the network for direct or later analysis.
1. How to get snoop files?
1-1. Command option
Generally “text” file captured on the screen is not useful.
Use the options “-o” and “-d” to get all packets. “-d” specifies the device name and “-o” specifies the output filename.
# /usr/sbin/snoop -d <device> -o <filename>
For example:
# /usr/sbin/snoop -d ce0 -o hostA.ce0.ng.snoop
If the system is running Solaris 8 or later, also use “-q” option.
For example:
# /usr/sbin/snoop -q -d ce0 -o hostA.ce0.ng.snoop
-q When capturing network packets into a file, do not display the packet count. This can improve packet cap-tuning performance.
There may be several interfaces on the system. The “device” is the one which could have the issue. For example, if it’s “NFS” issue, then specify the device which should be used for the NFS connection. “netstat -rn” would be useful to confirm the device(Interface) name for the specific destination. If the several interfaces are used for the connections(ex. IPMP) then get snoop on the all of the interfaces.
For example:
# /usr/sbin/snoop -q -d ce0 -o hostA.ce0.ng.snoop
# /usr/sbin/snoop -q -d ce1 -o hostA.ce1.ng.snoop
It may be better that the “filename” includes the hostname/interface where the snoop was got, because this information is important to analyze the cause.The network may be very high load (ex. data backup via Gigabit Ethernet..) and it’s difficult to capture all of the packets. “snoop” could influence the network performance and could drop packets. We can check the packet drops by “-D” option.
-D Display number of packets dropped during capture on the summary line.
For example:
# /usr/sbin/snoop -i snoop.out -D | grep -v “drops: 0 “
…….
775 0.00001 Sender -> Receiver drops: 2800 FTP-DATA C port=45410
776 0.00001 Sender -> Receiver drops: 2800 FTP-DATA C port=45410
……. ^^^^^^^^^^^
In that case,”-s snaplen” could be useful.
-s snaplen Truncate each packet after snaplen bytes. Usually the whole packet is captured. This option is useful if only certain packet header information is required. The packet truncation is done within the kernel giving better utilization of the streams packet buffer. This means less chance of dropped packets due to buffer overflow during periods of high traffic. It also saves disk space when capturing large traces to a capture file. To capture only IP headers (no options) use a snaplen of 34. For UDP use 42, and for TCP use 54. You can capture RPC headers with a snaplen of 80 bytes. NFS headers can be captured in 120 bytes. The size of “snaplen” depends on the issue. It may be enough with only TCP/IP header information, and may need all of the actual TCP data information. Try this option after initial analysis was done.
Note that TCP header may have some options like “sack”,”timestamp”. The usage is like below.
For example:
# /usr/sbin/snoop -q -d ce0 -o hostA.ce0.ng.snoop -s 100
We can also focus on specific packets only by using “filter” like “hostname”,”port number” etc. This would be useful if the issue can be focused on specific connection only.
For example:
# /usr/sbin/snoop -q -d eri0 -o hostA.eri0.ng.snoop port 8080
For example:
# /usr/sbin/snoop -q -d eri0 -o hostA.eri0.ng.snoop 192.168.1.1 192.168.2.2
Please use filters only if you are sure that no information is missing. If not, capturing all packets is the best practice possible, because “filtering” packets may also “filter” the important packets.
1-2. When should we use snoop utility ?
While the issue is happening. And also try to get while the issue is NOT happening, if possible. The 2 snoop files are very useful to analyze issues if the connection is something like the customer’s application and we are not familiar with the expected behavior.We can focus on only the differences between the two snoop files.
2. Information that needed to be identified to analyse the the snoop files for troubleshooting purpose.
2-1. Network structure
For example: HostA — Switch — Firewall — Switch — HostB
2-2. When was the snoop file created?
For example: hostA.ce0.ng.snoop ==> During the issue.
For example: hostA.ce0.ok.snoop ==> During normal conditions.
2-3. Where was the snoop file created?
For example: hostA.ce0.ng.snoop ==> Got this on ce0 on hostA.
2-4. What snoop options were used?
For example: snoop -q -o <filename> -d <device> ==> (no filter)
For example: snoop -q -o <filename> -d <device> port 8080
2-5. What is the IP address which has the issue?
For example: 192.168.2.5
For example: Between 10.8.4.23 and 10.15.33.122
For example: Not sure.
Note: The IP-address is better than the hostname.
2-6. What is the service which has the issue?
For example: Not sure.
For example: LDAP
For example: NFS
For example: The customer’s application
For example: TCP port 2521
For example: http proxy port 8080
2-7. What is the issue?
For example: Cannot connect from HostA to HostB.
For example: ftp data transfer rate is slow.
For example: Application shows connection timeout.
3. Sample network problem and troubleshooting using snoop
Problem description : A customer’s application on HostA sends 100Mbytes of data to HostB. Usually it finishes within 5 seconds but sometimes takes over 60 seconds.
Network structure:
HostA —– Switch —- Router —– Switch —– HostB
ce0:192.168.1.1 hme0:192.168.2.2
application (port:5050)
100Mbyte data ===>
Data:
hostA.ce0.slow.snoop –> Got on hostA while the transmission is slow.
hostB.hme0.slow.snoop –> Got on hostB at the same time.
hostA.ce0.fast.snoop –> Got on hostA while the transmission is fast.
hostB.hme0.fast.snoop –> Got on hostB at the same time.
explorer.hostA.tar.Z –> explore of hostA
explorer.hostB.tar.Z –> explore of hostB
All snoop files were got without any filter. (just snoop -o file -d interface)
Hi Ram
Firstly thanks a lot for the info!!!
To snoop packets on a specific port:
# /usr/sbin/snoop -q -d eri0 -o hostA.eri0.ng.snoop port 8080
# /usr/sbin/snoop -q -d eri0 -o hostA.eri0.ng.snoop -p 8080
Both of these will work or only the 1st or 2nd ,can you pls help here