troubleshooting linux application response issues

This is a stub for common problems in unix and ways to troubleshoot them.

Problem 1: Application that was working now responds with "Service Unavailable" or similar error

Common troubleshooting steps:
a) check status of all application. If this is a service you can do
service app status
b) Check app and system logs
check /var/log/messages
check other /var locations like /var/log/httpd
c) check if the disks are full
df -h
d) if a disk is full check if the application is debugging at a high level
e) if someone's home directory is full then, verify who's
du -ks /home/devel/* /home/admin/* sort -n tail -10
The above command lists the top 10 offenders. Send them an email

Problem 2: Application responds properly but drops a few messages once every 100,000 actions

Common troubleshooting steps:
a) check if packets were lost because interface was configured incorrectly
/sbin/ifconfig -a
This shows which interface has dropped messages, let's say it is eth0
run ethtool on eth0 to find if it is not set to duplex



Problem 3: I can send a message and I see the application responded in their logs but the sender never receives the response

use netstat -nr to and verify that the message comes in and goes out on the same interface
use tcpdump also to confirm the message is leaving the right interface

Problem 4: message does not even get to the box

use traceroute to see how the message is getting routed
use ping to see if the host is responding

Problem 5: I cannot ping a box
check if icmp echo is disabled