Automatically Restart Dead Services Via Bash Scripting

As a server administrator, there’s nothing that bugs me more than services that die all of a sudden. For a web server, an hour’s down time could mean losing a few hundred visitors. This could translate to a loss of advertising income too if you serve ads on your web pages.

I found a snippet of code that I modified to use as a simple service monitoring and restarting tool that I would like to share with all of you.

First off, let me state that I did not write the following script myself. It was adapted from a script I found somewhere (but I didn’t bookmark the source). Just create an empty file and enter the following lines in it:

#!/bin/bash

# Services Restarter - Automatically restart httpd and pure-ftpd if they die

/bin/netstat -ln | /bin/grep ":80 " | /usr/bin/wc -l | /bin/awk '{if ($1 == 0) system("/sbin/service httpd restart") }'
/bin/netstat -ln | /bin/grep ":21 " | /usr/bin/wc -l | /bin/awk '{if ($1 == 0) system("/sbin/service pure-ftpd start") }'

Now just save the file somewhere. As an example, I going to assume that you saved the file as /scripts/svcschk.

So what does the script do? Essentially, the script uses the netstat command line tool to see whether the server is listening on ports 80 (http service) and 21 (ftp service). If the server is not listening to the ports, then it will attempt to start the appropriate services.

Let’s chop the line that monitors the http service into smaller bits so we can understand what’s being done:

/bin/netstat -ln

Invokes the netstat command to list all listening ports and using presenting bound IP addresses in numeric format. The reason we’d want to use the numeric format for IPs is that it is much faster because we’re not resolving the IPs to hostnames.

| /bin/grep ":80 "

We then pipe the output of netstat to grep looking for addresses that are listening to port 80.

If you noticed, we actually checked for ":80 "; ie 80 with a space afterwards. The reason for the space is to filter out other services that run on ports 8000 or 8080 for example, which will be matched if we were to just use :80.

| /usr/bin/wc -l

Next, we pipe the output of grep to wc, which is essentially a command line word counter.

The -l flag used simply tells wc to count the total number of objects passed to it. Since we already grep-ed the relevant ports, this would return either 0 if the service has died or 1 if it’s alive and listening.

| /bin/awk '{if ($1 == 0) system("/sbin/service httpd restart") }'

Following that, we pass the output sent out by wc to awk, a fast text processing engine that can be programmed to do perform simple text processing jobs.

Here, we simply want awk to tell us if the argument passed to it was 0 (the number zero). If this is the case, then run /sbin/service httpd restart (the command to start the httpd service).

To make the script functional, you should set the appropriate ownership and permissions to it. For example: chown root:root /scripts/svcschk; chmod 700 /scripts/svcschk. This will set the ownership of the script to root and make it executable only to this user.

You can then set it to run every minute by adding an entry to your crontab similar to this:

*/1 * * * *    /scripts/svcschk > /dev/null 2>&1

Hope this code snippet and the explanations are useful to you. If you have any questions on them, feel free to ask me by commenting below.

3 responses to “Automatically Restart Dead Services Via Bash Scripting”.

  1. Foam Roofing Says:

    I just implemented your script–I hope it works! Thanks for taking the time to show how to do this simply. I’ve wasted time installing big enterprise level monitoring software that doesn’t even restart apache when it’s down, which seems like something critical to me… I also set up a cron job to restart apache every 10 minutes, but that also seems to not be working…

  2. Alessandro Says:

    You might want to try:

    # fuser 80/tcp && service httpd restart

    or anyway take advantage of fuser rather than netstat used that way.

    another approach is to monitor the process contained in the .pid file for those service, and restart if the process does not exist anymore.

    hope this helps
    Alessandro

  3. Deepak Says:

    G8 work
    realy g8 thanks for this