Website / Hosting monitor

 

This was another small project for my Raspberry Pi.

For some time I’ve wanted to keep tabs automatically on this site and to get email alerts if it stopped working for any reason as has happened a few times in the past. After playing around with setting up Lighttpd, MySql, and PHP on my Pi I decided to use it for this. As there is no SMTP mail on the Pi by default I had to set that up which was simple enough with the aid of google. I set up a gmail account specifically for serving the alert emails and after seeing https://support.google.com/mail/answer/78754 in the syslog I had to reduce login security to keep it working. I won’t cover the details of any of this Pi setup as it’s already well documented on the web – if it weren’t I wouldn’t have been able to do it.

If you have access to only one server and can’t use a machine at home as one then this project could be put on a free host such as www.000webhost.com/ but you’d need to check it offered support for the features use such as CURL and SMTP mail. Some free hosts may not give you access to cron scheduling (000wehost says they do) but you could run it on a browser page from home using the same technique as in www.digitalham.co.uk/dynamic-dns-woes/ – ie a meta-refresh in the header to make the script run every 5 minutes <meta http-equiv='refresh' content='900'>.



I decided to use CURL (not on the Pi by default) to access websites and internal PHP timing to measure the download time.

Here are the commands you’ll need to install CURL and restart the lighttpd webserver.

sudo apt-get install php5-curl (assuming you run PHP 5)
sudo /etc/init.d/lighttpd restart

Rather than just send email alerts when things went wrong I decided to store all results in a simple MySQL table. Here is the table definition…

CREATE TABLE IF NOT EXISTS `monitoring` (
`recno` int(10) unsigned zerofill NOT NULL AUTO_INCREMENT,
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`url` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`load_time` decimal(4,3) NOT NULL,
`curl` varchar(240) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`http` varchar(6) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL DEFAULT '0',
`datasize` mediumint(9) NOT NULL DEFAULT '0',
PRIMARY KEY (`recno`),
KEY `timestamp` (`timestamp`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

By monitoring another site on the same server as digitaham, a site on a different host entirely, and google I can look at what sort of results all those got when there is an error and get an idea of the answers to – is my server down? running slowly? or maybe just my home internet performing badly.

I used sudo crontab -e to run the script every 5 minutes

*/5 * * * * /usr/bin/php /var/www/sitechk.php > /dev/null

I found on my Pi 2 using the NOOBS operating system that came with the kit I bought that the path to PHP was different and the crontab entry on that became
/usr/lib/cgi-bin/php /var/www/sitechk.php

Make sure that the script has the execute privilege for world by either using your ftp program or chmod to 755.

The data collection script
It stores the results, emails error alerts, prunes old entries, and mostly for testing does simple echo reporting as it runs.

<?php
// connect to MySQL
$link = @mysql_connect('localhost','db-user','password');
@mysql_select_db('db-name');

function Visit($url,$down,$timeout)
 {
  global $down;
  $down = "$url";
  echo "Checking $url Timeout: $timeout ms<br>";
  $agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
  $ch=curl_init();
  curl_setopt($ch,CURLOPT_URL,$url );
  curl_setopt($ch,CURLOPT_USERAGENT, $agent);
  curl_setopt($ch,CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch,CURLOPT_VERBOSE,false);
  curl_setopt($ch,CURLOPT_ENCODING,"");
  curl_setopt($ch,CURLOPT_TIMEOUT_MS, $timeout);
  curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, FALSE);
  curl_setopt($ch,CURLOPT_SSLVERSION,3);
  curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, FALSE);

  $time = microtime();
  $time = explode(' ', $time);
  $time = $time[1] + $time[0];
  $start = $time;

  $page=curl_exec($ch);

  $time = microtime();
  $time = explode(' ', $time);
  $time = $time[1] + $time[0];
  $finish = $time;
  $load_time = round(($finish - $start), 3);

  if (curl_error($ch))
   {
    $curl = curl_error($ch);
    $down .= " CURL: $curl";
    $httpcode = "";
    $datasize = 0;
    curl_close($ch);
    $add_query = "INSERT INTO monitoring (url,load_time,curl,http) VALUES('$url','$load_time','$curl','$httpcode')";
    $result = mysql_query($add_query);
    echo "Request time:$load_time seconds CURL:$curl<br><br>";
    return;
   }
  else
   {
    $curl     = "";
    $info     = curl_getinfo($ch);
    $httpcode = $info['http_code'];
    $datasize = $info['size_download'];
   }

  curl_close($ch);

  $add_query = "INSERT INTO monitoring (url,load_time,curl,http,datasize) VALUES('$url','$load_time','$curl','$httpcode','$datasize')";
  $result = mysql_query($add_query);

  echo "Request time:$load_time seconds HTTP code :$httpcode <br>Data size:$datasize<br><br>";
  if ($httpcode >= 400 || $httpcode < 200) $down .= " HTTP: $httpcode Data size:$datasize";
  else $down = "";
  return;
 }

echo "Running SiteChk<br><br>";
$down = "";
$mail = ""
;
Visit("http://www.site1.co.uk",$down,"5000");
if ($down) $mail .= "$down\n";

Visit("http://www.site2.co.uk",$down,"5000");
if ($down) $mail .= "$down\n";

Visit("http://www.digitalham.co.uk",$down,"5000");
if ($down) $mail .= "$down\n";

Visit("http://www.google.co.uk",$down,"5000");
if ($down) $mail .= "$down\n";

if ($mail)
{
 echo "email: $mail<br><br>";
 mail("you@yourdomain","Monitoring - problem encountered",$mail);
}

$maint_query = "DELETE FROM monitoring WHERE timestamp < ADDDATE( NOW( ) , -7 )";
$result = mysql_query($maint_query);

echo "Script completed";
?>

URL Error Report
The reporting script shown below was from my initial testing. I have since produced one with a better layout and more statistics. Feel free to download this zip of the latest versions of both scripts.

<?php
// connect to MySQL
$link = @mysql_connect('localhost','db-user','password');
mysql_select_db('db-name');
?>
<html>
<head>
  <title>URL Report</title>
  <meta http-equiv="imagetoolbar" content="no">
  <link rel="StyleSheet" href="monitoring.css">
</head>
<body>
<?php
if (isset($_POST['selurl'])) $selurl = $_POST['selurl'];
else $selurl = "";

$my_query = "SELECT DISTINCT url FROM monitoring order by url";
$url_list = mysql_query($my_query);

echo "<form action=\"site_report.php\" method=post>\n";
echo "<select name='selurl'>\n";
while ($option = mysql_fetch_array($url_list))
 {
  $url = stripslashes($option['url']);
  echo "<option value='$url'";
  if ($selurl == $url) echo " selected";
  echo ">";
  echo "$url\n";
 }
echo "</select>\n";
echo "<br><br><input type='submit' alt='Select URL' border=0 name='select' value='Select URL'>\n";
echo "</form>\n";

if ($selurl)
 {
  $my_query  = "SELECT AVG(load_time) AS average FROM monitoring where url = '$selurl';";
  $my_result = mysql_query($my_query);
  $my_row    = mysql_fetch_array($my_result);
  $average   = $my_row ['average'];
  echo "<br><strong>$selurl average load time: $average</strong><br>\n";

  $http = "";
  $curl = "";
  $hdr  = 0;
  $my_query  = "SELECT * FROM monitoring WHERE url = '$selurl' AND http <> 200 order by recno DESC LIMIT 0,30";
  $my_result = mysql_query($my_query);
  while ($my_row = mysql_fetch_array($my_result))
   {
    $timestamp = $my_row ['timestamp'];
    $curl      = $my_row ['curl'];
    $http      = $my_row ['http'];
    $datasize  = $my_row ['datasize'];
    if (!$hdr)
     {
      echo "<br><strong>$selurl Latest Errors Report</strong><br>\n";
      $hdr++;
     }
    if ($http == 200) echo "$timestamp HTTP code: $http Datasize: $datasize<br>\n";
    else if ($curl)   echo "$timestamp CURL: $curl<br>\n";
   }

  echo "<br><strong>$selurl Slowest load times</strong><br>\n";
  $my_query  = "SELECT * FROM monitoring WHERE url = '$selurl' AND http = 200 order by load_time DESC LIMIT 0,30";
  $my_result = mysql_query($my_query);
  while ($my_row = mysql_fetch_array($my_result))
   {
    $timestamp = $my_row ['timestamp'];
    $load_time = $my_row ['load_time'];
    $datasize  = $my_row ['datasize'];
    echo "$timestamp Datasize: $datasize Load time: $load_time<br>\n";
   }

  echo "<br><strong>$selurl Fastest load times</strong><br>\n";
  $my_query  = "SELECT * FROM monitoring WHERE url = '$selurl' AND http = 200 order by load_time ASC LIMIT 0,30";
  $my_result = mysql_query($my_query);
  while ($my_row = mysql_fetch_array($my_result))
   {
    $timestamp = $my_row ['timestamp'];
    $load_time = $my_row ['load_time'];
    $datasize  = $my_row ['datasize'];
    echo "$timestamp Datasize: $datasize Load time: $load_time<br>\n";
   }
 }
?>
</body>
</html>


 Leave a Reply

(required)

(required)