Website Monitoring: Best Practices

In theory, the default monitoring check settings should work in the majority of cases. However, everyday practice shows that our case does not belong to this majority. Therefore, it is necessary and worthwhile to dedicate some time to the optimal configuration of monitoring. It is also important to properly prepare the monitored website and the organization itself. Here are a couple of good practices in using website monitoring services, collected by a team that assisted clients in configuring thousands of monitoring checks and analyzed thousands of malfunctions.

Use a target URL, without any redirection

A website often works at multiple addresses. If it is well configured, all alternative addresses redirect to the main one. When setting up website monitoring, enter this primary (destination) address. Although monitoring can follow redirections, but it counts them, and when there are too many, reports the problem.

Use friendly names

Even if each of the checks you define will have a different (unique) URL, give the tests their own names. For example, “Article list – speed test”.

By having clear names, you will be able to recognize checks in your reports much easier, whereas long URLs will not needlessly prolong SMS alerts, which sometimes results in two messages having to be sent out.

Not only the homepage

Although checking the homepage itself is sufficient to detect the website’s unavailability, other features – such as loading speed – are also worth monitoring on selected subpages. It is best to choose one URL from each type of subpage, e.g. one category page, one product or article, etc.

Add monitoring to whitelist on firewall

To ensure that the connections made by the monitoring application are not being blocked, add all monitoring stations to the whitelist of your web server firewall.

Certain security features may automatically block IP addresses from which frequent and regular connections are made.

Add monitoring to the whitelist on the mail server

Like above, email messages sent by monitoring can also be blocked by the system for various reasons. To prevent this, add the email address of the sender – or preferably the entire domain of the monitoring application – to the whitelist on the mail server.

Exclude test traffic from statistics

Visits generated by basic monitoring checks – whether availability or content – are invisible to Google Analytics as well as similar statistical packages based on JavaScript.

However, connections generated by more advanced checks – the ones rendering the site to measure the time of loading or to test the performance of functions or processes – will already be recorded. In order to exclude them from traffic statistics, create a view containing a filter based on the “User-Agent” variable, usually containing the name of the browser, and in case of monitoring services, the name of the robot.

Verify the content of the monitored website

Basic monitoring of availability only checks the response of the server, i.e. verifies whether the website really works. How it works is another issue. Server can return the correct header “200 OK” while displaying an empty or completely scattered site.

To detect such situations too, use content monitoring. Define the text (a piece of content or a code, if needed), which will be checked on the site. Ideally, it should not be a piece of template, but the content coming from the database. Then the monitoring will also verify whether the database works properly.

Test complete processes

When monitoring a website that is more complicated than a blog or a company site, you want to verify whether individual processes are working correctly. A good example is the shopping process in an online store.
When you are creating a scenario of such check, try to make sure that the whole process is tested, or at least that the testing robot can proceed as far as possible.
For example, to ensure that the test of the shopping process does not stop on payments, generate a discount code for monitoring purposes, reducing prices by 100%.

Properly handle test tickets

In many cases, monitoring of forms and processes generates test tickets on the website – inquiries from the contact form, newsletter subscriptions, orders in the e-shop. To avoid wasting employees’ time on processing such requests, the website may include mechanisms that remove or ignore records containing e.g. email addresses used for testing forms.

Monitor expiration of domains and SSL certificates

Your domain registrar and SSL certificate issuer will certainly remind you in an automated way about the expiration of services. However, sometimes it may happen that messages sent by them will not reach you or will be missed. Having a second layer of protection against missed expiration in the form of external monitoring will come in handy.

Select the appropriate monitoring frequency

Availability monitoring tests should not be performed less frequently than every 1 minute. Such tests do not cause a significant load for the monitored server. Reducing the frequency, for example, to 5 minutes, would result in:

• errors lasting less than 5 minutes could be completely undetected,
• short, one-minute incidents would be recorded as 5-minute unavailability,
• the uptime in a given period would be incorrect.

Other types of checks – e.g. those that track the loading time of sites or test processes according to scenarios – do not need to work that often. And due to their impact on service performance (e.g. generating test orders) they should not even work that often. Depending on the needs, they can be performed every 10, 30 or 60 minutes.

Set limits appropriate to your website

If you observe frequent but short connection timeout incidents, you may want to increase the maximum time limit for your server response. This is not a good idea, as 5 seconds is a really long time for the server to respond. If it takes longer for your server to respond, there is probably something wrong with it.

On the other hand, when monitoring speed of a website, the time limit for loading a page must be set to a value slightly higher than the average time of loading. For example, if the page normally loads fully in about 3 seconds, set the limit to 5 seconds. You will get an alert when your page loads at least 2 seconds longer than usual.

Do not use excessive alerting

The scheme for notifications must also be set up according to the needs.

Unplanned unavailability of a key website must be reported to the relevant persons immediately, regardless of the time of day and through various channels. Alerts should ideally be sent to them every few minutes (but more than 1) until the malfunction is resolved.

In the case of less critical websites or checks, the notification scheme should be more lenient. It is possible to block text messages at night and on holidays, or even allow them to be sent only during business hours. A notification of failure detection and its resolution is enough. When such non-critical website often happens to have temporary unavailability (lasting one or two minutes), you can set the check to send an alert only if the malfunction lasts at least 3 minutes.

Configure the maintenance windows

Nearly every website has scheduled unavailability. At some point, you need to make updates, install modifications, and so on. Usually it takes place at night, when traffic is minimal and does not particularly affect users. But monitoring, by recording all malfunctions, will include scheduled unavailability in its reports as well as in its uptime calculations. There is no point in manually switching monitoring off and on. Instead, you have to define maintenance windows – one-time or cyclical – that is, the moments when monitoring will be temporarily suspended. This will prevent scheduled shutdowns from being included in the uptime of the site.

Prepare procedures in the organization

Errors happen and even the best monitoring system won’t change that. You have to expect that one day it will detect a real malfunction and alert you, therefore your company must be prepared for that. It’s about minimizing the time it takes from detecting the malfunction to resolving it. This requires procedures to ensure that everyone knows what to do in an emergency situation without wasting time.

Choose an annual subscription

There are two important reasons to opt for annual rather than monthly payments.
The first one – more obvious – is the price, which is simply lower for a yearly subscription.
The second reason is less obvious, but no less important. With a monthly subscription, you have 11 more chances per year to lose monitoring as your subscription expires – for example, because your credit card is overdue.

Ask for help with configuration

Website monitoring is a SaaS (Software as a Service) type service and, consequently, self-service. You have the ability to configure all aspects of monitoring yourself. However, in the case of more complicated checks – e.g. monitoring processes according to scenarios – it might be a good idea to ask the service provider’s employees for help. It’s not that you can’t handle it. Of course you can. They’ve just done it so many times and seen so many different cases that they’ll do it better. They know dozens of “good practices” like these.