How to exclude crawler traffic from Analytics

In some cases, the crawler traffic (also referred to as a robot, bot, spider) may be registered within your website analytics solution. This article explains how to remove links with traffic and:

• Update traffic from Siteimprove Analytics using Data Settings
• Hide traffic from Siteimprove Analytics using filters
• Remove links from Google Analytics

From January 2020 analytics traffic generated by the Siteimprove Crawler will be automatically removed from Siteimprove Analytics.

Data Settings in Siteimprove

The easiest way to update unwanted crawler traffic from Siteimprove Analytics is to use Site Content Settings. Data settings are available for users with an Account Owner or Administrator role.

Note: Know crawler and bot data will process links from the time you enable Site Content Settings (i.e. not retroactively). You will not be able to retrieve this data afterward. However, you can remove the links if you want to start collecting the data again.

• Go to Quality Assurance > Inventory > Tracking > Links

Here you have the option to remove links from crawler traffic with known crawlers and bots. The bots and crawlers removed are defined using IAB Spiders & Bots list which is considered an industry standard. 

You also have the option remove links to traffic based on specific IP groups using the setting specific IP groups dropdown.  You can find out more about creating IP groups in the article, “How to create an IP group for filtering in Analytics”.

It is possible to remove links for bots and crawlers, and/or specific IP group from all site (default) or specifically selected sites on your account. Select the "Specific Sites" tab under to configure settings for specific sites. 

Hide traffic from Siteimprove Analytics using filters

In Siteimprove Analytics, unwanted traffic can be identified and filtered using either an Organizational or IP filter. The following explains how to set up a filter in each case.

Filter Siteimprove traffic using an Organizational filter

• Go to Settings > Analytics > Filters and select “New filter”
• Give the filter a name and choose if others can use it
• Click “Add filter element” and then “Organization”
• Select to filter by “Visitors not from this organization”
• Enter a match for organizations where the name contains Siteimprove
• Click on “Add match condition”
• You can use the “Test Now” button to test the filter
• Click on “Create filter” to save

Filter Siteimprove traffic using an IP filter

Some users may prefer to filter traffic using an IP filter. To do so, follow the instructions in this article: “How to set up an IP Filter”.

When creating the IP group for your filter use IP address listed in the Content Suite section of this article: “What IP addresses are used by Siteimprove?”

Enabling the filter

Once created, you can enable the filter using the filter dropdown on analytics pages within the platform.

Filters can also be incorporated within reports by selecting the filter when scheduling the report.

Note: You can also use filters to remove spam from your statistics. For further information on identifying spam crawler traffic see: How to spot spam in Analytics and what to do about it.

Update Siteimprove traffic from Google Analytics

In order to remove links in Siteimprove traffic from Google Analytics, you will need to create an account filter in Google Analytics and update the Siteimprove public-facing IPs addresses.

For further information see the following:

• What IP addresses and User agents are used by Siteimprove?
• Google Analytics article describing how to create an IP address filter.

Google filters will only be applied from the day it’s created.

Note: In some cases, you may see statistics for domains you do not wish to track in analytics. We suggest removing the Siteimprove Analytics JavaScript from those domains/subdomains. If this is not an option you can mark the domains as external within Analytics settings. For more information on this see: What are my options when unwanted domains show up in my Analytics statistics?

If you have any questions, please contact Siteimprove Technical Support with your request.