Crawler Migration

Modified on: Thu, 1 Jul, 2021 at 5:33 PM

What is the Siteimprove crawler?

Siteimprove uses a crawler to evaluate the contents of our customers' websites. The crawler has a starting point (index URL), which is typically the home page. Following a list of instructions (for example domain matches, robots.txt, and exclusions), the crawler is able to regularly index websites and then evaluate for various issues (e.g. broken links and misspellings).

Why is Siteimprove introducing a new crawler?

The Siteimprove crawler has been updated, improved and tweaked over the course of the years, and in the past couple of years, we have been working on introducing a new crawler utilizing the newest technologies available. 

We are making this update primarily for the following reasons:

  • To accommodate new web technologies

  • To be able to crawl websites that rely heavily on Javascript

  • To enable our NextGen Accessibility product

What will change?

Siteimprove will migrate all accounts that are not already using the new crawler. 

In November 2019, we will start to migrate accounts in batches. The migration will happen throughout the year 2020.

How does this migration affect Siteimprove customers?

No action is required by our users, and we do not anticipate any issues with the migration. If we identify any fluctuations in pages, links, or in other areas, our Technical Support Team will handle these on a case by case basis. 

When will the change happen?

When your account has been selected for migration, Account Owners will receive a notification email two weeks before the migration itself. There will also be a notification within the platform when the migration is taking place.

UPDATE: The migration of sites to our new crawler has been completed in mid-December 2020. This article will be visible until the end of February 2021 for your information. Afterward, it will be taken offline.

Please contact Siteimprove Technical support if you have any questions regarding this. 

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.