Why is my website crawl taking a long time?

Your website is responding slowly, therefore, our crawler has automatically slowed down so as not to overload the website.

Possible Solutions

• Exclude any areas of the website that you do not want to be checked.
• See: How to add and remove content from a crawl

A large amount of PDFs on the site

The time it takes to process a PDF is comparable to processing a web page. Having many PDFs is similar to crawling a site with lots of web pages.

Siteimprove needs time to check the PDFs and provide you with check results.

Possible Solutions

• If the PDF checks are not a priority (e.g. archived data) you can use settings to mark them as external to reduce the crawl time.
• See: Site Content Settings.

The login credentials provided to Siteimprove, for a website behind a login, have expired.

Possible Solutions

• Provide Siteimprove Technical Support with the new login credentials for the website.

The crawler has found a new section on the website and needs to be re-configured accordingly, e.g. a new forum, a web-shop with duplicate pages, etc.

Possible Solutions

• Exclude any areas of the website that you do not want to be checked.
• See: How to add and remove content from a crawl
• Open a ticket to ask Technical Support to review the crawler settings and configure the crawler to reduce duplicate pages or exclude unwanted areas of the website from the crawl.

The IP address for the crawler has been blocked by your firewall.

Possible Solutions

• Ensure that Siteimprove Crawler IP addresses can access the website.