Why is my website crawl taking a long time?

Modified on: Mon, 24 Jan, 2022 at 11:11 AM

Aliases and exclusions: How to add and remove content from a crawlAliases and exclusions: How to add and remove content from a crawlAliases and exclusions: How to add and remove content from a crawlThere can be a number of reasons preventing a website crawl from ending.
Typical reasons and possible solutions are provided in the table below. 

Reason for IssuePossible Solutions

Your website is responding slowly, therefore, our crawler has automatically slowed down so as not to overload the website.



Exclude any areas of the website that you do not want to be checked.


See: How to add and remove content from a crawl


A large amount of PDFs on the site
The time it takes to process a PDF is comparable to processing a web page. Having many PDFs is similar to crawling a site with lots of web pages. Siteimprove needs time to check the PDFs and provide you with check results.
If the PDF checks are not a priority (e.g. archived data) you can use an alias to mark them as external to reduce the crawl time. See: Aliases and exclusions: How to add and remove content from a crawl.

The login credentials provided to Siteimprove, for a website behind a login, have expired.



Provide Siteimprove Technical Support with the new login credentials for the website.



The crawler has found a new section on the website and needs to be re-configured accordingly, e.g. a new forum, a web-shop with duplicate pages, etc.



Exclude any areas of the website that you do not want to be checked.


See: How to add and remove content from a crawl


Open a ticket to ask Technical Support to review the crawler settings and configure the crawler to reduce duplicate pages or exclude unwanted areas of the website from the crawl.



The IP address for the crawler has been blocked by your firewall.



Ensure that Siteimprove Crawler IP addresses can access the website.


Contact Siteimprove Technical Support if you continue to have issues with a website crawl not ending. 

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.