Crawler
In this section you can find answers to common questions relating to Crawler
Clean-up process for Single Page Checks
Fri, 21 Jan, 2022 at 4:49 PM
Can Siteimprove crawl an intranet and other non-public sites?
Wed, 18 May, 2022 at 10:25 AM
How would switching from a non-JS crawl to a JS crawl affect my data?
Mon, 25 Apr, 2022 at 9:54 AM
Siteimprove's Crawler: Frequently Asked Questions
Wed, 18 May, 2022 at 10:15 AM
Crawler queuing
Tue, 18 Jan, 2022 at 3:06 PM
Why are there already exclusions on my site?
Thu, 1 Jul, 2021 at 5:32 PM
Update to the robots.txt parser
Mon, 11 Apr, 2022 at 10:56 AM
Why is my website crawl taking a long time?
Mon, 24 Jan, 2022 at 11:11 AM
Typical Reasons for Crawl Problems
Mon, 30 Aug, 2021 at 5:01 PM
What does “not available” mean in the Scan History?
Mon, 21 Feb, 2022 at 1:05 PM