What the Siteimprove Crawler Can and Cannot Crawl

Summary

The Siteimprove crawler scans accessible HTML-based content but may not capture restricted, embedded, or unsupported elements like certain IFrames.

Overview

This article defines crawler capabilities and limitations, including supported content types and known exclusions.

What content can Siteimprove's Crawlers crawl?

• HTML
• XML
• All non-scripted content
• Scripted content (such as JavaScript & AJAX)*
• Dynamically loaded content (written text in images, videos, etc.)*

*This type of content is scanned if the JavaScript crawler is enabled.

What content can Siteimprove's Crawlers not crawl?

• Online shops (such as Shopping Carts)
• Payment verification
• Content requiring interaction to be available, such as pages only available if searched for, and forms depending on fields being filled in.
• Software products/apps

For more information on this, see the article, Can Siteimprove crawl Single Page Applications (SPAs) and forms?

Why can't I find information on IFrames?

Our page reports do not include content within iframes, as we do not crawl or analyse iframe-embedded content. We'll identify when there is an iframe on the page and report that, but generally we don't report on the content within the iframe.

Since iframe content is hosted separately, it is not crawled by our services. Consequently, any information contained solely within iframes will not be detected or evaluated by our analysis.

Key Concepts

• Crawlable vs non-crawlable content
• Rendering limitations
• Embedded content visibility