What can I do if I’m over my Siteimprove page limit?

Summary

If your website has more pages than your Siteimprove page limit allows, there are several ways to reduce the number of pages being crawled before considering an increase to your page limit.

How to reduce the number of crawled pages

Identify what is increasing your page count

If your page count has increased unexpectedly, use Check History to identify newly discovered pages or sections that may be contributing to the increase. For more information, see How do I know which pages have been added or removed with the latest crawl?.

Look particularly for new sections, subdomains, duplicate pages or automatically generated URLs.

Exclude unnecessary content

Use Site Content Settings to exclude pages or sections that do not need to be monitored.

This can be particularly useful for:

• Duplicate or near-duplicate pages
• Old or archived content
• Test, development or legacy sections
• Internal search results
• Unnecessary directories or sub-sites

Where possible, exclude an entire section using a specific URL match rather than excluding individual pages.

Tip: Use specific URL matches when configuring Site Content Settings. Broad matches can unintentionally include or exclude more content than intended.

Review your robots.txt rules

Review your robots.txt file to make sure Siteimprove is not being allowed to crawl sections that you do not need to monitor.

Siteimprove respects robots.txt rules when crawling. If you use Siteimprove-specific rules, make sure they use the appropriate Siteimprove user agents and conditions.

See Update to the robots.txt parser for information about how Siteimprove interprets robots.txt.

Important: Changes to robots.txt can affect other crawlers as well. Make sure any changes are appropriate for your overall website crawling requirements.

Restrict URL parameters and dynamically generated pages

URL parameters can create a large number of additional URLs for the same underlying content.

Common examples include:

• Search URLs
• Tracking parameters
• Filtering and sorting parameters
• Faceted navigation
• Other dynamically generated URL variations

If these URLs do not need to be monitored individually, use Site Content Settings to exclude the relevant URL patterns.

Remove old or unused website content

Review old, archived, test and legacy content that is still publicly accessible.

If the content is no longer needed, consider removing it from your website. If it is still required but does not need to be monitored in Siteimprove, exclude it from the crawl.

Remove unused sites

Review the sites configured in your Siteimprove account and remove sites that are no longer needed, such as decommissioned websites, old domains or temporary sites.

Make sure you no longer need the site's historical data or reports before removing it.

Still over your page limit?

If you have reduced your crawl scope and still need to monitor more pages than your current limit allows, contact Siteimprove Support or your Siteimprove representative to discuss your options.

When contacting Support, provide your current page usage, page limit and any changes you have already made to reduce the number of pages being crawled.