Understanding Crawl Scheduling and Frequency

Summary

Crawl scheduling and frequency determine when and how often Siteimprove scans your website. The platform uses defined intervals, queueing logic, and system constraints to control crawl timing and ensure efficient processing.

Overview

Siteimprove automatically crawls your website to analyze content, accessibility, and quality. Crawl scheduling controls when crawls occur, while crawl frequency determines how often they run.

Because crawling is resource-intensive, the platform uses structured rules to manage timing, prevent overlap, and maintain consistent performance.

Key concepts

What is a crawl?

A crawl is an automated scan of your website that collects data about pages, content, and issues.

What is crawl frequency?

Crawl frequency defines how often a site is scanned, based on a set interval (for example, every 7 days).

What is a crawl interval?

A crawl interval is the minimum time between crawls for a site. This ensures that scans are spaced appropriately and do not overlap.

How crawl scheduling works

Siteimprove schedules crawls using a combination of:

• Defined crawl intervals
• System capacity and availability
• A queue that manages crawl order

Crawls are not triggered instantly. Instead, they are placed in a queue and processed when system resources allow.

System rules and constraints

The crawl system operates under the following rules:

• Only one crawl per site can run at a time
• Crawls cannot overlap
• Each site must follow its defined crawl interval
• The minimum crawl interval is 7 days
• Crawls are processed in a queue based on scheduled timing

Queue behavior

When multiple crawls are scheduled:

• Crawls are added to a shared processing queue
• Execution order is determined by scheduled start time and availability
• Delays can occur if other crawls are already in progress

This means a crawl may not start exactly at its scheduled time but will run as soon as capacity allows.

Why crawl timing may vary

Crawl timing can vary due to:

• Other crawls already running
• System load and capacity
• Queue prioritization

This is expected and ensures that all crawls complete successfully without impacting platform performance.

Why this matters

Understanding crawl scheduling helps you:

• Set realistic expectations for when data updates
• Interpret delays or timing differences correctly
• Plan content updates around crawl cycles

It ensures you can accurately understand when changes to your site will be detected and reported.

Result

Crawl scheduling ensures that your website is scanned consistently, efficiently, and without conflict, providing reliable data while maintaining system performance.

Additional information

If you need help choosing the right crawl setup for your site, contact Siteimprove technical support or consult your Customer Success Manager.