Crawl errors: Crawl Blocked due to Server Overload or Instability

Issue / Question

You are receiving the following errors: 

• HTTP 429 Too Many Requests
• 500 Internal Server Error
• 503 Service Unavailable
• Rendering error

Environment (If Applicable)

Specify the conditions where the issue occurs:

• Product version: Siteimprove (all supported versions)
• Platform / OS / Browser: Platform‑independent (issue occurs at the site configuration and crawl level, not browser‑specific)
• User role / permissions:  Account Owners
• Preconditions: 

Solutions

If you are seeing a HTTP 429 Too Many Requests error

• Check with your hosting provider and review crawl-delay rules in your robots.txt file. Account Owners can change the starting point by using Edit site URL in the options listed under the How to fix this dropdown menu in the Site summary panel under Manage sites. Adjusting crawl-delay requires contacting Customer Support.

If you are seeing a 500 Internal Server Error

• Contact your hosting provider to resolve the server issue. If the site is no longer needed, Account Owners can delete it by using the options listed under the How to fix this dropdown menu in the Site summary panel under Manage sites.

If you are seeing a 503 Service Unavailable

• Try again later or contact your hosting provider. If the site is no longer needed, Account Owners can delete it by using the options listed under the How to fix this dropdown menu in the Site summary panel under Manage sites.

If you are seeing a Rendering error

• Reduce overly complex elements (scripts, heavy resources). If the error continues, contact Customer Support.

Cause

The root cause of these issues server performance / complexity

Additional Information

• Related errors or messages: N/A
• Known limitations: N/A
• Configuration notes: N/A