Skip to main content

Index Troubleshooting

The Index section of the Control Panel shows you all the pages and documents that the crawler has found on your website.

The search index

Crawler IP Addresses and User Agents​

If your server, firewall, or CDN restricts access by IP or User Agent, whitelist the following values to allow the SS360 crawler to reach your content.

Crawler IP addresses:

  • 88.99.218.202
  • 88.99.149.30
  • 88.99.162.232
  • 88.99.29.101
  • 149.56.240.229
  • 51.79.176.191
  • 51.222.153.207
  • 139.99.121.235
  • 94.130.54.189
  • 116.202.85.24

User Agent — standard crawler:

Mozilla/5.0 (compatible; SiteSearch360/1.0; +https://sitesearch360.com/)

User Agent — JavaScript crawler:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36

The User Agent differs between the standard crawler and the JavaScript crawler. Whitelist both if your site uses JavaScript rendering.

Fix Problems with Your Search Index​

If search is not working as expected, the issue is often in the index itself.

When reviewing your index, look for missing pages, unwanted pages, and duplicate pages. If you find any of these, refer to the crawler settings to resolve them.

Also check for status messages that may explain unexpected search behaviour.

Why do my pages get skipped?​

Successfully indexed URLs display a green 200 status:

URL with 200 status

Skipped pages show a gray 800 or 802 status, depending on why the crawler skipped them. In both cases, a rule configured under crawler settings is telling the crawler to exclude those pages. If pages you want searchable are being skipped, remove or update the matching rule.

URL with 800 statusURL with 802 status

After updating the rule, index the individual URL to confirm it returns a 200 status.

Index single URL

You may also see red error statuses in your index.

Error status

URLs with an error status have not been successfully indexed and will not appear in search results.

Index status code reference​

For a full list of status codes and their meanings, see Index Status Codes.

How do I fix Client Error 499?​

Client Error 499 means the client closed the connection before the server could respond — typically due to a timeout — so the page or document could not be loaded.

Re-indexing the specific URLs or your entire site usually resolves this.

This error can also occur when Cloudflare blocks the SS360 crawler. Whitelist the crawler IPs and User Agents in Cloudflare under Firewall > Tools:

Cloudflare firewall

Adding User Agent on Cloudflare

Cloudflare can be configured as part of your CMS (e.g. WordPress, Magento, Drupal). If you are unsure how to proceed, contact your CMS support and ask them to whitelist the Site Search 360 crawler IPs.

How do I fix Client Error 403?​

A 403 error means your server is refusing the crawler's request for a specific page or file.

If whitelisting the crawler IP addresses and User Agents has not resolved the issue, check your HTTP method settings. Some CMSs — for example, Magnolia CMS — block HEAD requests by default. Add HEAD to the list of allowed methods so the crawler can reach your documents.

How do I fix Client Error 404?​

A 404 error means the page was not found. Open the URL to check whether the page still exists. If it does not:

  • Remove it from your sitemap.
  • Find any broken links pointing to the missing page and update or remove them.

How do I fix Server Error 500?​

A 500 error means your server is experiencing internal problems. The crawler cannot index your site until the server is stable again. Once your site is back up, delete your index and run a clean re-index.

Re-Indexing Your Site​

Many changes to your search setup require a re-index before they take effect. Re-indexing triggers a full recrawl of your site, applying any configuration changes made since the last crawl.

Changes that require a re-index include:

  • Adding or modifying data points
  • Adding or modifying result groups
  • Updating whitelist, blacklist, or no-index patterns

If you are unsure whether a re-index is needed, check the notification bar in the upper right corner of the Control Panel — it will prompt you when one is required.

Re-index notification

To re-index, click Re-Index Now in the notification, or go to the Index page and scroll down to the re-index button.

Re-index button

This button is only available when at least one source is set to Auto-Reindex. Only sources with Auto-Reindex enabled will be crawled.

Auto Re-Index

Update Validity Safeguard​

By default, SS360 validates incoming updates and blocks any data upload that would reduce your total index count by more than 30%. This prevents accidental large-scale content loss — for example, if an updated sitemap contains significantly fewer URLs than expected.

  • This safeguard is enabled by default and is not configurable in the UI.
  • To disable it, contact Support.

This safeguard does not affect the Empty Entire Index option. It only applies to new uploads or updates that would remove a large portion of existing indexed content.

Removing URLs from the Sitemap​

When a sitemap re-index runs, SS360 checks any URL that was previously indexed but is no longer present in the new sitemap. Before removing it, the crawler sends a request to verify whether the page is still reachable. If the page returns a successful response, SS360 keeps it in the index — even if it has been removed from the sitemap.

This behaviour is by design. It prevents pages from being dropped due to a temporarily incomplete or misconfigured sitemap.

A URL that is still live on your server will not be automatically removed from the index, even after a full sitemap re-index. Removing a URL from your sitemap alone is not enough to remove it from search results.

To remove stale URLs from the index, you have two options:

  • Remove URLs manually — go to the Index page, search for the URL, and delete it directly. You can also use bulk removal by URL pattern if multiple URLs share a common path.
  • Clear the index and re-index from scratch — go to Data Sources and use Empty Entire Index, then trigger a fresh re-index. This fully rebuilds the index from your current sitemap, so only URLs present in your data sources will be indexed. Plan this for a low-traffic period, as search results will be limited while the index is rebuilding.

Crawler Log Add-On​

By default, SS360 does not track which pages lead to broken links (404s). The Crawler Log is a paid add-on that enables this. When active, an Index Source column appears in the Index Control Status Table, showing the referring URL for each indexed entry.

source URL

This lets you trace the path from a working URL to a broken one, so you can update or remove the broken link directly.

To enable the Crawler Log, create a custom plan.