Crawler Settings
You have set up the crawler to crawl your site (or sitemap!), but you're dealing with missing, unwanted, or duplicate pages? Is crawling taking too long for your tight schedule? There is probably a crawler setting for that!
Missing Pages
If you notice that some search results are missing, the first thing to check is whether the missing URLs are indexed.
The Index section of the Control Panel allows you to look up any URL and check if it was indexed.
If a page is missing from the index and you use Sitemap Indexing, make sure the missing URL is included in your sitemap.
If you are crawling your website, you can try re-indexing the missing page manually.
You may get an error explaining why the page is not indexable. Common causes include:
- The page points to a different domain than your root URL(s).
- A noindex robots meta tag tells the crawler not to index the page.
- Blacklisting, whitelisting, or no-index rules are preventing the URL from being indexed.
Once you have fixed the issue and can index the single URL successfully, recrawl your entire site and check again for the missing page(s).
The crawler skips your root URL(s) by default. For example, if your root URL is domain.com, the homepage will not be indexed. To include it in search results, uncheck Skip Homepage.
The crawler does not follow links to external websites, including Facebook, Twitter, LinkedIn, and similar domains.
Refer to this post for a more in-depth look at missing pages.
What is the noindex robots meta tag and how does it affect my search results?
The noindex robots meta tag is commonly used to prevent search engines from indexing specific pages — for example, during development:
<meta name='robots' content='noindex,follow' />
If you want to keep pages hidden from Google but allow SS360 to index them, check the Ignore Robots Meta Tag box.
If you want pages to remain visible in Google but hidden from your on-site search results, use blacklisting or no-index rules. Alternatively, add the following meta tag to the unwanted pages:
<meta name="ss360" content="noindex" />
Unwanted Pages
If your index contains pages you do not want in your search results, you have a few options depending on where those pages come from.
By default, crawling https://domain.com also crawls https://blog.domain.com and any other subdomains. To exclude subdomain pages, turn off the Crawl Subdomains setting.
To remove specific pages or documents from your search results without deleting them from your website, apply blacklisting, whitelisting, or no-index rules.
Whitelist, blacklist, and no-index URL patterns
The best method depends on your site structure and which pages or documents you want in your index.
URL and XPath patterns are interpreted as regular expressions. Put a backslash (\) before special characters such as []^$.|?*+(){}.
These settings only apply to content added via sitemap(s) and root URL(s). Pages pushed to the URL List and FAQ entries are indexed regardless of these rules.
Blacklist URL patterns
Blacklist patterns tell the crawler to completely ignore specific areas of your site or types of documents.
In this example, pages under /wp-admin/, .php pages, and PDFs will not be indexed. The crawler also will not follow any links found on those pages.
Blacklisting takes priority over whitelisting. If there is a conflict in your settings, the whitelisted patterns are ignored.
Whitelist URL patterns
Whitelist patterns restrict the crawler to a specific area of your site.
For example, to limit search to blog pages only, whitelist /blog/. The crawler will then only index URLs that contain /blog/.
This is also useful for multilingual sites. For example, you could use the following patterns to limit search to French-language pages only:
Make sure your root URL matches your whitelisting pattern (e.g. https://website.com/fr/). If the root URL does not contain the whitelist pattern, it will be blacklisted — which means nothing can be indexed and there will be no search results.
No-index URL patterns
No-index URL patterns behave like the noindex,follow robots meta tag. The crawler follows the page and its outgoing links, but does not include the page itself in the results.
This differs from blacklisting, where the crawler fully ignores the page without checking it for other useful links.
Use no-index patterns for pages that should not appear in search results but contain links to important pages — for example, excluding a Blog landing page while still indexing individual blog posts, or excluding tag pages while indexing the tagged content.
The $ sign in the pattern indicates where the match should stop. URLs that link from the excluded page — such as /specific-url-to-ignore/product1 — will still be followed, indexed, and shown in search results.
No-index URL patterns take priority over whitelisting. If there is a conflict in your settings, the whitelisted patterns are ignored.
No-index XPaths
Sometimes you need to no-index pages that do not share a URL pattern. Instead of adding every URL individually, check whether you can no-index them based on a specific CSS class or ID.
For example, if you have product category pages you want to hide from search results while still crawling the products, and those category pages contain a distinct element such as <div class="product-grid"></div>, you can add it as a no-index XPath:
//div[@class="product-grid"]
The crawler will visit the category pages, then follow and index all outgoing URLs — so your product pages will be indexed and shown in results.
Using many no-index URL patterns or no-index XPaths slows down indexing, because the crawler must scan every page and check it against all rules. If you are confident that a page or directory and all its outgoing links can be safely excluded, use blacklist URL patterns instead — it is faster.
Whitelist XPaths
Whitelist XPaths restrict the crawler to specific pages that do not share a URL pattern.
For example, the following XPath limits search to Russian-language pages only:
Whitelist XPath takes priority over no-index XPath. If there is a conflict in your settings, the no-index XPaths are ignored.
Duplicate Pages
How can I remove duplicate pages?
If you find duplicate pages in your index, the following crawler settings can help.
Use Canonical URL
Canonical tags are a reliable way to avoid duplicate results in both your on-site search and in Google and other search engines. Learn more about the required changes on your side here.
For example, if these three URLs have identical content:
http://mysite.com/url1http://mysite.com/url2http://mysite.com/page1
Add the following tag to the first two pages to indicate that they point to the same canonical URL:
<link rel="canonical" href="http://mysite.com/page1" />
Once this is set up correctly on your site, turn on the Use Canonical URL toggle and re-index your site.
Ignore Query Parameters
If two URLs point to the same content but differ only in query parameters — for example:
http://mysite.com/url1http://mysite.com/url1?utm_campaign=google&sessionId=cb5q69wo5
The crawler treats them as separate entries. To consolidate them, turn on Ignore Query Parameters to strip URL parameters that do not affect page content.
Do not enable this setting if you use query parameters for any of the following:
- Pagination — e.g.
?p=1,?page=2 - Page identification — e.g.
?id=1,?id=2(not just sorting)
Ignoring all query parameters in these cases may cause the crawler to miss relevant pages. Consider these alternatives instead:
- Submit a sitemap with clean URLs and switch to Sitemap Indexing, which is faster and typically produces cleaner results.
- Add pagination to your no-index patterns (e.g.
\?p=) and blacklist other query parameter patterns under Blacklist URL patterns.
Lowercase All URLs
Before enabling this setting, confirm that your server is not case-sensitive. For example:
http://mysite.com/category/producthttp://mysite.com/Category/Product
Remove Trailing Slashes
Only enable this setting if the URLs with and without a trailing slash display the same page:
http://mysite.com/category/product/http://mysite.com/category/product
After adjusting any crawler settings, save your changes and re-index your site. When changing root URLs or sitemaps, empty the index first using Empty Entire Index to start with a clean slate.
Crawling Speed
Depending on the number of pages on your site, indexing time can range from a few minutes to several hours. Because a full re-index is required every time you make changes to the project, this can become time-consuming for larger sites.
If that is the case, switching to Sitemap Indexing as your primary update method is the most effective solution.
The crawler handles sitemaps more efficiently because it does not need to visit your site and follow links one by one. Instead, it reads a single list and adds URLs to your index in a fixed order.
The crawler can also detect changes in the sitemap by reading the <lastmod> tag of each page, which contains the date (and often the time) of the last modification.
What is Optimize Indexing?
When Optimize Indexing is enabled, the crawler only visits updated or new pages during each re-index. This significantly reduces re-indexing time and server load on both sides.
To use this feature:
- Upload your sitemap XML file under Sitemap Indexing, formatted according to these guidelines, and make sure every page has a
<lastmod>tag. - Turn on the Auto Re-Index toggle and check Optimize Indexing.
- Disable auto re-indexing for your root URL under Website Crawling, or remove it entirely if your sitemap contains all relevant pages.
The number of pages in your sitemap still affects indexing time. A project with tens of thousands of URLs will take longer to index even with Optimize Indexing enabled.
There is one case where the crawler ignores Optimize Indexing and performs a full re-index of all data sources: when you make changes that apply across all pages — such as reconfiguring data sources, setting up new content extraction rules, or creating a new result group. In these cases, every page needs to be checked and updated regardless of the <lastmod> tag.
Using a URL List as a temporary workaround
If you are actively updating a large project and need faster results, you can use a URL List of your site's most relevant content instead of other data sources.
This is not a long-term solution. You would need to manually track changes and update the URL list each time a page is added, removed, or modified. The URL list is also limited to 1,000 pages, as its primary purpose is indexing small batches of pages not found on the site or in the sitemap.
If you are not using Sitemap Indexing, you can adjust the indexing intensity in the crawler's advanced settings. This ranges from 1 (slowest, least server load) to 5 (fastest, most server load), and defaults to 2. Increase it if you want to crawl faster and your server can handle the additional load.