Starting the Crawler
The quality of your search results depends on the quality of your search index. To build that index, you need to tell the SS360 crawler which pages, documents, and other content on your site matter to your visitors.
A crawler is a bot that browses your website and builds the search index your search is based on. A search index is the list of pages and documents returned as search results when a visitor types a query into your search box.
You have two main options for directing the crawler to the right content:
Website Crawling
Enter a root URL — typically your homepage — and click Index. The crawler visits that URL, follows all internal links it finds, and adds the discovered pages to your index.
You can enter one root URL or several.
Sitemap Indexing
Sitemap Indexing is the preferred indexing method — it is the quickest and most efficient way to crawl a website.
If the crawler detects a valid sitemap XML file for your domain, it reads the URLs listed there directly. Sitemaps are typically found at:
https://www.yoursite.com/sitemap.xmlhttps://www.yoursite.com/sitemap-index.xml
The sitemap XML file must be correctly formatted for the crawler to process it. See the sitemap protocol guidelines.
If no valid sitemap is detected, the crawler automatically falls back to Website Crawling.
If you have a sitemap that was not detected automatically, provide its URL manually. Click Test Sitemap to confirm it is valid and to see how many URLs it contains. Once the test passes, click Index All Sitemaps.
With Website Crawling, the only way to check the number of indexed pages is to wait until a full crawl completes.
Using both methods together
You can run Website Crawling and Sitemap Indexing at the same time. When you do, sitemap content is processed first. If a URL appears in both a sitemap and a root URL crawl, it is indexed only once.
Make sure to enable the Auto Re-Index toggle under each crawling method you use.
Indexing Multiple Sites
If your content spans more than one domain or subdomain — for example:
- A blog at
http://blog.mysite.com/ - A main site at
http://mysite.com/ - Additional content at
http://myothersite.com/
You can combine all three into a single index using one or more of these approaches:
- Submit a combined sitemap — create a sitemap containing URLs from all sites, or submit multiple sitemaps (one per line) in the Sitemap Indexing panel. The crawler picks up only the URLs listed there.
- Submit multiple root URLs — enter one root URL per site in the Website Crawling panel.
- Use the API — add pages from any site via the API using your API key (available on the Batman plan or a custom plan with API access). You can index by URL or send a JSON object with the page content directly.
To control which pages from your configured sources end up in the index, see Crawler Settings.