Skip to main content

Advanced Crawler Settings

Depending on your website, you might have some special requirements that can only be fulfilled by accessing our advanced settings, which you will find under Website Crawling or Sitemap Indexing.

Advanced Settings

Secure and password-protected content​

How do I index secure or password-protected content?​

If you have password-protected content that you'd like to include in your search results, you need to authenticate the crawler so it can access the secure pages. There are several options:

  1. If you use HTTP Basic Authentication, fill out a username and password (not applicable if JavaScript-rendered content needs to be added to the search index)
  2. If you have a custom login page, use the Custom Login Screen settings instead (not applicable if JavaScript-rendered content needs to be added to the search index)
  3. Set a cookie to authenticate the crawler
  4. Whitelist the crawler's IP addresses so it can access all pages without a login (under Firewall > Tools). For the current list of IP addresses, see Crawler IP addresses and user agents.
  5. Provide a special sitemap.xml with deep links to the hidden content
  6. Detect the crawler with the following User Agent string in the HTTP header:
    Mozilla/5.0 (compatible; SiteSearch360/1.0; +https://sitesearch360.com/)
  7. Add another User Agent for the JavaScript crawler if you need JS-rendered content to appear in your search results:
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
  8. Push your content to the HTTP REST API
Example: Whitelisting IP addresses in Cloudflare (option 4)

Cloudflare IP whitelist example

Example: Creating a Cloudflare Firewall Rule for the User Agent (option 6)

Cloudflare Firewall Rule for SS360 User Agent

How do I crawl content behind a custom login page?​

  1. Under Advanced Settings > Custom Login Screen, check the box called "Active."
  2. Provide the URL of your login page, e.g. https://yoursite.com/login
  3. Provide the login form XPath:
    On your login page, right-click the login form element, press Inspect, and find its id in the markup. For example:
<form name="loginform" id="loginform" action="https://yoursite.com/login.php" method="post">

Take id="loginform" and address it with the following XPath: //form[@id="loginform"] 4. Define the authentication parameter names and map them with the credentials for the crawler to access the content.
To find the parameter name for your login field, right-click the field and press Inspect. For example:

<input type="text" name="log" id="user_login" class="input">

Take log and use it as the Parameter Name. The login (username, email, etc.) is the Parameter Value. Click Add and repeat the same process for the password field. 5. Save and go to the Index section where you can test your setup on a single URL and re-index the entire site to add the password-protected pages to your search results.

Custom Login Screen

Some login screens have a single field — usually for the password — in which case you only need one parameter name-value pair.

Custom login screens cannot be used alongside JavaScript crawling. If your login screen is JS-rendered, use one of the alternative indexing methods in the previous section (items 3 through 8).

Cookies​

How do I use cookies with the crawler?​

Sometimes it is useful to tell the crawler to set a specific cookie when accessing your website.

For example, if you have a location cookie that determines which language your search results are in, you can set this cookie to us for your English-language project or de for your German-language project.

Cookies for the crawler

Indexing intensity​

What is indexing intensity?​

Indexing intensity controls how quickly the crawler moves through your website. You can set the intensity anywhere from 1 (slowest, least stress on your server) to 5 (fastest, higher stress on your server).

Indexing intensity

If you want to increase crawling speed further, consider switching to sitemap indexing and enabling the Optimize Indexing setting.

JavaScript crawling​

How do I index JavaScript content?​

The Site Search 360 crawler can index content that is dynamically loaded via JavaScript. To enable JS crawling, activate the respective toggle under Website Crawling > Advanced Settings, then re-index your site.

JavaScript crawling is an add-on feature. Set it up in your custom plan.

JS crawling is not enabled for free trial accounts by default. Reach out if you need to test it before signing up for a paid plan.

JavaScript crawling takes more time and resources than standard crawling. If search results are missing content that only appears after JavaScript executes, add the JavaScript Crawling add-on to your custom plan to keep it active after your trial period ends.

Alternatively, you can push JavaScript-rendered content directly to the API.