A page/product is missing from my search results. What can I do?

The URL is present in the Index Log but not in the search results:

The most common reason for this would be the blacklisting/no-indexing rules you’ve set up. Any URL fitting these rules is automatically skipped by our crawler and is thus missing from the search results. The inner workings of said rules are explained in the respective article, but suffice to say, they override the default setup of the search.

Each correctly indexed URL available to your customers through search result is listed under a 200 status:

No-indexed URLs have a 800 status:

Blacklisted ones are adorned with 802:

If you see the green 200 status next to a URL but it isn't coming up for some keywords that are clearly present on that page, click on the blue info icon to check what is actually indexed (and therefore, searchable) for this URL.

If you can't find something potentially important to your users in this preview, make sure to review your content extraction rules under “Data Structuring” and rewrite them if need be.

If you want the search to be limited to the URL only, simply select "Add URL Path to Title".

If the indexed content is limited to the URL, chances are the content of the page is rendered with JavaScript, and you'd need to configure JS Crawling to fetch it. It is a paid add-on but if you want to enable and test it, message us and we’ll set you up at a moment’s notice.

The URL is missing from the Index Log:

In this case, you will need to check if the URL can be added manually by typing it in under “Index Single URL” in your Index Log.

If not, you will see the following error message:

This would mean that there are some no-indexing/blacklisting rules that force specific URLs out of your search results.

Another explanation would be the presence of Canonical URLs (which you can read up on in our docs). Those are used to avoid duplicate search results and are enabled by ticking off the appropriate box under “Additional Settings” in Website Crawling or Sitemap Indexing. Essentially it means that different URLs contain the exact same content and are combined into one, which is why some pages might disappear from your Index Log.

Lastly, your website might have the noindex robots meta tag in its structure. This tag keeps certain site pages hidden from Google and other crawlers, which is why some URLs might be skipped in the indexing process. Our crawler can still include them in the Index Log if you tick off “Ignore Robots Meta Tag” under Website Crawling or Sitemap Indexing, same as with Canonical URLs.

You can learn more about this meta tag in our docs.

If you can index the missing URL manually, it means that the crawler might be at fault, having failed to pick up all available content, in which case you will need to review your Data Sources setup:

• Make sure the URL you’re looking for doesn’t have a 404 status signifying a broken link and the URL that led to it has been successfully indexed as well.

• If you have opted for Sitemap Indexing rather than Website Crawling, check for the missing URL within the sitemap and try adding it manually under “Index Single URL”, “URL List” or directly to the sitemap. Provided that you can’t, we’d suggest you try combining Sitemap Indexing with Website Crawling to ensure full coverage in case some URLs are present on your website only. For instance, if you have PDFs on the site but not in the sitemap, you’ll render them instantly available to be presented in the search results by enabling Website Crawling.

• Same as before, check for contradictory No-indexing/Blacklisting rules, the Noindex Robots Meta tag and Canonical URLs.

• Ascertain that the missing URL is not password-protected and thus totally invisible to the crawler. If you have such restrictions (say, a login page) on your site, you’ll have to enable crawling manually. Check out our instructions on it in the article dedicated to password-protected content and another one pertaining to getting past Custom Login Screens specifically.