Skip to main content

With the robot setting "Follow rules for googlebot" more pages are found and crawled

With the robot setting "Follow rules for googlebot" more pages are found and crawled

Benjamin Weßel avatar
Written by Benjamin Weßel
Updated over 3 weeks ago

Symptom

  • If the option Follow rules for googlebot is set in the settings under Onpage > Analysis management > example.tld > Advanced > Robot settings, then more pages are found and crawled in the analysis compared to the setting Follow rules for *

  • the robots.txt file contains the following text:

User-agent: *
Disallow: /

Cause

The XOVI crawler is blocked by the robots.txt file.

Solution

  • Adjust the robots.txt accordingly to allow the XOVI crawler

or

  1. Log in to suite.xovi.net

  2. navigate to Onpage > Analysis management > example.tld > Advanced

  3. Set the Robot setting option to Ignore instructions for robots

  4. click on START ANALYSIS WITH CHANGED SETTINGS

Did this answer your question?