Google routinely crawls your mobile and desktop product pages and images to check for quality issues. If we're unable to perform these crawls, we won't be able to show your items in listings.
In addition, for landing pages, if we detect crawling errors while fetching either the mobile or desktop landing page of an item, Google will disapprove the item for both mobile and desktop devices until we are able to access the landing page successfully.
The most common reasons for product crawl issues are:
- Page not found (404) error: The provided URL is incorrect or broken and returns a 404 error. Please verify that the URL is accurate and your website is currently live.
- Server's robots.txt disallows access: Your robots.txt file is blocking access to Google crawlers. Google uses AdsBot-Google for ads and Googlebot for organic search. Even if one is allowed, the other might be blocked. Ask your webmaster to update the file to allow both AdsBot-Google and Googlebot to access your landing pages.
- Invalid URL: Your URL contains invalid characters or doesn't have the format of a valid link.
Note: Once you've resolved the issue, it may take up to 48 hours for your product to reappear in listings.
There are a number of other issues that may also prevent Google from crawling your page.
Other common issues- Page requires authentication: The URL provided is protected by some sort of authentication protocol that prevents Google from accessing the content.
- HTTP 4xx and 5xx responses: The server hosting your website returned an HTTP error that prevented us from accessing the content.
- Hostname cannot be resolved: We were unable to resolve the hostname of your server to an IP address and as such could not access the page.
- Malformed HTTP response: The response from your server was garbled.
- Firewalls, geo-restrictions or private IP: Your website may be behind a firewall, have geo-restrictions or use a private IP that blocks our crawlers. If you restrict specific regions, you must allow crawling from the US because Google often operates from there. Ask your webmaster or hosting provider to update your firewall and server settings to allow the AdsBot-Google user agent.
- Network error: A connection issue occurred during the crawl.
- Timeout reading page: The server didn't respond in time, so the crawl for this product stopped.
- Server redirects too often: The server redirected the request too many times, which caused the crawl to fail.
- Redirect errors: The server returned an invalid, empty or excessively long URL that prevented the crawl.
- Robots.txt unreachable: We couldn't access your robots.txt file, which stopped the crawl. You can read more about the robots exclusion protocol here.
- Regional landing pages not accessible: If using the regional availability and pricing add-on feature, ensure that all landing pages with region ID can be crawled. For example, URLs like the following should be accessible to Google.
- https://mystorewebsite.com/p/myproductlandingpage?region_id=123456
- https://mystorewebsite.com/p?offer=myproduct®ion_id=123456