Troubleshoot uncrawlable landing pages

Google requires that your ad destination and creatives are crawlable by Google AdsBot web crawlers to verify that users are led to an ad destination that’s relevant to the ad they’ve clicked. Ensure that Google can crawl your site effectively by using a URL structure that follows URL Structure Best Practices for Google Search. Learn more about the Destination requirements policy.

Follow the steps in this article to troubleshoot these issues.

On this page

• Find the source of the uncrawlable URL
• Check for robots.txt files
• Test your landing page to find out if it is crawlable
• Other reasons a landing page can't be crawled
• Find uncrawlable URLs in creative code

Find the source of the uncrawlable URL

1. Open the creative.
2. Click Creative status.
3. If it's not expanded, click Display & Video 360 review. Copy the uncrawlable landing page URL from the feedback.
4. Preview the creative, then click it to open the landing page. Observe the final URL in your browser after all redirects. Compare this URL with the uncrawlable URL you copied.
   • If the declared landing page URL matches the flagged URL but differs from the final destination URL after redirects, either update the declared landing page URL in Display & Video 360 or Google Ads to match the final destination, or modify any click trackers or redirects to ensure they lead to the declared domain. For Campaign Manager 360, verify the Use a static landing page URL option if applicable.
   • If the flagged URL matches the final destination URL, but it's reported as uncrawlable, click Creative details to review the creative's “Landing page URL”. You'll need to either change it to a different, crawlable URL, or fix the issues preventing the current landing page from being crawled.
   • If the URLs don't match in other ways, or the source isn't clear, continue to the next step to investigate further.
5. Find where the uncrawlable landing page is:
   • Open the creative's HTML and JavaScript files and search for the URL Display & Video 360 reported as uncrawlable or mismatched.
   • If the creative is synced from Campaign Manager 360, contact the advertiser's Campaign Manager 360 trafficker. Ask them to check and correct the URL to ensure it matches the final destination and is crawlable.
   • If Display & Video 360 can't detect a landing page for the creative, your Display & Video 360 advertiser website is used instead. To find this URL, open your advertiser and click Advertiser settings, then Basic details. The URL will be shown in the ”Website” field.

Check for robots.txt files

Website administrators with access to the Google Search Console for the advertiser's website can use the robots.txt Tester to troubleshoot. If you don't have access, follow the steps below.

1. Copy the URL and paste it in a browser's address bar.
2. Add /robots.txt to the end of the domain name, then press Enter. This will show you the robots.txt file for the main domain. For example, you should use https://example.com/robots.txt if your domain is https://example.com/.
   • Note: If the file is blank or doesn't exist, you might be redirected to the homepage or see a 404 error. If your site doesn’t have a robots.txt file, Google's crawlers are allowed. Google's crawlers are allowed by default for the paths not covered by other rules. However, ensure there isn't a broader server configuration blocking access.
3. If your site has a robots.txt file, ensure these user-agents are present and permit crawling of your landing pages:
   • User-agent: AdsBot-Google
   • User-agent: Googlebot
4. Confirm that these user-agent lines aren't followed by the Disallow: / line that would block access to your landing page.
5. If they're disallowed, work with the website administrator to remove the / or change the line to Allow: /.
6. If the landing page is on a subdomain, check if the subdomain has a separate robots.txt file. For example, https://subdomain.example.com/robots.txt. Make sure this robots.txt also allows the AdsBot-Google and Googlebot user agents.

Examples

Robots.txt example	Crawlers allowed
User-agent: AdsBot-Google Disallow: User-agent: Googlebot Disallow:
User-agent: AdsBot-Google Allow: / User-agent: Googlebot Allow: /
User-agent: AdsBot-Google Disallow: / User-agent: Googlebot Disallow: /
User-agent: AdsBot-Google Allow: User-agent: Googlebot Allow:
User-agent: AdsBot-Google User-agent: Googlebot

Test your landing page to find out if it is crawlable

1. Go to the page in Google Chrome.
2. Select the Chrome menu , then click More Tools, then Developer Tools.
3. In the Developer Tools panel, click More options , then More Tools, then Network Conditions.
4. In the Network Conditions section, turn off “Use browser default” in the “User Agent” cell.
5. In the user agents menu, select "Googlebot." Reload the page to find out if Googlebot can access it.
6. In the user agents menu, select "Custom". In the field below, update the user agent path to AdsBot-Google (+http://google.com/adsbot.html). Reload the page to find out if AdsBot can access it.

Other reasons a landing page can't be crawled

If the Google AdsBot is allowed by robots.txt and the URL seems correct, other issues might prevent crawling or cause disapprovals:

• The landing page isn't loading or is intermittently inaccessible: If you open the page in a browser and encounter an HTTP error such as 404 not found or 500 server error, or if the page loads intermittently, contact your website administrator. They need to ensure the landing page is consistently accessible and loads correctly in all locations, environments (mobile and desktop), and browsers. You should also check for geo-restrictions that might be blocking Google's crawlers. If geo-restrictions are necessary for your business, you should use campaign geo-targeting settings rather than blocking crawlers, as this can lead to "Destination not crawlable" errors. Intermittent accessibility can also cause these errors.
• The landing page requires sign-in: Ad landing pages must be publicly accessible. Use another landing page or work with the website administrator to remove the sign-in requirement from the page.
• The landing page has too many forwards or redirects: Landing pages must have fewer than 10 redirects. Excessive redirects can contribute to "Destination not working" or "Destination mismatch" if the final URL is not as expected or the redirect chain breaks. Work with your website administrator to reduce the number of redirects. Learn more about redirects.
• The landing page includes a search engine that returns no results when the page loads: If the landing page starts a site search, make sure the site search engine returns a result, or the crawler might consider the page invalid. Work with the site's website administrator to return some default search results instead of showing an error page or a message such as "No results".
• The landing page URL includes a pound sign (#): Display & Video 360 is unable to process URLs with pound signs. Either remove the pound sign from the URL, or use a different landing page.
• The landing page is not crawlable due to a timeout: Google is trying to crawl the landing page, but the request takes too long to complete. Timeouts can happen occasionally for different reasons. If it happens frequently, this can cause the "Destination not working" issue. Contact the website administrator to investigate and resolve server performance or page load time issues.
• The landing page leads to a CAPTCHA page: Ad landing pages must be publicly accessible and CAPTCHA makes the destination pages uncrawlable. Contact the website administrator to either remove the CAPTCHA in your landing pages or to allow Google’s special-case crawlers to bypass the CAPTCHA. Learn more about the List of Google’s special-case crawlers.

After fixing the identified landing page problems, your ad or creative will be re-reviewed automatically. You should monitor the status of your ad or creative. If the status doesn't update, or if you believe your page is compliant and was incorrectly flagged, you can appeal the decision through the Google Ads or Display & Video 360 interface.

Find uncrawlable URLs in creative code

Landing page URLs can be set in several places in a Display & Video 360 creative. Sometimes they're set directly in the creative code instead of in Display & Video 360. Use the following steps to find uncrawlable URLs that are set directly in the creative code: