Approval & site review

Fix "Site down or unavailable" in AdSense: make your site reachable for Google's crawlers

Last updated Published By the editorial team6 min5 sources

Key takeaways

  • AdSense reports a site as down, unavailable or unreachable when its crawlers cannot load the site, even if it works in your own browser.
  • Google's AdSense crawler is Mediapartners-Google, and AdSense uses Google-Display-Ads-Bot to verify a site when you add it.
  • Google lists login protection, IP or geographic restrictions, robots.txt blocks, web application firewalls, 404s, redirect chains, name server problems and slow servers as crawler access issues.
  • If you use HTTPS, Google requires a certificate from a recognized certification authority, not a self-signed one, and HTTP should redirect to HTTPS.
  • Google says changes to a page may take up to 1 or 2 weeks to be reflected in the AdSense crawler's index.
Close-up of optical fiber cable strands
Image: Optical fiber cable-10ASD,Asurnipal,CC BY-SA 4.0,via Wikimedia Commons.

“Site down or unavailable” (shown in Google’s help pages as “Your site is unreachable”) means Google’s crawlers could not load your site during the review. The cause is almost never a real outage: it is usually DNS, HTTPS, a redirect, a robots.txt rule, a country block or bot protection that treats Google’s crawler as an unwanted bot. Fix access, verify it from outside, then request review.

Which crawlers need to reach your site?

AdSense uses two Google crawlers that you need to let through:

Crawler Role (per Google) robots.txt token
Mediapartners-Google “visits your site to determine its content in order to provide relevant ads” Mediapartners-Google
Google-Display-Ads-Bot Used “to verify your site when you add a site to AdSense” Google-Display-Ads-Bot

Google’s crawler documentation classifies Mediapartners-Google as a special-case crawler. These “may ignore robots.txt rules and so they operate from a different IP range than the common crawlers”, published in a special-crawlers.json file, and their reverse DNS matches rate-limited-proxy-***-***-***-***.google.com. That detail matters: an allowlist built only for Googlebot’s IP ranges may not cover the AdSense crawler.

What causes “Site down or unavailable”?

Google’s “Fix AdSense crawler issues” page groups the causes into three families. Here they are with the usual real-world trigger.

Google’s issue Typical trigger Fix
Login-protected site Password-protected staging, membership plugin, “coming soon” mode Remove the login during review; after activation, add a crawler login
Restricted network or IP range Country blocking, hosting firewall, IP allowlist Remove the restriction or make the content public
Blocked in robots.txt A group naming Mediapartners-Google with Disallow: / Remove that group
Blocked by a web application firewall CDN or WAF bot protection challenging Google Let the crawler bypass the firewall
Site can’t be found 404 on the submitted URL, site not live, wrong domain Publish the site, fix the URL
Multiple redirects Chains like http to https to www to a language folder Shorten to one redirect
POST data required Pages that only answer POST requests Serve content over GET
Name server problem Broken or partially propagated DNS, geo-DNS rules Fix DNS configuration
Server problem Server down, slow or overloaded Use reliable hosting

The “not ready” page adds an HTTPS requirement: if you use HTTPS, you need “a valid SSL certificate from a recognized certification authority (not self-signed)” and HTTP should redirect to HTTPS.

How do I find which one is blocking the crawler?

Work from the outside in. Each step either clears a cause or finds it.

1. Confirm the exact URL in AdSense

Open “Sites” in AdSense and read the URL you submitted. A typo, a missing www that does not resolve, or an old domain is the simplest cause. If the URL is wrong, remove the site and add it again with the correct URL, as Google instructs.

2. Check DNS and HTTPS from outside your network

From a terminal (or an online DNS checker), confirm that both the bare domain and www resolve, and that the certificate is valid for the name you submitted:

dig +short example.com
dig +short www.example.com
curl -sI https://example.com/

A certificate warning, an expired certificate, or a certificate issued for another hostname is enough to fail. If you recently changed DNS or hosting, wait until the old records have expired everywhere before requesting review.

3. Follow the redirects

curl -sIL http://example.com/ | grep -iE "^(HTTP|location)"

You want at most one hop to the final HTTPS URL, then a 200. Chains, loops, or a final 403, 429 or 5xx explain the error. Language or country redirects based on IP are a frequent trap: the crawler may be sent to a version that does not exist.

4. Read robots.txt

Open https://example.com/robots.txt and look for any group that names Mediapartners-Google or Google-Display-Ads-Bot with a Disallow. Google’s help page says that to grant access you remove these lines:

User-agent: Mediapartners-Google
Disallow: /

A generic User-agent: * block is not the problem for AdSense: Google’s crawler documentation says “The global user agent (*) is ignored” for Mediapartners-Google. It may still hurt your Search visibility, which is a separate issue.

5. Check firewalls, bot protection and geo-blocking

This is the most common cause on otherwise healthy sites. Look at your CDN or firewall security log for requests whose user agent contains Mediapartners-Google or Google-Display-Ads-Bot and that were blocked, challenged or rate-limited.

  • Country blocks. If you block or challenge traffic from some countries, the crawler can be caught. Google lists “Geographic or IP restrictions” as a cause.
  • JavaScript or CAPTCHA challenges. A crawler cannot solve them. A challenge page is, for the crawler, an unreachable site.
  • Cloudflare Bot Fight Mode. Cloudflare’s documentation says it “cannot be customized, adjusted, or reconfigured via WAF custom rules” and that you “cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules”. If the log shows Google requests challenged by it, turning it off is the practical option during review. Cloudflare points to Super Bot Fight Mode when you need skip rules.
  • Hosting security plugins. WordPress security plugins and server-level rate limits can block “unknown” bots. Allow Google’s published crawler ranges rather than trusting the user agent alone; Google warns that “the user agent string can be spoofed.”

Testing with a fake user agent (curl -A "Mediapartners-Google" ...) only detects user-agent rules. It cannot reproduce IP-based or reputation-based blocking, so the logs are the real evidence.

6. Remove login walls and maintenance modes

“Coming soon”, maintenance and password plugins return a login or placeholder page to anonymous visitors. Google asks you to remove the login temporarily so AdSense can reach the site. After activation, you can set up a crawler login under “Account”, “Access and authorization”, “Crawler access”; Google requires the site to be verified in Search Console for that.

7. Check server reliability

A site that times out under load, or a free host that sleeps when idle, can be down exactly when the crawler arrives. Check uptime history and response times for the review period.

After the fix: how do I request the review again?

  1. Confirm from a different network (a phone on mobile data is enough) that the homepage and several inner pages load without challenge or login.
  2. In AdSense, click “Sites”, then the site.
  3. Confirm the code is in place, click “Next”, then “Request review”.

Google says the review usually takes a few days but can take 2-4 weeks. Google also notes that crawler changes “may take up to 1 or 2 weeks” to be reflected in its index, so do not panic if the status does not change the next day. What each status means is covered in AdSense site statuses explained.

Common mistakes

  • Testing only from your own computer. Your IP, cookies and session are whitelisted by habit; the crawler’s are not.
  • Allowlisting Googlebot only. AdSense uses different crawlers, and Mediapartners-Google is a special-case crawler with its own IP ranges.
  • Leaving a challenge on “only for some countries”. You do not control where the crawler’s requests come from, and Google lists geographic restrictions as a cause.
  • Changing hosting or DNS in the middle of a review. Finish migrations before you submit.
  • Fixing the homepage only. The review can cover all pages. A WAF rule that protects /blog/ or /wp-json/ differently can still fail it.
  • Confusing it with ads.txt status. An ads.txt “Not found” is a different signal with different causes; see ads.txt “Not found” and “Unauthorized”.

Static hosts and Jamstack platforms have their own variants of these issues (preview deployments, trailing slashes, platform bot protection); see AdSense on static and Jamstack sites.

Frequently asked questions

My site works in my browser. Why does AdSense say it is unavailable?

Your browser is not Google's crawler. The crawler arrives from Google's own IP ranges, without your cookies or session, without solving JavaScript challenges and with its own user agent. A firewall, bot filter, country block or login wall can let you in and stop it. Test from outside your network and check your server or CDN logs for blocked Google requests.

Does blocking all bots with User-agent: * in robots.txt block AdSense?

Not for pages with ads. Google's crawler documentation says the global user agent (*) is ignored by Mediapartners-Google, and the AdSense crawler page says pages roboted out with User-agent: * are still crawled. Only a group that names Mediapartners-Google or Google-Display-Ads-Bot blocks them.

How long after fixing access will AdSense see my site?

Google does not give a fixed delay for the review crawl. Its crawler page says changes to a page may take up to 1 or 2 weeks to be reflected in its index, and the site review itself usually takes a few days but can take 2-4 weeks. Request the review only once you have confirmed the fix from outside your network.

Can I keep Cloudflare Bot Fight Mode on while applying for AdSense?

Be careful. Cloudflare's documentation says Bot Fight Mode cannot be bypassed with WAF custom rules or Page Rules, so you cannot carve out an exception for a specific crawler there. If your logs show Google requests being challenged, turn it off during review or use a configuration that supports skip rules.

Sources

  1. What to do when your site is not ready to show ads · Google AdSense Help
  2. Fix AdSense crawler issues · Google AdSense Help
  3. About the AdSense ads crawler · Google AdSense Help
  4. Google's special-case crawlers · Google for Developers
  5. Bot Fight Mode · Cloudflare Docs