Fix "Site down or unavailable" in AdSense: make your site reachable for Google's crawlers
Key takeaways
- AdSense reports a site as down, unavailable or unreachable when its crawlers cannot load the site, even if it works in your own browser.
- Google's AdSense crawler is Mediapartners-Google, and AdSense uses Google-Display-Ads-Bot to verify a site when you add it.
- Google lists login protection, IP or geographic restrictions, robots.txt blocks, web application firewalls, 404s, redirect chains, name server problems and slow servers as crawler access issues.
- If you use HTTPS, Google requires a certificate from a recognized certification authority, not a self-signed one, and HTTP should redirect to HTTPS.
- Google says changes to a page may take up to 1 or 2 weeks to be reflected in the AdSense crawler's index.

“Site down or unavailable” (shown in Google’s help pages as “Your site is unreachable”) means Google’s crawlers could not load your site during the review. The cause is almost never a real outage: it is usually DNS, HTTPS, a redirect, a robots.txt rule, a country block or bot protection that treats Google’s crawler as an unwanted bot. Fix access, verify it from outside, then request review.
Which crawlers need to reach your site?
AdSense uses two Google crawlers that you need to let through:
| Crawler | Role (per Google) | robots.txt token |
|---|---|---|
Mediapartners-Google |
“visits your site to determine its content in order to provide relevant ads” | Mediapartners-Google |
Google-Display-Ads-Bot |
Used “to verify your site when you add a site to AdSense” | Google-Display-Ads-Bot |
Google’s crawler documentation classifies Mediapartners-Google as a special-case crawler. These “may ignore robots.txt rules and so they operate from a different IP range than the common crawlers”, published in a special-crawlers.json file, and their reverse DNS matches rate-limited-proxy-***-***-***-***.google.com. That detail matters: an allowlist built only for Googlebot’s IP ranges may not cover the AdSense crawler.
What causes “Site down or unavailable”?
Google’s “Fix AdSense crawler issues” page groups the causes into three families. Here they are with the usual real-world trigger.
| Google’s issue | Typical trigger | Fix |
|---|---|---|
| Login-protected site | Password-protected staging, membership plugin, “coming soon” mode | Remove the login during review; after activation, add a crawler login |
| Restricted network or IP range | Country blocking, hosting firewall, IP allowlist | Remove the restriction or make the content public |
| Blocked in robots.txt | A group naming Mediapartners-Google with Disallow: / |
Remove that group |
| Blocked by a web application firewall | CDN or WAF bot protection challenging Google | Let the crawler bypass the firewall |
| Site can’t be found | 404 on the submitted URL, site not live, wrong domain | Publish the site, fix the URL |
| Multiple redirects | Chains like http to https to www to a language folder | Shorten to one redirect |
| POST data required | Pages that only answer POST requests | Serve content over GET |
| Name server problem | Broken or partially propagated DNS, geo-DNS rules | Fix DNS configuration |
| Server problem | Server down, slow or overloaded | Use reliable hosting |
The “not ready” page adds an HTTPS requirement: if you use HTTPS, you need “a valid SSL certificate from a recognized certification authority (not self-signed)” and HTTP should redirect to HTTPS.
How do I find which one is blocking the crawler?
Work from the outside in. Each step either clears a cause or finds it.
1. Confirm the exact URL in AdSense
Open “Sites” in AdSense and read the URL you submitted. A typo, a missing www that does not resolve, or an old domain is the simplest cause. If the URL is wrong, remove the site and add it again with the correct URL, as Google instructs.
2. Check DNS and HTTPS from outside your network
From a terminal (or an online DNS checker), confirm that both the bare domain and www resolve, and that the certificate is valid for the name you submitted:
dig +short example.com
dig +short www.example.com
curl -sI https://example.com/
A certificate warning, an expired certificate, or a certificate issued for another hostname is enough to fail. If you recently changed DNS or hosting, wait until the old records have expired everywhere before requesting review.
3. Follow the redirects
curl -sIL http://example.com/ | grep -iE "^(HTTP|location)"
You want at most one hop to the final HTTPS URL, then a 200. Chains, loops, or a final 403, 429 or 5xx explain the error. Language or country redirects based on IP are a frequent trap: the crawler may be sent to a version that does not exist.
4. Read robots.txt
Open https://example.com/robots.txt and look for any group that names Mediapartners-Google or Google-Display-Ads-Bot with a Disallow. Google’s help page says that to grant access you remove these lines:
User-agent: Mediapartners-Google
Disallow: /
A generic User-agent: * block is not the problem for AdSense: Google’s crawler documentation says “The global user agent (*) is ignored” for Mediapartners-Google. It may still hurt your Search visibility, which is a separate issue.
5. Check firewalls, bot protection and geo-blocking
This is the most common cause on otherwise healthy sites. Look at your CDN or firewall security log for requests whose user agent contains Mediapartners-Google or Google-Display-Ads-Bot and that were blocked, challenged or rate-limited.
- Country blocks. If you block or challenge traffic from some countries, the crawler can be caught. Google lists “Geographic or IP restrictions” as a cause.
- JavaScript or CAPTCHA challenges. A crawler cannot solve them. A challenge page is, for the crawler, an unreachable site.
- Cloudflare Bot Fight Mode. Cloudflare’s documentation says it “cannot be customized, adjusted, or reconfigured via WAF custom rules” and that you “cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules”. If the log shows Google requests challenged by it, turning it off is the practical option during review. Cloudflare points to Super Bot Fight Mode when you need skip rules.
- Hosting security plugins. WordPress security plugins and server-level rate limits can block “unknown” bots. Allow Google’s published crawler ranges rather than trusting the user agent alone; Google warns that “the user agent string can be spoofed.”
Testing with a fake user agent (curl -A "Mediapartners-Google" ...) only detects user-agent rules. It cannot reproduce IP-based or reputation-based blocking, so the logs are the real evidence.
6. Remove login walls and maintenance modes
“Coming soon”, maintenance and password plugins return a login or placeholder page to anonymous visitors. Google asks you to remove the login temporarily so AdSense can reach the site. After activation, you can set up a crawler login under “Account”, “Access and authorization”, “Crawler access”; Google requires the site to be verified in Search Console for that.
7. Check server reliability
A site that times out under load, or a free host that sleeps when idle, can be down exactly when the crawler arrives. Check uptime history and response times for the review period.
After the fix: how do I request the review again?
- Confirm from a different network (a phone on mobile data is enough) that the homepage and several inner pages load without challenge or login.
- In AdSense, click “Sites”, then the site.
- Confirm the code is in place, click “Next”, then “Request review”.
Google says the review usually takes a few days but can take 2-4 weeks. Google also notes that crawler changes “may take up to 1 or 2 weeks” to be reflected in its index, so do not panic if the status does not change the next day. What each status means is covered in AdSense site statuses explained.
Common mistakes
- Testing only from your own computer. Your IP, cookies and session are whitelisted by habit; the crawler’s are not.
- Allowlisting Googlebot only. AdSense uses different crawlers, and
Mediapartners-Googleis a special-case crawler with its own IP ranges. - Leaving a challenge on “only for some countries”. You do not control where the crawler’s requests come from, and Google lists geographic restrictions as a cause.
- Changing hosting or DNS in the middle of a review. Finish migrations before you submit.
- Fixing the homepage only. The review can cover all pages. A WAF rule that protects
/blog/or/wp-json/differently can still fail it. - Confusing it with ads.txt status. An ads.txt “Not found” is a different signal with different causes; see ads.txt “Not found” and “Unauthorized”.
Static hosts and Jamstack platforms have their own variants of these issues (preview deployments, trailing slashes, platform bot protection); see AdSense on static and Jamstack sites.
Frequently asked questions
My site works in my browser. Why does AdSense say it is unavailable?
Your browser is not Google's crawler. The crawler arrives from Google's own IP ranges, without your cookies or session, without solving JavaScript challenges and with its own user agent. A firewall, bot filter, country block or login wall can let you in and stop it. Test from outside your network and check your server or CDN logs for blocked Google requests.
Does blocking all bots with User-agent: * in robots.txt block AdSense?
Not for pages with ads. Google's crawler documentation says the global user agent (*) is ignored by Mediapartners-Google, and the AdSense crawler page says pages roboted out with User-agent: * are still crawled. Only a group that names Mediapartners-Google or Google-Display-Ads-Bot blocks them.
How long after fixing access will AdSense see my site?
Google does not give a fixed delay for the review crawl. Its crawler page says changes to a page may take up to 1 or 2 weeks to be reflected in its index, and the site review itself usually takes a few days but can take 2-4 weeks. Request the review only once you have confirmed the fix from outside your network.
Can I keep Cloudflare Bot Fight Mode on while applying for AdSense?
Be careful. Cloudflare's documentation says Bot Fight Mode cannot be bypassed with WAF custom rules or Page Rules, so you cannot carve out an exception for a specific crawler there. If your logs show Google requests being challenged, turn it off during review or use a configuration that supports skip rules.
Sources
- What to do when your site is not ready to show ads · Google AdSense Help
- Fix AdSense crawler issues · Google AdSense Help
- About the AdSense ads crawler · Google AdSense Help
- Google's special-case crawlers · Google for Developers
- Bot Fight Mode · Cloudflare Docs
Related guides
- AdSense approval checklist: everything to verify before you request a site review
A pre-submission AdSense checklist mapped to Google's eligibility rules, setup tasks and site review reasons, so you fix issues before you request review.
- AdSense site statuses explained: what each status on the Sites page means and what to do
What "Requires review", "Getting ready", "Needs attention" and "Ready" mean on the AdSense Sites page, plus the ads.txt column, and the exact next step for each.
- ads.txt "Not found" or "Unauthorized": a technical checklist for redirects, HTTPS, status codes and robots.txt
Diagnose ads.txt Not found and Unauthorized statuses: root vs www, HTTP vs HTTPS, redirect rules, 404 and soft 404, robots.txt, content type, with curl tests.
- Getting a static or Jamstack site approved for AdSense
Getting a static or Jamstack site approved for AdSense: where the code and ads.txt go, why platform subdomains and previews trip reviews, and what stays the same.