ads.txt & sellers.json

ads.txt "Not found" or "Unauthorized": a technical checklist for redirects, HTTPS, status codes and robots.txt

Last updated Published By the editorial team6 min5 sources

Key takeaways

  • Google starts ads.txt crawling at the root domain: a file at www.domain.com/ads.txt is only crawled if domain.com/ads.txt redirects to it.
  • Google crawls ads.txt on both HTTP and HTTPS; if one scheme returns 404, previously crawled entries can be purged even though the other scheme works.
  • The ads.txt response must have HTTP status 200 OK; Google ignores a response whose status code says the file was not found, even if a body is returned.
  • Google supports multiple redirects that stay within the original root domain and a single redirect to a destination outside it.
  • A robots.txt rule such as Disallow: /ads, or one that disallows Googlebot, can make Google's crawler ignore the ads.txt file.
Magnifying glass lying on an antique wooden table
Image: Magnifying glass on antique table,Stéphane Magnenat,CC BY-SA 4.0,via Wikimedia Commons.

AdSense shows “Not found” when its crawler cannot fetch a valid ads.txt from your root domain, and “Unauthorized” when it fetches one that lacks your publisher ID. Not found is almost always a delivery problem: the bare domain does not serve or redirect to the file, one of HTTP or HTTPS returns 404, the status is not 200, or robots.txt blocks the path. Test each case with curl, not just a browser.

Not found vs Unauthorized: which problem do you have?

Status Meaning Where the fault usually is
“Not found” No ads.txt file detected Server, redirects, DNS, robots.txt, CMS routing
“Unauthorized” File found, your publisher ID missing The content of the file

For “Unauthorized”, the fix is in the text: add the line from the “Ads.txt snippet” in AdSense, and check for a ca-pub- prefix or a typo (see ads.txt syntax). The rest of this guide covers “Not found”, where the file often looks fine in a browser but Google’s crawler sees something else.

Which URL does Google actually crawl?

Google’s Ensure your ads.txt files can be crawled page sets three rules:

  1. Root domain first. “Ads.txt crawling will start at the root domain, and the root domain needs to return from, or redirect to, the ads.txt file.” The root domain is the registrable domain (the IAB spec defines it as the public suffix plus one label), so for blog.example.co.uk it is example.co.uk.
  2. Both schemes. “The Google crawler attempts to crawl ads.txt files on both HTTP and HTTPS.” Each must either serve the file or redirect to the scheme that does.
  3. Status 200. If the status code “indicates the file was not found (e.g., status code 404)”, then “the response will be ignored” and “the file will be considered non-existent”, even if a body is returned.

How do I test it like a crawler?

A browser follows redirects silently, reuses cached responses and may show a page your server returns with an error code. Use curl from a terminal:

curl -sIL http://example.com/ads.txt
curl -sIL https://example.com/ads.txt
curl -sIL http://www.example.com/ads.txt
curl -sIL https://www.example.com/ads.txt

-I shows headers only, -L follows redirects, -s hides the progress bar. For each command, read the chain of HTTP/ status lines and location: headers. A healthy result ends with 200 and a content-type: text/plain header. Then read the body once:

curl -sL https://example.com/ads.txt

You should see your plain-text lines, not HTML.

What redirects are allowed?

The Ads.txt FAQs and the IAB ads.txt 1.1 specification agree:

Redirect Allowed?
example.com/ads.txt to www.example.com/ads.txt Yes (within the root domain)
http:// to https:// on the same host Yes
example.com/ads.txt to sub.example.com/ads.txt Yes
example.com/ads.txt to example.com/page/ads.txt Yes (Google’s own example)
Several hops, all within example.com Yes
One hop to other-domain.com/ads.txt Yes, a single hop outside the root domain
other-domain.com/ads.txt then redirects again No: the spec says treat it as an error

The spec lists 301, 302 and 307 as the redirect codes crawlers should follow. Redirects done with JavaScript or a meta refresh are not HTTP redirects; a crawler reading /ads.txt sees an HTML page instead.

What are the common causes of “Not found”?

The file only exists on www (or only on the bare domain)

If your site lives on www.example.com and the bare domain has no web server, or serves a parking page, Google never reaches the file. Google’s instruction: “If you need to include ‘www’ to access your site, then you need to set up a redirect from domain.com/ads.txt to www.domain.com/ads.txt.” A sitewide bare-to-www redirect works if it preserves the path. A redirect that sends every request to the home page does not.

HTTP or HTTPS returns 404

Google gives this exact scenario: if there is only an ads.txt file for HTTP, the HTTPS crawl returns 404 and “the previously crawled entry will be purged even though an ads.txt file is crawled via HTTP.” Redirect the failing scheme to the working one. On an HTTPS site, the cleanest setup is a permanent redirect from every http:// URL to https://.

The status code is not 200

Two variants:

  • Hard 404 with a body. Some setups return a custom error page with status 404. Google ignores it.
  • Soft 404. Many CMSs and single-page apps answer every unknown URL with status 200 and the home page or an app shell. The status looks fine, but the content is HTML. The IAB spec says the content type should be text/plain and that “all other Content-types should be treated as an error and the content ignored”.

The fix in both cases is to make /ads.txt a real static file (or a route that returns plain text) that takes precedence over the CMS fallback. Host-specific steps are in deploying ads.txt on static hosts and CDNs and ads.txt on WordPress without a plugin.

robots.txt blocks it

Google’s examples: with User-agent: * and Disallow: /ads, “the ads.txt file will be ignored by crawlers that respect the robots.txt standard”, because /ads also matches /ads.txt. With User-agent: Googlebot and Disallow: /, it “will be ignored by the Google crawler.” Google’s suggested fixes:

User-agent: *
Allow: /ads.txt
Disallow: /ads

or change Disallow: /ads to Disallow: /ads/. Google adds that robots.txt changes “may not be reflected until the crawlers attempt to visit your site again.”

Firewalls, bot protection and geo-blocking

If a web application firewall, a bot-protection challenge, HTTP authentication or a country block intercepts the request, the crawler receives a challenge page or an error instead of the file. Your browser may pass the check while an automated client does not. The IAB spec says a 401 (restricted) response means the ad system “should seek direct contact with the site”. Exclude /ads.txt from challenges and authentication. The same issues block site review, as described in site down or unavailable.

Wrong location or name

The file must be at the root path, exactly /ads.txt, lowercase. example.com/blog/ads.txt or example.com/ads.txt/ are not the same URL. If your site is on a subdomain, see ads.txt and subdomains.

Too early

Google says changes “may take a few days”, and “up to a month” for sites with few ad requests. Fix delivery first, then use “Check for updates” on the site’s page in AdSense and wait.

Diagnostic table

curl result Diagnosis Fix
200, text/plain, your lines Delivery OK Check content, wait for recrawl
200, text/html, page markup Soft 404 or CMS route Serve a real static file
404 on HTTPS only Missing on one scheme Redirect HTTP and HTTPS to the same file
404 on bare domain, 200 on www No root redirect Redirect example.com/ads.txt to www
301 to / (home page) Path dropped by redirect Preserve the path in the redirect rule
403 or 503 with a challenge page Firewall or bot protection Exempt /ads.txt
401 Password protection Remove auth on /ads.txt
Connection error on bare domain No DNS record or server Add DNS and a redirect for the apex
Two external redirects Chain leaves the root domain twice Point directly to the final file

What if I don’t control the root domain?

If the top-level domain belongs to a platform, the Ads.txt troubleshooter says you can ask the owner to host the file on your behalf, or “to set up a subdomain referral for you”. You cannot fix that from your own subdomain alone.

When curl shows a clean 200 text/plain on all four URL variants and the file contains your exact line, the remaining factor is time. For background on what the alert costs while you wait, see the Earnings at risk alert.

Frequently asked questions

My site uses www. Where should ads.txt be?

It can live at www.example.com/ads.txt, but Google starts crawling at the root domain, so example.com/ads.txt must redirect to it. Google's crawl help says an ads.txt file on www.domain.com/ads.txt will only be crawled if domain.com/ads.txt redirects to it. A sitewide redirect from the bare domain to www usually covers this, as long as it keeps the /ads.txt path.

Can ads.txt redirect to another domain?

Once. Google supports a single HTTP redirect to a destination outside the original root domain, for example example1.com/ads.txt to example2.com/ads.txt. Redirects that stay inside the root domain (to www, to a subdomain or to another path) can be chained. The IAB specification says that if the third-party location redirects again, it should be treated as an error.

What is the difference between Not found and Unauthorized?

"Not found" means Google found no usable ads.txt file on the site. "Unauthorized" means it found a file, but your publisher ID is not in it. The fixes differ: for Not found, check the URL, redirects, status code and robots.txt; for Unauthorized, add or correct the AdSense line, usually a missing line, a typo or a ca-pub- prefix.

Does robots.txt affect ads.txt?

Yes. Google warns that the ads.txt file may be ignored if robots.txt disallows the URL path where it is posted or disallows Google's crawler. A rule like Disallow: /ads also matches /ads.txt because robots.txt rules are prefix matches. Change it to Disallow: /ads/ or add Allow: /ads.txt.

Sources

  1. Ensure your ads.txt files can be crawled · Google AdSense Help
  2. Ads.txt FAQs · Google AdSense Help
  3. Ads.txt guide · Google AdSense Help
  4. Ads.txt troubleshooter · Google AdSense Help
  5. Ads.txt Version 1.1 · IAB Tech Lab