ads.txt & sellers.json

Validating ads.txt yourself: curl checks, the IAB Tech Lab open-source crawler and scripted audits

Last updated Published By the editorial team6 min5 sources

Key takeaways

  • AdSense's own status in the Sites page is the authoritative check for your account; self-validation tells you whether the file is reachable and correctly formed before Google recrawls.
  • A curl request with -sIL shows the redirect chain, final status code and Content-Type, which are the delivery conditions Google's crawl help and the IAB specification describe.
  • The IAB Tech Lab adstxtcrawler is an open-source reference crawler under the 2-clause BSD license that reads a list of domains and stores ads.txt records in a SQLite database.
  • The adstxtcrawler README calls it an example prototype suitable only for very modest production usage; its code imports Python 2 modules and predates the OWNERDOMAIN and MANAGERDOMAIN variables.
  • A short Python 3 script using only the standard library can check many domains at once for a 200 response, the Content-Type and the presence of your AdSense line.
Source code displayed in a text editor
Image: Source code,Micke,CC BY-SA 3.0,via Wikimedia Commons.

To validate ads.txt yourself, check delivery and content separately. For delivery, fetch the file with curl -sIL on HTTP, HTTPS, the bare domain and www, and confirm a final 200 with text/plain. For content, look for your exact google.com, pub-..., DIRECT, f08c47fec0942fa0 line. For many domains, use a script or the IAB Tech Lab’s open-source crawler, keeping its limits in mind.

What should a validation actually check?

Two documents define what a consumer of ads.txt expects: Google’s Ensure your ads.txt files can be crawled and the IAB Tech Lab ads.txt 1.1 specification. Combined, they give this checklist:

Check Requirement Source
Location /ads.txt on the root domain, or a redirect from it Google crawl help, IAB spec
Schemes Google tries both HTTP and HTTPS Google crawl help
Status Final response 200; a 404 means no file Google crawl help, IAB spec
Redirects 301, 302, 307; any number inside the root domain, one hop outside IAB spec, Google FAQ
Content-Type text/plain; other types “should be treated as an error” IAB spec
robots.txt Must not block the path or Google’s crawler Google crawl help
Content Your line with your pub- ID and DIRECT Google’s Ads.txt guide

None of these replaces the status AdSense shows under “Sites”. Self-validation tells you whether a problem is on your side before you wait days for a recrawl.

How do I check one site with curl?

Run the four URL variants. -s silences progress, -I requests headers only, -L follows redirects:

curl -sIL http://example.com/ads.txt
curl -sIL https://example.com/ads.txt
curl -sIL http://www.example.com/ads.txt
curl -sIL https://www.example.com/ads.txt

Each output shows one header block per hop. Read the HTTP/ status lines and location: headers in order, and check that the last block has status 200 and content-type: text/plain. A shorter summary of the final result:

curl -sL -o /dev/null -w '%{http_code} %{content_type} %{url_effective} %{num_redirects}\n' https://example.com/ads.txt

Then check the content for your publisher ID:

curl -sL https://example.com/ads.txt | grep -i 'pub-0000000000000000'

Some servers answer a HEAD request (-I) differently from a GET. If the -I result looks wrong but the site seems fine, repeat with the -w command above, which uses GET. Interpreting each failure pattern is covered in ads.txt Not found or Unauthorized.

What is the IAB Tech Lab adstxtcrawler?

adstxtcrawler is the IAB Tech Lab’s open-source reference crawler, written in Python. Its README describes it as “an example crawler for ads.txt files” that “saves them to a SQLite DB table”, and states that the license is the 2-clause BSD license.

Setup, as documented in the README:

sqlite3 adstxt.db < adstxt_crawler.sql
./adstxt_crawler.py -t target_domains.txt -d adstxt.db
echo "select * from adstxt;" | sqlite3 adstxt.db

The targets file lists one domain or URL per line. Options are -t (targets file), -d (database), -v (verbosity, repeatable) and -p (thread pool size, default 4). The Python dependencies listed in requirements.txt are requests and pyopenssl.

Reading adstxt_crawler.py shows what it does and does not check:

Behavior What the code does
URL fetched http://{host}/ads.txt only; redirects are followed by the requests library
Request headers User-Agent AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler, Accept: text/plain
Timeout 5 seconds
Accepted responses Status 200 only
HTML detection Rejects the body if it contains the strings html, body, div or span anywhere, case-insensitive
Content-Type Not checked
Cleanup Removes a byte order mark and every character outside printable ASCII except line breaks, which also removes tabs
Delimiters Comma; # starts a comment (a tab check exists in the code, but tabs are already stripped by the cleanup)
Variables Follows subdomain= one level; stores contentproducerdomain and contentdistributordomain
ads.txt 1.1 variables OWNERDOMAIN and MANAGERDOMAIN are not stored
Output Table adstxt with SITE_DOMAIN, EXCHANGE_DOMAIN, ADSYSTEM_DOMAIN, SELLER_ACCOUNT_ID, ACCOUNT_TYPE, TAG_ID, ENTRY_COMMENT, UPDATED

What are the crawler’s limits?

The README is direct: “This is an example prototype crawler and would be suitable only for a very modest production usage.” Points to know before you rely on it:

  • Python 2 code. The README says “Python 2 or better”, but the script imports urlparse, a Python 2 module. The Python 3 import is left as a comment on the line above. Other parts, such as applying a text regular expression to encoded bytes, also fail on Python 3. Expect to patch it or run it in a Python 2 environment.
  • Maintenance. The repository is not archived; GitHub shows its last push on June 6, 2024. GitHub’s license detection shows no license for the repository, while the README states the 2-clause BSD license.
  • Strict HTML filter. Because the filter searches the whole lowercased file, a valid ads.txt with a comment containing a URL ending in .html, or a domain containing span, is rejected. Google does not document such a rule; it is a limit of this crawler.
  • No Content-Type check, which the IAB spec asks consumers to make.

It is still useful for understanding how a crawler parses the format, and for a quick database of many sites. For research on many domains, see researching ads.txt with the Tranco list.

The IAB also publishes ads.txt-parser, a Go project under the Apache-2.0 license, described as “a reference implementation of ads.txt parsing code (currently focused on app-ads.txt)”. It is a parsing library for developers, not a ready-made site checker.

How can I audit several sites with a script?

If you manage a handful of sites, a short Python 3 script using only the standard library is easier than patching the crawler. Save it as audit_ads_txt.py:

import sys
import urllib.request

SCHEME = "https"
pub_id = sys.argv[1]
for domain in sys.argv[2:]:
    url = f"{SCHEME}://{domain}/ads.txt"
    req = urllib.request.Request(url, headers={"User-Agent": "ads-txt-audit"})
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            ctype = resp.headers.get("Content-Type", "")
            final_url = resp.geturl()
            body = resp.read().decode("utf-8-sig", errors="replace")
    except Exception as err:
        print(f"{domain}\tERROR\t{err}")
        continue
    authorized = False
    for line in body.splitlines():
        record = line.split("#", 1)[0]
        fields = [f.strip() for f in record.split(",")]
        if (len(fields) >= 3 and fields[0].lower() == "google.com"
                and fields[1] == pub_id and fields[2].upper() == "DIRECT"):
            authorized = True
    print(f"{domain}\t{ctype}\t{final_url}\tauthorized={authorized}")

Run it with your publisher ID and the bare domains:

python3 audit_ads_txt.py pub-0000000000000000 example.com example.org

Each output line gives the Content-Type, the final URL after redirects, and whether a google.com, <your ID>, DIRECT record was found. HTTP errors such as 404 appear as ERROR lines. Change SCHEME to "http" to test the other scheme.

What the script does not do: it does not enforce the one-hop rule for redirects outside the root domain, does not read robots.txt and does not follow subdomain= declarations. Treat it as a quick audit, and use curl for the details.

Which results need action?

Result Meaning Next step
text/plain, authorized=True File OK for AdSense Wait for the recrawl or use “Check for updates”
text/plain, authorized=False Line missing or wrong ID Fix the content; see ads.txt syntax
text/html CMS page or soft 404 Serve a real static file
ERROR HTTP Error 404 No file on that scheme Add the file or redirect to it
ERROR with a timeout or SSL error Server, DNS or certificate problem Fix hosting first
Final URL on another domain Redirect outside the root domain Allowed once; check there is no second external hop

For subdomain setups, run the checks on the root and on each declared subdomain, as described in ads.txt and subdomains. If you use the variables from ads.txt 1.1, check them by hand as shown in OWNERDOMAIN and MANAGERDOMAIN, because neither the IAB crawler nor the script above reads them.

Open-source tools mentioned

ToolLicence
adstxtcrawlerBSD-2-Clause
ads.txt-parserApache-2.0

Frequently asked questions

If my own checks pass, will AdSense show Authorized?

Not necessarily right away, and not with certainty. Your checks confirm that the file is reachable and contains your line. AdSense decides the status from its own crawl, and Google says updates may take a few days, or up to a month for sites with few ad requests. Use "Check for updates" on the site's page in AdSense after fixing the file, then wait.

Is the IAB adstxtcrawler still usable today?

As a reference, yes; as a production tool, with care. The repository is not archived, but its last push was in June 2024, the README calls it an example prototype for very modest production use, and the code imports urlparse, a Python 2 module, so it does not run unmodified on Python 3. It also ignores the OWNERDOMAIN and MANAGERDOMAIN variables from ads.txt 1.1.

Why does a browser show my file while a crawler fails?

Browsers hide details a crawler cares about. They follow redirects silently, reuse cached copies, pass bot challenges with cookies and display pages served with error status codes. A crawler sees the raw response: a 404 status, an HTML page, a challenge or a redirect that drops the path. Testing with curl from a terminal shows those raw responses.

Should I use an online ads.txt validator?

You can, but check what it actually tests and do not paste account data into tools you do not trust. Many online checkers only fetch one URL and parse lines. The checks in this guide also cover HTTP and HTTPS, bare domain and www, status codes and Content-Type, which are the delivery issues behind most Not found statuses.

Sources

  1. Ensure your ads.txt files can be crawled · Google AdSense Help
  2. Ads.txt guide · Google AdSense Help
  3. Ads.txt Version 1.1 · IAB Tech Lab
  4. adstxtcrawler · IAB Tech Lab (GitHub)
  5. ads.txt-parser · IAB Tech Lab (GitHub)