Researching ads.txt adoption with the Tranco top sites list: a tested, reproducible scan
Key takeaways
- Tranco is a research-oriented ranking of the top 1 million sites, built by averaging several providers over 30 days to resist manipulation, and each list has a permanent ID for reproducibility.
- Lists are free to download as CSV, updated daily, and also available in Google BigQuery and through a Python package.
- In our scan of the top 500 domains of list V38GN on November 18, 2026, 75 returned a valid-looking ads.txt, and 64 of those had a google.com DIRECT line.
- Most top-ranked domains are infrastructure (CDNs, APIs, trackers), so 361 of 500 fetches failed; filter the list before drawing conclusions about publishers.
- The IAB Tech Lab publishes a reference ads.txt crawler, but it has no licence file and was last updated in 2024.

Tranco is a free, research-grade ranking of the top million websites. Pair it with a short script and you can measure ads.txt adoption yourself: pick the top N domains, fetch each /ads.txt, and count files that contain Google lines. This guide gives a tested Python scan, the real results we got for the top 500, and the traps that make naive numbers misleading.
What is Tranco, and why do researchers use it?
The Tranco website describes it as “a research-oriented top sites ranking hardened against manipulation”. It was created by researchers at KU Leuven and other universities, and the method was published at NDSS 2019 (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczyński and Joosen).
Key facts from the methodology page:
- Several providers combined. The default list combines Chrome UX Report (CrUX), Cloudflare Radar, Farsight (default list only), Majestic and Cisco Umbrella rankings. Alexa and Quantcast were used earlier and have been deprecated.
- Averaged over 30 days with the Dowdall rule (a rank-weighted score), which makes the list more stable and harder to game.
- One million domains by default, with an option that includes subdomains.
- A permanent ID per list. The list we used, generated on November 18, 2026, has the ID
V38GN(list page). Cite the ID so anyone can reproduce your work.
The site states that lists are updated daily by 0:00 UTC, downloadable as CSV (top-1m.csv.zip, and top-1m-incl-subdomains.csv.zip), available in Google BigQuery as tranco.daily.daily, and accessible from Python and Go packages.
Licences and terms
Check this before you publish or sell anything built on the list. The Tranco website does not state a licence for the combined list itself, and the DistriNet/tranco-list repository has no licence file. The site lists licences for some sources, for example Majestic under CC BY 3.0, CrUX under CC BY-SA 4.0 and Cloudflare Radar under CC BY-NC 4.0 (non-commercial). The Python package is MIT-licensed.
Step 1: get a list
With the Python package (pip install tranco), as shown in its README:
from tranco import Tranco
t = Tranco(cache=True, cache_dir='.tranco')
latest = t.list() # latest daily list
print(latest.list_id) # cite this ID
top = latest.top(1000) # first 1,000 domains
print(latest.rank("wikipedia.org"))
You can also request a list for a past date with t.list(date='YYYY-MM-DD'). Or download the CSV from the website: each line is rank,domain.
Step 2: scan ads.txt files
This is the script we ran. It reads the CSV, fetches https://<domain>/ads.txt with 10 parallel workers and a 10-second timeout, and classifies the answer:
"""adstxt_scan.py: python adstxt_scan.py top-1m.csv 500 results.json"""
import csv, sys, json, ssl, urllib.request, concurrent.futures
UA = "example.com ads.txt research (https://www.example.com/contact)"
csv_path, N, out = sys.argv[1], int(sys.argv[2]), sys.argv[3]
domains = [row[1] for row in csv.reader(open(csv_path))][:N]
ctx = ssl.create_default_context()
def fetch(d):
try:
req = urllib.request.Request(f"https://{d}/ads.txt", headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=10, context=ctx) as r:
body = r.read(2_000_000).decode("utf-8", "replace")
status = r.status
except Exception as e:
return {"domain": d, "ok": False, "err": type(e).__name__}
lines = [l.split("#")[0].strip() for l in body.splitlines()]
recs = [l for l in lines if l.count(",") >= 2]
google = [l for l in recs if l.lower().startswith("google.com")]
valid = status == 200 and "<html" not in body[:500].lower() and len(recs) > 0
return {"domain": d, "ok": True, "status": status, "records": len(recs),
"google_lines": len(google),
"google_direct": sum("direct" in l.lower() for l in google), "valid": valid}
with concurrent.futures.ThreadPoolExecutor(10) as ex:
res = list(ex.map(fetch, domains))
json.dump(res, open(out, "w"), indent=1)
v = [r for r in res if r.get("valid")]
print("domains", len(res), "valid", len(v),
"with google.com", sum(r["google_lines"] > 0 for r in v),
"with google.com DIRECT", sum(r["google_direct"] > 0 for r in v),
"fetch errors", sum(not r["ok"] for r in res))
Replace the User-Agent with your own name and a contact URL. A descriptive User-Agent lets site owners see who is crawling and why.
What counts as “valid” here is deliberately simple: HTTP 200, not an HTML page, and at least one line with three or more comma-separated fields. It is a filter for “looks like an ads.txt file”, not a syntax validator. For strict checks, use a dedicated validator as described in open-source ads.txt validators.
Real results: top 500 of list V38GN
We ran the script on November 18, 2026 against the first 500 domains of list V38GN:
| Result | Domains |
|---|---|
| Valid-looking ads.txt | 75 |
… with at least one google.com line |
67 |
… with a google.com DIRECT line |
64 |
| Returned 200 but not a valid ads.txt (HTML page, empty file) | 63 |
| Other HTTP status (202) | 1 |
| Fetch errors (HTTP errors 189, connection errors 168, timeouts 4) | 361 |
The median valid file had 120 records. Examples of domains with a valid file included youtube.com, twitter.com, nytimes.com, cnn.com, bbc.com, theguardian.com, weather.com and wordpress.com.
These numbers describe one scan, on one day, from one network. They will differ on your run.
How to read these numbers without fooling yourself
The denominator matters most. “75 out of 500” does not mean 15% of big websites use ads.txt. The top of any traffic ranking is full of domains that people never visit directly: CDNs, API hosts, ad and analytics endpoints, update servers. Most have no web page at the bare domain, which explains many of the 361 failures. Before you calculate adoption, filter the list to domains that serve pages, for example by checking that the home page returns HTML.
A quick pre-filter keeps only domains whose home page returns HTML:
def serves_html(d):
try:
req = urllib.request.Request(f"https://{d}/", headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=10, context=ctx) as r:
return "text/html" in r.headers.get("Content-Type", "")
except Exception:
return False
Run it first, then compute adoption over the domains where it returns True, and report both counts.
One URL is not the whole specification. The ads.txt specification describes where crawlers should look and how to handle redirects and subdomains; our script only tried https://<domain>/ads.txt and followed redirects. A domain that serves its file only on www. or only over http may be counted as missing. See ads.txt syntax for the format, and the IAB Tech Lab text linked from IAB Tech Lab specs for the crawler rules.
Blocking is common. Some large sites reject scripts with unknown User-Agents or rate-limit cloud IP ranges. Record error types, as the script does, rather than treating every failure as “no ads.txt”.
Google lines are not proof of AdSense. A google.com line can belong to AdSense, Ad Manager or other Google products, and a line can be stale. Use it as an adoption signal, not as a list of AdSense publishers.
The IAB Tech Lab reference crawler
The IAB Tech Lab publishes adstxtcrawler, a Python reference implementation that reads a list of target domains, fetches their ads.txt files and stores the parsed records in SQLite (usage: adstxt_crawler.py -t <targets file> -d <database>). It is useful to see how the spec authors intended crawling to work. Two caveats: the repository has no licence file, so reuse rights are unclear, and its last push dates from June 2024.
Going further
- Compare over time. Keep the JSON output and rerun monthly with a new list ID. Adoption trends are more meaningful than one snapshot.
- Scale with public crawls. For millions of pages without crawling them yourself, HTTP Archive already records ads.txt presence per page; see researching ad tech with HTTP Archive.
- Check sellers.json too. Parse the seller IDs in each file and look them up in the ad system’s sellers.json to see how often lines point to real, active accounts.
Common mistakes
- Publishing a percentage without the list ID and date. Nobody can check it.
- Counting failures as “no ads.txt”. Separate errors from real 404s.
- Scanning too hard. Thousands of parallel requests can look like an attack. Keep concurrency low.
- Ignoring the licence question when using the list in a commercial product.
- Treating “valid-looking” as valid. Use a real parser for syntax claims.
Open-source tools mentioned
| Tool | Licence |
|---|---|
| Tranco Python package | MIT |
| IAB Tech Lab adstxtcrawler (no licence file in the repository) | NOASSERTION |
Frequently asked questions
Why use Tranco instead of another top sites list?
Tranco was designed for research. It averages several providers over 30 days, which makes it harder to manipulate and more stable from day to day than any single source, and every list gets a permanent ID so others can reproduce your results. Its authors published the method at NDSS 2019. For any study, cite the list ID you used.
Is it legal and polite to scan other sites' ads.txt files?
ads.txt files are public by design so that buyers can crawl them. Keep the scan polite anyway: a descriptive User-Agent with a contact URL, a few parallel requests at most, a timeout, and one request per domain. Do not scan the full million from a home connection, and respect any block or rate limit you hit. This is not legal advice.
Why did so many fetches fail in the scan?
The top of the list contains many domains that are not websites people read: CDNs, API hosts, ad and analytics endpoints, and DNS-heavy services. Many have no web server at the bare domain or reject unknown clients. Our script also tried only https on the bare domain, without the www subdomain or http fallback that the ads.txt specification describes.
Can I use Tranco data commercially?
The Tranco website does not state a licence for the combined list, and it lists the licences of some underlying sources, including Cloudflare Radar data under CC BY-NC 4.0, which is non-commercial. If your use is commercial, contact the Tranco team and check each source's terms before relying on the data.
Sources
- Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation · Tranco (DistriNet, KU Leuven)
- Tranco methodology · Tranco
- Tranco list V38GN · Tranco
- tranco-python-package README · DistriNet on GitHub
- adstxtcrawler · IAB Tech Lab on GitHub
Related guides
- Validating ads.txt yourself: curl checks, the IAB Tech Lab open-source crawler and scripted audits
Check your ads.txt without third-party validators: curl tests that mimic a crawler, the IAB Tech Lab open-source crawler and its limits, and a short audit script.
- ads.txt syntax for AdSense, line by line: pub ID, DIRECT, TAG ID, comments and variables
The AdSense ads.txt line field by field: google.com, your pub ID, DIRECT, f08c47fec0942fa0, plus comments, variables and the formatting errors that break parsing.
- HTTP Archive and the Web Almanac: open data on ad scripts, ads.txt and page weight
Use HTTP Archive and the Web Almanac to measure ads.txt adoption, ad tech usage and third-party weight across millions of sites, with SQL you can adapt.
- IAB Tech Lab and IAB Europe specifications: what an AdSense publisher needs to know
Which industry specs matter to an AdSense publisher: ads.txt 1.1, sellers.json, SupplyChain, OpenRTB, TCF and GPP, with current versions and what each asks of you.