How Much Are Bots Costing You? IAB Tech Lab Wants Content Owners To Find Out

Recorded: May 27, 2026, 1:21 p.m.

Original

Summarized

How Much Are Bots Costing You? IAB Tech Lab Wants Content Owners To Find Out | AdExchanger

image/svg+xml:

Topics
Latest
Marketers
Agencies
Publishers
Technology
Platforms
Identity
Measurement
Data Privacy
Artificial Intelligence
CTV
Commerce
AdExplainer
Exclusive Report
Daily News Roundup

Opinion
All Columns
Data-Driven Thinking
On TV & Video
The Sell Sider
Content Studio
Comic
Contributor Guidelines

About Us
Advertise
Newsletter
AdExchanger Advisory Board
About Us
Contact Us

Events
Programmatic AI Las Vegas
Programmatic I/O 2026
AdExchanger Awards
Webinars
All Events
Network Events

Podcasts
AdExchanger Talks
The Big Story
Inside the Stack

Programmatic AI

Programmatic I/O

Become an AdHero

Topics
Latest
Marketers
Agencies
Publishers
Technology
Platforms
Identity
Measurement
Data Privacy
Artificial Intelligence
CTV
Commerce
AdExplainer
Exclusive Report
Daily News Roundup

Opinion
All Columns
Data-Driven Thinking
On TV & Video
The Sell Sider
Content Studio
Comic
Contributor Guidelines

Events & Awards
Programmatic AI Las Vegas
Programmatic I/O 2026
AdExchanger Awards
Webinars
All Events
Network Events

Podcasts
AdExchanger Talks
The Big Story
Inside the Stack

Subscribe Free
Sign Up

About Us
Advertise
Newsletter
AdExchanger Advisory Board
About Us
Contact Us

CONNECT

Home AI How Much Are Bots Costing You? IAB Tech Lab Wants Content Owners To Find Out

AI
How Much Are Bots Costing You? IAB Tech Lab Wants Content Owners To Find Out By Joanna Gerber

Wednesday, May 27th, 2026 – 8:30 am
SHARE:

The bots are hungry for content.
The content owners are hungry for monetization.
And the IAB Tech Lab is working on new recommendations that will – hopefully – help everyone get fed.
On Wednesday, the Tech Lab released its guidance for bot management strategies, which is open for public comment until June 26.
The purpose of the public comment period is to invite content owners to provide feedback and ask questions so that the final document can be as helpful as possible.
Because right now, everyone’s strategy looks a little bit different – and a lot of them aren’t as fruitful as they might sound.
For example, blocking crawlers entirely isn’t an effective strategy, said Hillary Slattery, the Tech Lab’s senior director of project management for programmatic, since doing so blocks everything, including useful bots like ads.txt crawlers that verify which companies are authorized to sell inventory.
Content owners (and this means anyone who owns content on the web, not just publishers, Slattery emphasized) that block indiscriminately eventually see their programmatic demand disappear “because their supply chains couldn’t be validated,” she said.
The problem with bot traffic “is not the existence of crawling,” according to the draft guidelines. Rather, the problem is that many content owners have “little to no understanding” of how much it costs to deliver their content to bots, nor how the content is being “accessed, interpreted or used.”
‘Crawl Me Maybe’
The IAB Tech Lab has spent the better part of the past year fighting the good fight for publisher monetization standards – and it hasn’t been easy.
Slattery joked that her nickname for the Content Monetization Protocols Working Group, which is the body that devised the draft guidelines, is “crawl me maybe,” since every online publisher seems to have a different perspective on how their content should or shouldn’t be accessible to bots.
The new document provides ways for content owners to better understand how their content is being used, and what they’re receiving in return.
When asked if the guidance includes any specific suggestions regarding what content owners should and should not permit in terms of scraping, Slattery didn’t mince words.
“Hello no,” she said, “that’s between the content owners and their gods.”
What the IAB Tech Lab does want to drive home to content owners, however, is the importance of understanding how much it costs their businesses to deliver content to bots.
At a minimum, content owners need to know “which bot is crawling your content, how often and how much it’s costing you,” said Slattery – all of which can be answered by a company’s content delivery network (CDN), like Fastly or Akamai. But the caveat – and it’s a big one – is that CDNs aren’t generating those reports unless they’re asked.Their clients need to take the initiative to ask for a report that lists which bots are crawling their content, how often and how much it’s running them, said Slattery.
Because once you understand who’s hitting your site and how often, the next question is how much you’re paying to serve each of those visits.
It costs money for a website or application to deliver content, and an AI crawler doesn’t need to access a full webpage with formatting the same way a human does. One way content owners can save money is by delivering a plain-text file to bots, which is equally effective and cheaper than a page aesthetically built for a human, which might include headers, footers and imagery.
Delivering the full version of a webpage to a bot, rather than a simplified text file, isn’t just a waste of budget, but also of energy, Slattery said. It’s “melting the goddamn planet for no reason.”
Crawl or nothing thinking
Once content owners have a report from their CDN, they need to understand the “value exchange,” said Slattery, as in how much bang they’re getting for their buck when it comes to content scraping.
Claude, for instance, has an extremely stark crawl-to-refer ratio, with roughly 8,800 crawls for every referral it drives as of April – which is pretty bad, but looks great compared to the 65,000:1 ratio Business Insider reported back in January.
But some content owners might not care about referrals, and are fine with getting scraped under any circumstances as long as they get cited or paid or whatever metric matters most to them.
And if referrals are their primary focus, maybe they’d opt to turn away Claude’s crawlers but allow those coming from DuckDuck Go, which has an impressive 1.5:1 crawl-to-refer ratio.
Companies need to find the “strategic trade-off” that best suits their goals and priorities, said Slattery.
“Don’t turn everything off,” she added. “It’s not a light switch; it’s a mixing board.”

Tagged in:

//
Claude

//
content monetization

//
DuckDuckGo

//
Hillary Slattery

//
IAB Tech Lab

//
Publisher Monetization

//
scraping

//
web crawlers

Next In AI

AI Needs Humans In The Lead, Not Just In The Loop, Says JPMorgan Chase Programmatic Lead

Related Stories

PODCAST: AdExchanger Talks
When AI Meets Media Quality

Daily News Roundup
Scrapers Gonna Scrape; No More Fast-Forwarding The Ads, DVR Friends

AI
Can An AI Solution Fix Misaligned Marketing Orgs?

Must Read

CTV
Why Critics Say Email-Based IDs Don’t Work For CTV

Email targeting in CTV has a credibility problem as buyers and sellers question whether one-to-one identity even fits a channel built for broader reach.

PODCAST: AdExchanger Talks
How ‘Wrapped’ Insights Become Audience Segments

How does Spotify translate quirky Wrapped labels, like “divorced dad hipster,” into ad audiences? And is AI-generated content safe for brands? Spotify’s Global Head of Ad Product Katie English weighs in.

Marketers
Pirated Sports Streams Are Warping TV’s Most Important Ratings

Although tides of ad revenue flow based on the ratings of certain tentpole TV events, a new crop of scammers now operate illicit sports livestreaming rings, and there’s almost nothing broadcasters can do about it.

CTV roundup
AI Is Redefining Premium Content – Which May Not Be A Good Thing

At AdExchanger’s Programmatic AI conference, media experts discussed how the rise of AI-generated content is changing the industry’s understanding of “premium” content.

PODCAST: The Big Story
Prog AI Live: AI’s Slippery Slop

Recorded live in Las Vegas at Prog AI, the AdExchanger team tackles a tricky question: As AI floods the feed with chaotic, addictive content and people engage with it, what does “premium” even mean anymore?

AdExplainer
The Programmatic Auction Is Changing In Real Time – Here’s How

Two decades after the first RTB auction, programmatic is more complex than ever – and that’s before you even consider generative AI.

Popular

AdExplainer
The Programmatic Auction Is Changing In Real Time – Here’s How

Two decades after the first RTB auction, programmatic is more complex than ever – and that’s before you even consider generative AI.

AI
AI Needs Humans In The Lead, Not Just In The Loop, Says JPMorgan Chase Programmatic Lead

Automation has its place, but good marketing still needs “heart and science,” according to JPMorgan Chase’s Melissa Bonnick. Humans aren’t going anywhere.

Marketers
Pirated Sports Streams Are Warping TV’s Most Important Ratings

Publishers
Microsoft To Publishers: Don’t Block The AI Bots

Rather than fighting the rising tide of AI search engines and agentic tools, Microsoft AI’s Nikhil Kolar says publishers and retailers should license access to their sites and create content that speaks directly to AI crawlers.

CTV roundup
AI Is Redefining Premium Content – Which May Not Be A Good Thing

At AdExchanger’s Programmatic AI conference, media experts discussed how the rise of AI-generated content is changing the industry’s understanding of “premium” content.

Join the AdExchanger Community
Join Now

Your trusted source for in-depth programmatic news, views, education and events.
AdExchanger is where marketers, agencies, publishers and tech companies go for the latest information on the trends that are transforming digital media and marketing, from data, privacy, identity and AI to commerce, CTV, measurement and mobile.

NEXT EVENT
Programmatic AI
May 18-20, 2026Park MGM, Las Vegas
Learn More

ABOUT ADEXCHANGER
About Us
Advertise
Contact Us
Events
Subscribe
RSS
Cookie Settings
Privacy & Terms
Accessibility
Diversity, Equity, Inclusion & Belonging

CONNECT

The IAB Tech Lab has released guidance for content owners regarding bot management strategies, aiming to help them understand the costs associated with delivering content to automated systems. This guidance is intended to address the disparity between the demand for monetization by content owners and the consumption of content by bots. A key finding addressed by the laboratory is that indiscriminate blocking of crawlers is an ineffective strategy because it simultaneously blocks valuable entities, such as ads.txt crawlers, which are necessary for validating inventory and ensuring proper distribution. Hillary Slattery, the Tech Lab’s senior director of project management for programmatic, noted that blanket blocking leads to the disappearance of programmatic demand because the supply chain cannot be validated.

The fundamental issue identified in the context of bot traffic is not merely the existence of crawling, but the lack of understanding among content owners regarding the cost incurred to deliver content to bots, as well as how that content is accessed, interpreted, or utilized. The new documentation seeks to provide content owners with the necessary information to assess this cost. To achieve this transparency, content owners need to know which bots are accessing their content, the frequency of access, and the associated cost. While content delivery networks like Fastly or Akamai possess this information, they will not report it proactively; content owners must initiate the request for reports detailing which bots are crawling their content, their frequency, and the resulting costs.

Furthermore, the cost of delivery should be analyzed in relation to the value exchange derived from the scraping activity. Content owners must assess how much return they receive for the content being accessed. For example, this involves examining metrics such as the crawl-to-refer ratio. The text cites an example where AI models like Claude exhibit a stark crawl-to-refer ratio, whereas other search behaviors might present different ratios. Content owners must determine which metrics—such as referrals, citations, or other relevant measures—are most critical to their business goals. This analysis necessitates finding a strategic trade-off, recognizing that some content owners may be content with scraping occurring as long as they receive citations or payments according to their priorities.

To optimize delivery costs, content owners can implement strategies that reduce the data volume sent to crawlers. Delivering a plain-text file to bots is proposed as a more cost-effective alternative compared to serving a fully formatted webpage, which includes aesthetic elements like headers, footers, and imagery. This approach saves both budget and energy consumption by eliminating the unnecessary transmission of full visual content. Ultimately, content owners are advised against an all-or-nothing approach; Slattery suggests that managing bot traffic should be treated like a mixing board, allowing for a nuanced strategy based on specific goals rather than a simple on or off switch.