Microsoft To Publishers: Don’t Block The AI Bots

Recorded: May 26, 2026, 1:14 p.m.

Original

Summarized

Microsoft To Publishers: Don’t Block The AI Bots | AdExchanger

image/svg+xml:

Topics
Latest
Marketers
Agencies
Publishers
Technology
Platforms
Identity
Measurement
Data Privacy
Artificial Intelligence
CTV
Commerce
AdExplainer
Exclusive Report
Daily News Roundup

Opinion
All Columns
Data-Driven Thinking
On TV & Video
The Sell Sider
Content Studio
Comic
Contributor Guidelines

About Us
Advertise
Newsletter
AdExchanger Advisory Board
About Us
Contact Us

Events
Programmatic AI Las Vegas
Programmatic I/O 2026
AdExchanger Awards
Webinars
All Events
Network Events

Podcasts
AdExchanger Talks
The Big Story
Inside the Stack

Programmatic AI

Programmatic I/O

Become an AdHero

Topics
Latest
Marketers
Agencies
Publishers
Technology
Platforms
Identity
Measurement
Data Privacy
Artificial Intelligence
CTV
Commerce
AdExplainer
Exclusive Report
Daily News Roundup

Opinion
All Columns
Data-Driven Thinking
On TV & Video
The Sell Sider
Content Studio
Comic
Contributor Guidelines

Events & Awards
Programmatic AI Las Vegas
Programmatic I/O 2026
AdExchanger Awards
Webinars
All Events
Network Events

Podcasts
AdExchanger Talks
The Big Story
Inside the Stack

Subscribe Free
Sign Up

About Us
Advertise
Newsletter
AdExchanger Advisory Board
About Us
Contact Us

CONNECT

Home Publishers Microsoft To Publishers: Don’t Block The AI Bots

Publishers
Microsoft To Publishers: Don’t Block The AI Bots By Anthony Vargas

Tuesday, May 26th, 2026 – 12:45 am
SHARE:

Microsoft AI's Nikhil Kolar chats with AdExchanger's Allison Schiff at Programmatic AI 2026 in Las Vegas.

Microsoft has one big piece of advice for publishers and retailers that are cautiously navigating the AI era: Let the bots scrape your sites.
That’s according to Nikhil Kolar, VP of publisher product at Microsoft AI, who spoke at AdExchanger’s Programmatic AI event in Las Vegas last week.
Rather than fighting against the rising tide of AI-based search engines and agentic tools, Kolar said publishers and retailers should actually be creating content that can speak directly to these bots (and thus the LLMs generating new forms of traffic). And site owners should also update their robots.txt files to avoid restrictions on crawlers that will diminish the site’s visibility to AI engines.
Currently, though, four out of five websites block traffic from AI bots and crawlers, according to Kolar. As a result, “the content or products that you have are not legible to agents.” Which means, he added, that “your business is closed” and “you get no discovery, no recommendations that you’re a part of and no demand.”

Leaning on licensing
There is a clearly self-serving angle for Microsoft in pitching publishers on the idea that they ought to open up their sites to AI crawling. After all, Microsoft has its own AI models and AI chatbot searches, and it needs publisher data to inform their outputs.
But Microsoft also wants publishers to receive fair value for their data, Kolar said. He pointed to Microsoft’s Publisher Content Marketplace as a sign of good faith.
The initiative, which was announced in February, facilitates licensing agreements between publishers and AI developers. The initial focus was for publishers to monetize by licensing their content or data to be used by Microsoft Copilot. But Microsoft has since opened the ecosystem to other AI firms, with Microsoft acting as the clearing house for licensing the publisher data.
AdExchanger also spoke with Kolar after his presentation, and he pointed to a distinction between using publisher data for “training” and using it for “grounding” AI models. Simply put, “training” describes establishing the deep initial data pool that underlies an AI model, which is pulled from the vast swathes of content that are already been published online. But “grounding” refers to pulling data from current, trusted sources through more sophisticated model context protocol (MCP) connections. Microsoft’s focus with its publisher marketplace is on the latter, Kolar told AdExchanger.
To that point, the publisher marketplace started with just a small roster of premium publishers.
People Inc. was first to join Microsoft’s initiative, having already struck deals with OpenAI and Meta, said Jonathan Roberts, chief innovation officer at People Inc., in a fireside chat with the Chief Marketer Network’s senior director of content and community, Lynne d Johnson, that also took place at Programmatic AI.
Microsoft’s publisher roster has since grown to eight, Roberts said, and Microsoft’s (quite ambitious) goal is to sign up the entire open web.
The marketplace licensing model has strong selling points to publishers.
But Roberts noted that the marketplace is mutually beneficial for publishers and Microsoft. Any time a publisher’s data is used to inform an AI tool’s reasoning, the site gets paid, he said. And all of the computing power behind those real-time AI inferences will run on Microsoft’s Azure cloud tech, so Microsoft collects on cloud compute costs.
“That makes it not a cost for Microsoft,” Roberts said. “This is a business.”
To block or not?
However, Roberts seemed to break from Kolar’s advice to publishers not to block AI crawlers.
Instead, Roberts advised pubs to begin by blocking everything as a way to determine which bots are necessary, such as search indexers and ad-tech-related crawlers. He said People Inc. currently gives access to 38 crawlers and blocks between 30,000 to 35,000 different crawlers a day, adding up to millions of daily unauthorized attempts to scrape its sites.
Plus, blocking all crawlers gives publishers a stronger starting point from which to negotiate said licensing deals, Roberts said. Although he acknowledged that the vast majority of sites on the web have no such clout, even if they block everything.
But Roberts said, in response to a question from the Prog AI audience about their apparent disagreement, that he thinks he and Kolar are actually in agreement on bot blocking.
Kolar’s advice is more aimed at retail and merchant brand sites, Roberts said, since those shopping sites want AI chatbots to recommend their products. Although, he added, large retailers like Walmart might still want to block bot-scraping by default to prevent competitors from building databases of their products.
The calculus for traditional pubs like People Inc. is different, Roberts said.
“We’re blocking bots so that we can permission them and so we can have control over who has accesses and how,” he said. “That is not a burn-it-all-down approach. That is, ‘Let’s do it the right way.’”
Kolar agrees that publishers are tired of reacting and are eager to take more proactive measures.
This isn’t the web’s first rodeo, after all.
Kolar shared how, in some 430 meetings that the Microsoft AI product team had with publishers last year – including 50 he personally attended – the most consistent through line from publishers was a repeated sense of powerlessness when it comes to changes in online behavior by people or platforms.
“The single thing that I heard most from publishers was, ‘Social happened to us, mobile happened to us, search happened to us. We don’t want AI to happen to us,’” he said.
So, heading into the agentic AI era, Kolar implored publishers to “do something” instead of being merely responsive to painful change that is forced upon them.
“You have a lot more power than you think you have,” he said.

Tagged in:

AI licensing

//
AI models

//
bot traffic

//
Chief Marketer Network

//
Cloud Computing

//
generative AI search

//
Jonathan Roberts

//
Lynne d Johnson

//
Meta

//
microsoft

//
Microsoft Azure

//
Model Context Protocol

//
Nikhil Kolar

//
OpenAI

//
People Inc

//
Programmatic AI 2026

Next In Publishers

Bloomberg Wants Contextual Targeting To Work In Live Video

Related Stories

Publishers
People Inc. Looks Inward For Growth As Its Search Traffic Downsizes

Publishers
People Inc. Cuts 6% Of Its Workforce

Publishers
The AI Search Reckoning Is Dismantling Open Web Traffic – And Publishers May Never Recover

Publishers
The IAB Tech Lab Isn’t Pulling Any Punches In The Fight Against AI Scraping

Must Read

Marketers
Pirated Sports Streams Are Warping TV’s Most Important Ratings

Although tides of ad revenue flow based on the ratings of certain tentpole TV events, a new crop of scammers now operate illicit sports livestreaming rings, and there’s almost nothing broadcasters can do about it.

CTV roundup
AI Is Redefining Premium Content – Which May Not Be A Good Thing

At AdExchanger’s Programmatic AI conference, media experts discussed how the rise of AI-generated content is changing the industry’s understanding of “premium” content.

PODCAST: The Big Story
Prog AI Live: AI’s Slippery Slop

Recorded live in Las Vegas at Prog AI, the AdExchanger team tackles a tricky question: As AI floods the feed with chaotic, addictive content and people engage with it, what does “premium” even mean anymore?

AdExplainer
The Programmatic Auction Is Changing In Real Time – Here’s How

Two decades after the first RTB auction, programmatic is more complex than ever – and that’s before you even consider generative AI.

Marketers
Publicis Acquires LiveRamp In A Major Shakeup For Indie Data Collaboration

Hundreds of exasperated and unexpected ad industry phone calls were made on Sunday, as agencies and ad tech vendors discussed the fallout of Publicis Groupe’s $2.2 billion acquisition of LiveRamp over the weekend.

AI
These AI Agents Want To Handle All The Annoying Parts Of Media Buying

Meet Kovva, a new AI ad tech startup tackling the unglamorous gruntwork that programmatic has never fully automated.

Popular

AdExplainer
The Programmatic Auction Is Changing In Real Time – Here’s How

Two decades after the first RTB auction, programmatic is more complex than ever – and that’s before you even consider generative AI.

AI
AI Needs Humans In The Lead, Not Just In The Loop, Says JPMorgan Chase Programmatic Lead

Automation has its place, but good marketing still needs “heart and science,” according to JPMorgan Chase’s Melissa Bonnick. Humans aren’t going anywhere.

AI
These AI Agents Want To Handle All The Annoying Parts Of Media Buying

Meet Kovva, a new AI ad tech startup tackling the unglamorous gruntwork that programmatic has never fully automated.

CTV roundup
AI Is Redefining Premium Content – Which May Not Be A Good Thing

At AdExchanger’s Programmatic AI conference, media experts discussed how the rise of AI-generated content is changing the industry’s understanding of “premium” content.

Marketers
Publicis Acquires LiveRamp In A Major Shakeup For Indie Data Collaboration

Join the AdExchanger Community
Join Now

Your trusted source for in-depth programmatic news, views, education and events.
AdExchanger is where marketers, agencies, publishers and tech companies go for the latest information on the trends that are transforming digital media and marketing, from data, privacy, identity and AI to commerce, CTV, measurement and mobile.

NEXT EVENT
Programmatic AI
May 18-20, 2026Park MGM, Las Vegas
Learn More

ABOUT ADEXCHANGER
About Us
Advertise
Contact Us
Events
Subscribe
RSS
Cookie Settings
Privacy & Terms
Accessibility
Diversity, Equity, Inclusion & Belonging

CONNECT

Microsoft provided advice to publishers regarding the rise of artificial intelligence, suggesting that publishers and retailers should allow AI bots to scrape their websites rather than blocking them. This stance comes from Nikhil Kolar, VP of publisher product at Microsoft AI, who discussed this at AdExchanger’s Programmatic AI event in Las Vegas. Kolar argued that publishers should aim to create content that can speak directly to these bots and that site owners should update their robots.txt files to prevent restrictions on crawlers that diminish visibility to AI engines. Kolar noted that currently, four out of five websites block traffic from AI bots and crawlers, which results in the content and products being inaccessible or unintelligible to AI agents, effectively limiting discovery, recommendations, and demand for publishers.

Microsoft’s approach also involves leveraging licensing mechanisms. The Publisher Content Marketplace was introduced to facilitate licensing agreements between publishers and AI developers, initially focusing on monetizing content and data for Microsoft Copilot. Microsoft has since expanded this ecosystem, acting as a clearinghouse for licensing publisher data to various AI firms. Kolar distinguished between using publisher data for "training" and using it for "grounding" AI models. Training involves establishing the deep initial data pool drawn from published online content, whereas grounding involves pulling data from current, trusted sources through protocols like the Model Context Protocol. Microsoft’s publisher marketplace primarily focuses on the latter, grounding.

The licensing model is presented as mutually beneficial. When a publisher's data informs an AI tool’s reasoning, the site receives compensation, and Microsoft benefits from collecting the associated computing costs on the Azure cloud technology. Jonathan Roberts, chief innovation officer at People Inc., suggested a different initial strategy for publishers. Instead of outright blocking all crawlers, Roberts advised publishers to begin by blocking everything to ascertain which bots are necessary, such as search indexers and ad-tech crawlers. This approach allows publishers to gain a stronger negotiating position for licensing deals, enabling them to control access and permissions before engaging in licensing discussions.

While Kolar advocated for allowing scraping, Roberts suggested a controlled method, acknowledging that blocking all crawlers provides a necessary starting point for negotiation. This divergence in advice reflects differing priorities based on the publisher’s operational context. Kolar's advice appears more aimed at retail and merchant sites seeking AI product recommendations, whereas Roberts' advice is more tailored to traditional publishers like People Inc., who prioritize control over access and permission management. Both perspectives, however, align on the need for publishers to take proactive measures rather than merely reacting to shifts in online behavior, emphasizing that publishers possess greater power in shaping the trajectory of AI integration into the web. Kolar stressed that publishers are eager to assert control over changes occurring in the online environment.