5 minBusiness
Publishers Weigh Pay-Per-Use AI Licensing as Deal Market Stalls
Licensing deals between publishers and AI companies are slowing even as automated scraping surges, pushing the industry toward metered pricing models that would pay for content only when it contributes to an AI answer.
Licensing agreements between news publishers and artificial intelligence companies are slowing even as demand for their content keeps climbing, a contradiction that is pushing the industry toward a new approach to payment: metered pricing.
Publishing executives have begun discussing the shift openly. At a recent Digiday town hall in London, one executive said discussions with AI companies had slowed. Katya Gorchinskaya, a Ukrainian journalist and media consultant who built J·Index, a database of thousands of AI implementations in newsrooms, told the Reuters Institute for the Study of Journalism that the market for deals with model makers is effectively closed.
At the same time, automated demand for content is rising sharply. Jonathan Roberts, chief innovation officer at People Inc., said on The Media Copilot podcast that the number of different bots trying to scrape content from the company’s media properties, which include People, InStyle and Investopedia, now runs into the tens of thousands every day. The company blocks all but 38 of them.
Fiona Spooner, managing director of consumer revenue at the Financial Times, described a single trial account that tried to access about a million articles in eight hours from 700,000 IP addresses. Many media properties treat such traffic as cost-prohibitive and block it outright, a defensible position that nonetheless offers little leverage if the goal is to monetize content inside AI systems.
Metered pricing has been developing for some time. TollBit was among the first to offer a pay-per-crawl model, directing bots to a gate where they could pay a price set by the site or leave; the company says its technology is installed on more than 7,000 publisher sites. Cloudflare followed last year by blocking AI bots by default for new sites on its network and giving publishers tools to charge by the crawl.
The emphasis has since shifted toward pay-per-use, under which a content provider is paid only when its material contributes to an answer rather than at the moment it is scraped. Cloudflare made that switch in July with Ceramic.ai and as its first partners, and Google and Microsoft are testing similar models. Cloudflare’s program remains in beta, but Lara Cohen, the company’s vice president of strategic partnerships for media, creators and AI, said it is working toward a system that alerts publishers when a buyer wants their content and lets them accept, decline or negotiate.
The logic rests on data. Research from AirOps covering more than 548,000 pages found that only about 15% of what ChatGPT retrieves ends up in its final answers. That figure should not be stretched too far: uncited material still shapes answers. An Ahrefs study of 1.4 million prompts found ChatGPT leans heavily on Reddit to understand topics while rarely crediting it, and material left out of one answer may be essential for another query.
That is why who pays matters as much as how much. An AI app answering a user’s question wants to pay only for what it uses, while whoever builds the index that app draws on benefits from having the widest possible corpus of content.
A third party complicates the picture: the middleman. A growing industry of AI data brokers, including Exa, Parallel and Tavily, crawls the internet and sells access to the resulting corpus, often to AI developers. Parallel, valued at $2 billion after a $100 million round in April, says more than 100,000 developers use its tools at companies such as Harvey, Notion and Opendoor. Roberts is skeptical of the model. «These businesses take content from the internet for free and sell it for money,» he said. «They do not create things, they resell things.»
Pricing varies, but brokers typically bill by the request. Exa charges $1 for every 1,000 pages of full text, and Parallel’s Extract costs $1 per 1,000 URLs, or roughly a tenth of a cent per article in publisher terms. Some services pay publishers nothing, though that is beginning to change. Since May, Parallel has partnered with several publishers, including The Atlantic, Fortune and PR Newswire, on a program called Index that pays according to the measured value of a publication’s contribution to work its agents perform for customers, rewarding unique, hard-to-replace content.
Parallel’s approach is a step toward the more sophisticated payment system the industry needs, but it is one layer of an economy with several. The open question is what the meters should measure, and whether a system built on pay-per-use can scale without leaving publishers who supply the underlying material unpaid.
4
