Websites Are Done Blocking AI Bots—Now They Want to Charge Them

We truly live in a strange era. On one side, more than 5.6 million websites now block OpenAI’s GPTBot, and 5.8 million block Anthropic’s ClaudeBot. On the other, over 13% of AI crawlers completely ignore robots.txt anyway. Webmasters are exhausted, slapping “No AI Allowed” signs everywhere—yet nothing changes.

And here’s the uncomfortable truth: robots.txt is polite fiction. It worked in the 1990s, when the web was small and well-behaved. In 2025, when trillion-dollar AI companies vacuum up content to train massive models, politeness is meaningless.

That’s why a brand-new standard just dropped to end the charade: Really Simple Licensing (RSL) 1.0. Its message is simple—stop blocking, start charging.

Why Blocking AI Bots Is a Losing Battle

Robots.txt was never enforcement—it was a request. And requests don’t stop companies with massive scraping budgets.

Today:

• AI crawlers routinely ignore robots.txt
• Blocking hurts visibility more than scrapers
• Content creators provide value but get nothing in return

In practice, publishers have become unpaid API providers for AI training. RSL doesn’t pretend otherwise—it formalizes the relationship and introduces money into the equation.

What Is Really Simple Licensing (RSL)?

RSL 1.0 is a lightweight, machine-readable licensing standard that allows website owners to define how AI systems may use their content—and under what financial terms.

Instead of saying “go away”, RSL says “here are my rules—let’s talk business.”

Technically, RSL acts as a companion to the Robots Exclusion Protocol (RFC 9309). It introduces an XML-based vocabulary that AI systems can automatically interpret.

The Three RSL Permission Levels

RSL defines clear usage tiers:

• ai-all → Full permission to use content
• ai-input → Content can be indexed but not reused for training
• ai-index → Limited usage rights

There’s also a “Contribution” option, designed for:

• Academic research
• Non-profit organizations
• Public-interest projects

This allows creators to support research without giving commercial AI companies a free pass.

How RSL Is Implemented

One of RSL’s smartest design choices is compatibility. It integrates seamlessly with existing web infrastructure:

• robots.txt
• HTTP headers
• RSS feeds
• HTML meta tags

No new systems. No reinvention. Just a few extra lines that can say:

• “This content costs X per request”
• “Contact us to negotiate licensing”

RSL also supports complementary protocols such as:

• Open License Protocol (OLP)
• Crawler Authorization Protocol (CAP)
• Encrypted Media Standard (EMS)

Big Players Are Already On Board

RSL isn’t theoretical—it has serious backing.

Supporters include:

• Cloudflare
• Akamai
• Associated Press
• Stack Overflow

The idea originated with Matthew Prince, CEO of Cloudflare, and it’s now becoming concrete infrastructure. Over 1,500 organizations already support the standard.

Micropayments for AI Scraping: The Supertab Model

One of the most interesting implementations comes from Supertab, a micropayment service that’s been testing RSL-style billing for two quarters.

Their system uses a “tab” model, similar to a bar tab:

• Content usage accumulates
• Payment happens once a threshold ($1 or $5) is reached
• No constant paywalls or micro-transactions

Results so far:

• 3× increase in paying readers
• 10% of users who open a tab subscribe within 3–4 months

Now Supertab applies the same logic to AI bots. Scraping requests add up—and bots pay later. Unlike humans, bots don’t forget their bill. They operate on defined budgets.

Will RSL Save Journalism? Not Exactly—but It’s a Start

RSL won’t magically fix copyright law or save journalism overnight. But it does something crucial: it stops pretending.

Content creators already fuel AI systems. RSL simply acknowledges the economic reality and introduces a framework where value exchange is explicit, automated, and negotiable.

The real question is whether OpenAI, Google, Anthropic, Meta, and others will actually comply. If major infrastructure players enforce it, they may not have much choice.

Example: A Paid AI Training License

Below is a simplified example of an RSL license that allows AI training—but only under a paid, negotiated agreement:

<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/">
    <license>
      <permits type="usage">ai-train</permits>
      <payment type="subscription">
        <!-- Using content for AI training requires custom license -->
        <custom>https://example.com/contact-form.html</custom>
      </payment>
    </license>
  </content>
</rsl>

In plain English:

• AI models may train on the content
• Only with a paid subscription
• Licensing terms are negotiated directly with the publisher

RSL can be embedded in:

• robots.txt
• HTML pages
• HTTP headers
• Media files
• RSS feeds

No new infrastructure required—just structured metadata AI systems can’t “pretend” not to understand.

Automating AI Licensing at Internet Scale

One of RSL’s most powerful features is automation.

Through the RSL Open Licensing Protocol (OLP), publishers can:

• Eliminate manual negotiations
• License content programmatically
• Charge AI companies at scale

Example: RSL License Server

<rsl xmlns="https://rslstandard.org/rsl">
  <!-- Using content requires obtaining license from an OLP server -->
  <content url="/videos" server="https://example-server.org/api">
    <license>
      ...
    </license>
  </content>
</rsl>

This allows AI companies to:

• Query a license server
• Obtain permissions automatically
• Track usage and billing transparently

Publishers can host their own license server or join the nonprofit RSL Collective, which manages access, licensing, and payments on their behalf.

Reclaiming Lost Revenue Through Collective Licensing

RSL also enables collective licensing, similar to how music royalties work.

Instead of negotiating alone, publishers can:

• Pool their content
• Set collective licensing terms
• Receive royalties every time AI systems use their work

Example: Paid Collective License

<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/" server="https://api.rslcollective.org">
    <license>
      <payment type="inference">
        <!-- Using content requires obtaining RSL Collective license -->
        <standard>https://rslcollective.org/license</standard>
      </payment>
    </license>
  </content>
</rsl>

This model supports:

• Pay-per-inference compensation
• Automatic attribution
• Recurring royalties

Publishers can join the RSL Collective for free and start receiving payments when AI companies consume their content.

Secure Licensing for Non-Public Content

RSL goes beyond public web pages. It also supports secure licensing of proprietary assets, including:

• Books
• Videos
• Datasets
• Research archives

Using the Encrypted Media Standard (EMS), content can remain encrypted until a valid license is obtained.

Example: Encrypted Book License

<rsl xmlns="https://rslstandard.org/rsl">
  <!-- Decrypting EPUB content requires obtaining a license key -->
  <content url="/books/book1.epub.enc" encrypted="true"
           server="https://example-server.org/api">
    <license>
       ...
    </license>
  </content>
</rsl>

This replaces:

• Ad-hoc data sharing
• Password-based access
• Custom integrations

With a standards-based, encrypted delivery model that protects content in transit.

Creating Machine-Readable Content Catalogs

RSL integrates with RSS and Schema.org, enabling publishers to create standardized catalogs of licensable content and datasets.

Example: RSS License Catalog

<rss xmlns:rsl="https://rslstandard.org/rsl" version="2.0">
  <channel>
    <title>Datasets - UCI Machine Learning Repository</title>
    <link>https://archive.ics.uci.edu/datasets</link>
    <item> 
      <title>Iris dataset</title> 
      <link>https://archive.ics.uci.edu/dataset/53/iris</link>
      <rsl:content url="https://archive.ics.uci.edu/public/53/iris.zip">
        <rsl:license>
          <rsl:payment type="free"/>
        </rsl:license>
      </rsl:content>
    </item>
  </channel>
</rss>

This allows AI companies to:

• Discover licensable assets automatically
• Understand usage rights instantly
• License datasets without friction

Why This Changes Everything

RSL acknowledges a simple reality:
Content is already being used like an API—so it should be licensed like one.

Instead of pretending AI scraping can be stopped, RSL:

• Makes licensing explicit
• Makes compensation programmable
• Makes compliance auditable

Whether AI giants will fully embrace it remains to be seen—but with Cloudflare, Akamai, major publishers, and payment infrastructure backing it, RSL is the first serious attempt to turn AI content usage into a sustainable economy instead of a legal gray zone.

Conclusion

The web has grown up. Blocking AI bots is theater, and robots.txt is no longer enough. Really Simple Licensing represents a shift from denial to negotiation, from polite requests to enforceable economics.

Creators didn’t choose to become data suppliers—but if they are, they should at least get paid.

The game is changing. Let’s see who plays along.

👉 Learn more at rslstandard.org