AI Giants Back New Standard for Opting Out of AI Data Scraping via robots.txt

Summary:OpenAI, Google, Microsoft, and Anthropic have announced their adoption of a new standard that allows websites to opt out of AI data scraping through the robots.txt file. This initiative, proposed by Kevin Kelley from Anthropic, introduces a shared 'policy token' signal, enabling website operators to uniformly apply their AI training preferences across multiple AI companies without configuring them individually. The move aims to enhance control over AI model training data collection and provide a

---

Key Breakthrough

OpenAI, Google, Microsoft, and Anthropic have announced the adoption of a new standard for AI training data collection preferences. This standard introduces a 'policy token' that can be embedded in a website's robots.txt file, allowing website operators to express their data usage preferences to AI companies. The goal is to enhance transparency and control over AI model training data collection.

Technical Mechanism Analysis

The standard, proposed by Kevin Kelley from Anthropic and submitted to the Internet Engineering Task Force (IETF), operates as follows:

1. Policy Token: A special token can be added to the robots.txt file to specify AI data collection preferences, such as allowing, denying, or restricting data usage.
2. Cross-Company Sharing: The token is recognized and adopted by multiple AI companies, enabling website operators to apply their preferences uniformly across different AI providers without configuring them individually.
3. Extensibility: The standard supports various data usage preference settings, including data anonymization and usage scope restrictions, providing greater flexibility for website operators.

Engineering Trade-offs and Performance

• Advantages:

  • Strengthens website control over AI data collection, enhancing transparency in AI ethics and data privacy.
  • Simplifies the configuration process for website operators, avoiding the need to set preferences for each AI company separately.
  • Promotes trust and cooperation between AI companies and website operators, fostering the healthy development of the AI industry.

• Challenges:

  • Implementation details need further refinement, such as handling differing interpretations of the policy token by different AI companies.
  • There is a risk of website operators misconfiguring or abusing the policy token, necessitating the establishment of oversight mechanisms.
  • The standard imposes new technical requirements on AI companies, potentially increasing development and maintenance costs.

Developer Implementation and Deployment Recommendations

For website operators:

1. Understand the Standard: Read the standard documentation thoroughly to understand the configuration method and optional parameters of the policy token.
2. Evaluate Data Usage Preferences: Clearly define specific data usage preferences based on your needs.
3. Configure robots.txt: Add the policy token to the robots.txt file and test to ensure its effectiveness.
4. Continuous Monitoring: Regularly check AI companies' data collection behavior to ensure it complies with the website's policy settings.

For AI companies:

1. Support the New Standard: Integrate the new standard into existing data collection processes as soon as possible.
2. Provide Tools and Documentation: Offer tools and detailed documentation to help website operators configure the policy token.
3. Establish Feedback Mechanisms: Maintain communication with website operators to promptly address issues encountered during implementation.

Conclusion

The introduction of this new standard marks a significant step forward in AI data ethics and privacy protection. By strengthening website control over AI data collection, the standard is expected to promote the transparent development of the AI industry and foster trust and cooperation between AI companies and website operators.

---

Source: The Verge AI (2026-10-30)

— END —

Tags: #AI Ethics Follow #Data Privacy Follow #AI Policy Follow #OpenAI Follow #Google Follow