EleutherAI Critique: Limitations of the Foundation Model Transparency Index

Summary:EleutherAI published a blog post critiquing the Foundation Model Transparency Index (FMTI) released by Stanford's CRFM. The critique argues that the FMTI misrepresents transparency by conflating it with thorough documentation of commercial products, overemphasizes easily addressable formal criteria, and undervalues the substantive contributions of openly released models. EleutherAI contends that the FMTI's scoring system is reductive and fails to capture the true essence of transparency, which s

---

Core Issues and Criticisms

EleutherAI's critique of the Foundation Model Transparency Index (FMTI) by Stanford's CRFM highlights the following key points:

1. Conflation of Transparency with Documentation: The FMTI equates transparency with the thoroughness of documentation for commercial products, rather than focusing on transparency as a mechanism for achieving ethical values such as accountability.

2. Overly Simplified Scoring System: The FMTI's scorecard approach is overly simplistic, minimizing the nuances of analysis and encouraging people to view it as a score to optimize. This approach overemphasizes criteria that are easy for companies to address on paper without driving real change.

3. Systemic Bias Against Open Models: Despite open models scoring higher, the FMTI is systematically biased against them. For example, BLOOM-Z is incorrectly marked as not reporting compute ownership, even though this information is available in its open-access paper.

4. Confusion Between Models and Hosted Services: The FMTI conflates models with hosted model services. Models are research artifacts that can be released via papers and open-source, while hosted services are full products, often maintained by corporations and governed by Terms of Service. The FMTI's criteria are better suited for evaluating hosted services but are applied to standalone models.

5. Transparency as a Tool, Not an End: EleutherAI emphasizes that transparency should be seen as a tool for achieving other ethical values, not as an end in itself. The FMTI's approach ignores the role of transparency in mitigating specific harms or serving particular purposes, rendering it hollow.

Technical Value and Industry Impact

1. Driving Improvement in Transparency Standards: EleutherAI's critique provides an opportunity for the AI community to re-evaluate transparency standards, emphasizing the importance of open science and independent research.

2. Promoting Recognition and Adoption of Open Models: By highlighting the contributions of open models to transparency, EleutherAI calls for greater recognition of their value, promoting open science practices.

3. Guiding Discussions on AI Ethics and Governance: The critique has sparked deeper discussions on AI ethics and governance, underscoring the critical role of transparency in building trustworthy AI systems.

Recommendations for Developers

1. Focus on Transparency as a Tool: Developers should focus on the role of transparency in achieving ethical goals, rather than pursuing superficial transparency metrics.

2. Engage in Open Science Practices: By engaging in open science practices, developers can contribute to advancing transparency and reproducibility in AI research.

3. Critically Evaluate Transparency Standards: Developers should critically assess existing transparency standards and actively participate in the development of more robust evaluation frameworks.

---

Source: EleutherAI Blog (2023-10-26)

— END —

Tags: #EleutherAI #Transparency #AI Ethics #Open Science #Model Evaluation