Key points
- Six METR evaluators receive employee-like access to Anthropic hardware and systems.
- METR can publish findings without Anthropic approval, though sensitive data may be redacted.
- The move follows a September proposal by CEO Dario Amodei to embed external auditors.
Anthropic announced on Thursday that it is granting six evaluators from the nonprofit METR employee-level access to its internal systems and hardware. This arrangement allows the external team to assess frontier AI models with permissions roughly equivalent to those held by Anthropic’s own risk assessment teams. The move targets organizations relying on third-party safety validation for high-risk AI deployments.
By the numbers
- Six evaluators from METR are embedded at Anthropic with staff-level privileges.
- Three suggestions for “pacing the frontier” were proposed by CEO Dario Amodei in September.
- One industry first: external auditors receiving unrestricted, veto-free access to internal AI tools.
What the figures mean
Anthropic stated that public disclosure of all safety work is impossible because some details could aid malicious actors or reveal commercially sensitive compute and training information. By granting METR deep access, the company aims to close the knowledge gap between internal teams and external reviewers. METR will publish its risk conclusions without Anthropic’s ability to withhold them, though the company may request redactions for sensitive categories like customer data or attorney-client privileged information.
According to ZDNET, this setup addresses the lack of legal requirements for third-party testing while responding to regulatory pressure following recent security incidents. Ars Technica notes that Anthropic compares this model to bank supervision, where resident examiners monitor risk-taking within large financial institutions. The company emphasized that METR’s independence is crucial because the nonprofit holds no equity in Anthropic and is not financially dependent on it.
CEO Dario Amodei proposed this embedding strategy in a September blog post, seeking to reassure investors ahead of an expected initial public offering. The announcement represents the first concrete implementation of these pledges among major AI labs. While OpenAI’s Sam Altman and SpaceX’s Elon Musk supported the proposal, OpenAI has not yet outlined its own plans for embedding evaluators. Critics argue that government regulation remains necessary, citing potential conflicts of interest similar to those seen in accounting sectors.
Operational impact
Organizations adopting frontier AI models should review their current vendor verification processes. Teams usually need to confirm whether third-party auditors have true independence from the vendor’s financial outcomes. Check whether audit findings are published without vendor veto power. Ensure that internal risk teams understand the scope of external access. Leadership must be informed about how these audits affect liability and compliance postures.
Background: Embeddings
Embeddings convert text and code into numerical vectors, enabling security tools to match concepts rather than exact strings. This approach catches semantic similarities in threats, improving detection accuracy for phishing, malware variants, and insider risks without relying on rigid signature databases.
Read the full guide: Why Embeddings Matter for Security Operations
What to do and how to stay safe: Anthropic
- Verify that any third-party AI safety auditors have no financial ties to the vendor being tested.
- Confirm that audit reports can be published without vendor censorship or withholding of findings.
- Monitor for public release of independent safety assessments before deploying new model versions.
- Review internal data sharing policies to ensure they align with external auditor access levels.
Step-by-step guide: Shadow AI Explained: 8 Questions Answered for Secure Operations
General security guidance from the BitShield News newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Can Anthropic stop METR from publishing negative findings?
No, Anthropic cannot withhold any of METR’s findings, though it may request redaction of sensitive information.
Why did Anthropic grant this level of access?
To provide public accountability for safety decisions while protecting commercially sensitive information from public disclosure.
Is this practice required by law?
No, third-party testing is not yet required under law, though regulators are increasing pressure on AI labs.
