Former Anthropic Researcher Tells NYC Council That AI Firms Have Lost Control of Their Own Models

A former researcher at one of the world's leading artificial intelligence companies has issued a stark public warning that the companies building advanced AI systems do not fully understand or control the technology they are creating. Jacob Coxon, who previously worked at Anthropic, delivered the message during testimony before the New York City Council, according to a report by Al Jazeera. His appearance adds a prominent industry insider's voice to a growing chorus of concern about the pace and safety of AI development.

Coxon's central argument is that AI laboratories are racing ahead with development even though they lack reliable mechanisms to constrain the behaviour of their own models. The whistleblower characterised the situation as one in which humans are no longer firmly in the driver's seat. That claim carries unusual weight coming from someone with direct experience inside a major AI developer, rather than from an external critic or academic observer.

Beyond the question of control, Coxon raised a second and arguably more unsettling concern: the prospect of AI systems becoming capable of improving themselves. According to his testimony, models will soon be able to conduct research and development on their own successors, a scenario sometimes described in the industry as recursive self-improvement. If that happens, Coxon warned, the work of human researchers could become redundant.

That warning touches on one of the most debated concepts in the field. At present, humans still design model architectures, curate training data, tune parameters and evaluate outputs. Self-improvement would mean AI systems taking over significant parts of that pipeline, potentially accelerating progress far beyond the speed at which human oversight can keep pace. Critics of the current trajectory argue that such acceleration would widen the gap between what systems can do and what their creators can verify or govern.

The testimony also implicitly challenges the assurances that major AI companies routinely offer to regulators and the public. Those companies typically describe extensive internal safety testing, red-teaming and usage policies as evidence that their models are managed responsibly. Coxon's account suggests that such measures may not amount to genuine control, particularly as systems grow more capable and less transparent in their internal reasoning.

The venue for the warning is significant. The New York City Council is a municipal body, yet it has increasingly examined the local impact of AI, from automated decision-making in housing and employment to the technology's effect on workers and consumers. Coxon's decision to testify there reflects a broader trend of AI oversight being debated at the city and state level while national and international rules remain incomplete or contested.

New York has already emerged as an active testing ground for AI regulation, with proposals targeting algorithmic discrimination, automated hiring tools and the use of AI in city services. Testimony from an industry insider can shape how lawmakers frame those rules, particularly when it undercuts the argument that companies can be trusted to police themselves without binding external requirements.

Coxon's allegations echo themes raised by other former employees of large AI firms, some of whom have left to found or join safety-focused organisations. Departing insiders often cite a tension between commercial pressure to ship products quickly and the slower, more cautious work of ensuring those products are safe. Whistleblower accounts have become an important, if contested, source of information for policymakers who otherwise rely heavily on company disclosures.

The claim that humans do not fully control advanced AI also intersects with a long-running philosophical and technical debate. Some researchers argue that today's systems remain narrow tools with no genuine goals of their own, and that talk of losing control is overstated. Others contend that as models are given more autonomy, longer-term memory and the ability to act through software tools, the practical risks multiply even without any sci-fi-style consciousness.

Coxon's warning about self-improving systems belongs to the second camp. Even if current models are not autonomous in any meaningful sense, the argument goes, the trajectory points toward systems that can set sub-goals, evaluate their own outputs and iterate with minimal human input. At that point, the traditional levers of oversight, such as reviewing training data or auditing final models, may become far less effective.

For regulators, the testimony raises difficult questions about jurisdiction and timing. AI development is global and largely concentrated in a handful of companies, while the harms and benefits are felt locally. City councils can address deployment in their own services and set procurement standards, but they cannot directly dictate how frontier models are trained. That mismatch is a recurring frustration in AI policymaking at every level of government.

The broader public debate is also shifting. Concerns about AI have moved from science fiction scenarios toward concrete disputes over copyright, bias, job displacement and the reliability of automated systems in healthcare, finance and policing. Coxon's remarks sit at the intersection of these practical worries and the more speculative risk that humanity cedes meaningful oversight of increasingly capable machines.

Anthropic, Coxon's former employer, is among the companies that publicly emphasise safety research, including work on interpretability and alignment. The company has positioned itself as a cautious player relative to some competitors. A warning from a former researcher there is therefore notable: it suggests that even within firms that market themselves as safety-conscious, employees may believe the pace of deployment outruns the safeguards.

Coxon's recommendation, as conveyed in the report, is essentially one of caution and urgency. He argues that development is proceeding too quickly given the limits of current control methods. Whether lawmakers translate that message into legislation, funding for independent evaluation or stricter procurement rules remains to be seen, but the testimony gives them a citable, first-hand account to weigh against industry assurances.

What the episode makes clear is that the debate over AI governance is no longer confined to technical journals or corporate press releases. It has moved into city chambers, where elected officials must decide how much confidence to place in companies that, by their own former employees' accounts, may not fully understand the systems they are unleashing. For now, the whistleblower's core message stands as a challenge to both the industry and its regulators: control is asserted far more often than it is demonstrated.