three stories about who's building the off switch -- for AI models that are starting to build
themselves, for a model that can now find and exploit real security holes on its own, and for the humans running both, now that California's governor wants the legal authority to unplug one on demand.
VoiceStudio Passes 16,000 GitHub Stars in a Week as Developers
VoiceStudio, a free, fully local, open-source alternative to ElevenLabs, has gone from a niche repository to one of GitHub's fastest-growing AI projects this week, crossing 16,598 stars with single days adding over 2,600 new ones. It handles voice cloning, voice design, video dubbing, dictation, transcription, and audiobook creation across 646 languages entirely on a user's own machine -- no API keys, no subscription, no usage meter, and (unlike a cloud service) nothing leaves the computer it runs on. It's released under the AGPL-3.0 license, which requires anyone who distributes a modified version to open-source their changes, though audio generated with it is free to use commercially. Why it's taking off: cloud voice-cloning tools mean uploading a sample of your actual voice to someone else's server, right as a summer of AI voice-deepfake scam stories has made that trade-off feel a lot riskier. "Run it yourself, keep your voice data on your own hardware" is exactly the pitch developers and privacy-conscious creators have been waiting for, and it doesn't hurt that it's free.
1) OpenAI's GPT-6 Astra Becomes the First Model to Cross a "Critical"
On September 3, OpenAI released GPT-6 Astra and, for the first time, classified one of its own models as "Critical" under its Preparedness Framework -- the company's internal system for grading how dangerous a model's capabilities are before release. The finding: Astra can autonomously discover previously unknown security flaws and build working exploits against hardened, well-defended systems, without a human directing every step. Tested without production safeguards, it scored 100% on ExploitBench, OpenAI's internal exploit-development benchmark, found two real zero-day vulnerabilities during evaluation, and built both a working browser sandbox escape and a full privilege-escalation chain to root access on hardened systems. Rather than hold the model back, OpenAI shipped it anyway wrapped in new restrictions: its most advanced offensive cyber capabilities stay gated behind a vetted-access program called Daybreak Blue, aimed at defenders trying to find and patch holes before attackers do, and the released model now refuses 91.5% of cyber-jailbreak attempts in testing, up from 59% for its predecessor, GPT-5.6 Sol. The same week, OpenAI also launched a friendlier sibling built on the same base model -- Astra for Law, a legal-research and writing tool for law firms.
The concept, simply: a "zero-day" is a security flaw nobody has patched yet because nobody who could fix it knows it exists -- the name refers to the "zero days" defenders have had to respond. A "Preparedness Framework" is a lab's own rulebook for testing a model's most dangerous capabilities -- things like cyberattacks, bioweapons help, or self-replication -- before deciding whether it's safe to release, and "Critical" is the most severe tier a model can be rated on. Why it matters: this is the first time a major lab has publicly admitted it shipped a model that meets its own definition of a serious weapon-grade capability, betting that gating and refusal training keep the risk contained rather than not releasing the model at all. Whether that bet holds is exactly the kind of question that shows up two stories down, where California's governor is asking whether labs should even get to make that call alone.
2) Claude Now Leads More Than a Quarter of the Work That Builds the Next
On September 17, Anthropic published data showing Claude now "leads" 26% of the research and engineering tasks that go into building its own successor models -- up from under 1% in February, a jump from essentially zero to more than a quarter in six months. "Leads" means Claude can take a high-level prompt and carry a research or engineering task through to completion largely on its own, with a human reviewing rather than directing each step. More than 90% of Anthropic's R&D work today involves Claude as at least a close collaborator, handling large chunks of a task under direct human supervision. The company says it now runs roughly 30,000 AI agents concurrently on internal research and engineering work, with automated monitors intervening to block roughly 1 in every 47,000 agent actions as unsafe or out of bounds. The concept, simply: this is what people mean by "recursive self-improvement" -- an AI system doing meaningful work on the process that builds the next, better version of itself, rather than only being used by humans after it's built. It doesn't mean the model is unsupervised or choosing its own goals; humans still set the direction and review the output.
Why it matters: this is the clearest public number yet on how far AI companies have gotten toward automating their own development -- the thing every lab says it's racing toward and rarely measures out loud. A sixfold jump in six months, if it continues, changes how fast the next model after Claude's successor could arrive, and how much of that pipeline a human is actually watching closely by the time it does.
3) California's Governor Orders a Path Toward a Legally Mandated AI "Kill
On September 18, California Governor Gavin Newsom signed an executive order directing a working group of experts to deliver, within two months, recommendations for strengthening the state's AI safety laws -- including a possible requirement that AI developers retain the technical ability to deactivate a frontier model on demand, and that independent third parties, not just the labs themselves, produce and review safety plans. The order builds on SB 53, the 2025 law Newsom signed that already requires frontier AI developers to publicly disclose their safety frameworks and report serious safety incidents to the state. "We're not waiting to act," Newsom said, framing the order as urgent rather than symbolic. The concept, simply: a "kill switch" for a frontier AI model is exactly what it sounds like -- a guaranteed way to shut a system down in an emergency, even if the people running it don't want to. Today that capability exists only because a lab chooses to build it in; the working group is being asked whether California should make it a legal requirement instead.
Why it matters: this is the first time a state government has moved to make emergency shutdown capability a legal requirement rather than a voluntary lab commitment -- and it lands the same week two of the stories above show exactly why a regulator might not want to take a lab's word for it: a model that's starting to help write its own successor, and a model a lab itself admits just crossed a critical weapons-capability line.
Crusoe Raises $3.9 Billion, Valuation Jumps to $30.9 Billion Crusoe, the AI-cloud infrastructure company that started out powering data centers with otherwise-wasted natural gas from oil wells, closed a $3.9 billion Series F this week, pushing its valuation to $30.9 billion. The round lands amid a broader surge in AI infrastructure financing -- tens of billions of dollars have flowed into GPU-cloud and data-center startups over the past year -- as demand for the raw compute behind models like GPT-6 Astra and Claude keeps outpacing supply.
Daybreak Blue is pitched as access for defenders, not attackers -- but "gated" only reduces risk if the vetting behind it is real. Before trusting any AI vendor's safety framing, ask who actually gets access to the powerful version and how that's checked, not just what the public version refuses to do.
Anthropic's 26% figure is a more useful early-warning signal than another leaderboard score: it measures how much of a lab's own work no longer needs a human driving. If that number keeps climbing at the same rate, the pace of new model releases -- and the amount of human review behind each one -- is likely to change faster than most people expect.
AGPL-3.0, like the one VoiceStudio ships under, is friendlier than it sounds for personal or internal use but can require you to open-source your own changes if you distribute a modified version commercially. Check licensing terms before betting a product on a trending open-source repo, not after.