VSvarunsingla.com

← All entries

Day 176· · 5 min read

three stories about an industry arguing with itself over its own speed, all landing in the

Governance & Safety

same week. Anthropic booked its first profitable quarter days after its own CEO argued AI development must slow down. Separately, two of the field's own safety researchers quit their labs to build independent oversight from the outside instead.

Viral app of the day

EcoGPT: A Viral "Eco-Friendly" App Built on a Debunked AI Water-Use Myth

EcoGPT, an app marketed around AI's environmental footprint, has been downloaded more than 530,000 times -- averaging over 8,000 downloads a day -- fueled by viral short-form videos of creators holding up jars of murky brown water to warn that fresh drinking water will disappear because of AI data centers. The claim has been repeatedly fact-checked and debunked: independent estimates put the US lawn-irrigation industry's water use at roughly 60 times all direct and indirect AI data-center water use combined. Why it's taking off: the app isn't spreading because of a technical breakthrough -- it's spreading because it packages a real, if wildly exaggerated, anxiety (AI's environmental cost) into a visceral 60-second visual (dirty water) that plays perfectly in a short-form feed. That's a distribution pattern worth recognizing on its own: guilt-driven, emotionally simple content consistently outperforms nuanced, accurate explanations, whether or not the underlying app or claim holds up to scrutiny.

By the numbers
$11.5B
Anthropic's Q2 revenue, up 14x year-over-year, alongside its first operating profit
38.8%
Real enterprise coding tasks the best model+harness combo actually resolved on Real-SWE
530K+
Downloads of EcoGPT, the eco-guilt app built on a debunked water-use claim
60x
More water the US lawn-irrigation industry uses than all direct+indirect AI water use

1) AI Safety's Own People Are Walking Out the Door

Two of the AI industry's own safety researchers just quit to watch it from the outside. Joe Benton, who led Anthropic's Scalable Oversight team, and Josh Engels, a safety researcher at Google DeepMind, both resigned within days of each other in mid-September to join METR (Model Evaluation and Threat Research), an independent nonprofit that stress-tests frontier models for dangerous capabilities. Benton told reporters that "all of the transparency about these risks that is coming from the companies is entirely voluntary," and said he wants to build pressure from outside instead. The concept, simply: "scalable oversight" is the research problem of how humans keep checking an AI system's work once that system gets smarter or faster than the humans reviewing it. Doing that work inside a lab means your employer decides what gets published and when. An independent evaluator like METR has no product to ship and no investor call to protect -- it can publish an uncomfortable finding about any company's model without asking permission first. Why it matters: self-reporting only works as far as a company is willing to hurt its own launch schedule. Benton's specific complaint -- that competitive pressure pushes every major lab to cut corners on safety testing rather than fall behind -- is the same dynamic critics have been naming for two years. Two credible insiders leaving prestigious jobs specifically to build outside accountability is a stronger signal about how that pressure actually feels internally than any corporate safety report.

2) The Industry Argues, in Public, Over Its Own Speed Limit

Anthropic CEO Dario Amodei published an essay titled "We Must Pace the Frontier," arguing that the rate of AI capability improvement needs to slow down deliberately so that safety and alignment work has time to catch up. Microsoft chairman and CEO Satya Nadella responded publicly that Microsoft "welcomes" the "deliberate pacing needed to get alignment right," and announced that Microsoft will publish a "Code of Conduct" governing its own first-party MAI models for public consultation -- alongside ideas like independent "embedded evaluators" built into how models are developed, not just tested afterward. The concept, simply: the "pacing" debate is really a coordination problem. Any single lab that slows down on its own risks handing the market to a faster, less cautious competitor -- so public commitments like Amodei's essay and Nadella's response are attempts to make caution a shared, visible norm rather than a unilateral (and commercially risky) choice. A published Code of Conduct is one way to make a promise checkable instead of just a talking point. Why it matters: words are cheap until they're followed by a specific, auditable mechanism. Nadella name-checking "embedded evaluators" is a meaningfully more concrete idea than a general call for caution -- it implies putting outside checks inside the development process itself, not just at the end. Whether Microsoft's actual published document (due the day after this essay) contains anything enforceable, or reads as a values statement, is the real test of whether this debate produces change or just more essays.

3) Anthropic Turns a Profit as Its IPO Comes Into View

Anthropic told investors it booked more than $11.5 billion in second-quarter revenue -- a roughly 14-fold increase from the same period a year earlier -- and posted an adjusted operating profit for a second straight quarter, with inference gross margins reported above 85% on its Claude API business. The disclosures came as CFO Krishna Rao held early, high-level meetings with investors ahead of what could become one of the largest IPOs in history, with some backers reportedly discussing a valuation near $2 trillion. The concept, simply: "gross margin" measures how much money is left from each dollar of revenue after paying for the compute that serves it -- before counting the much larger cost of training the next model. An 85%+ gross margin on inference means Anthropic's core product, once built, makes real money on every additional customer; the company's earlier losses were a function of spending on training future models, not a broken business model serving today's.

Why it matters: read next to story two, this is the tension underneath the whole "pacing" debate made concrete. A company arguing publicly that the industry should slow down is, in the same week, showing investors explosive growth and a path to one of the largest IPOs ever. Those two positions aren't necessarily contradictory -- but the pressure to keep growth like this going is exactly the competitive force Benton pointed to when he said no single company can safely slow down on its own.

Market signal

Private Benchmarks Are Exposing a Big Gap Between Leaderboard Scores and Real Work Specific Labs launched Real-SWE, a benchmark that runs frontier coding agents against private, licensed production codebases from real companies -- a fintech platform processing 100,000+ bank statements, an app with 200,000+ users -- instead of the public GitHub repositories most coding benchmarks use. The best model-and-harness combination resolved just 38.8% of tasks; every other combination scored lower, down to 16.2%. Because the code is private and licensed, no model could have trained on the answers in advance. That gap matters because public coding benchmarks are increasingly suspected of measuring familiarity with popular open-source repositories as much as general coding skill. A benchmark built specifically so the answers can't have leaked into training data is a harder, more honest yardstick -- and the industry is starting to build more of them as public leaderboard scores become less trustworthy signals of real-world reliability.

Practical takeaways
Weigh insider departures alongside official safety reports, not instead of them.

When researchers with direct internal access resign specifically to build outside accountability, as Benton and Engels did this week, that's a different and often more candid kind of signal than a company's own published safety commitments -- read both, but don't take either as the full picture.

Treat a lab's pacing or Code of Conduct announcement as a starting point, not a result.

Look for the specific, checkable mechanism behind the statement -- an external audit, an "embedded evaluator," a published document with real constraints -- rather than the headline itself. A promise to be careful is not the same as a way to verify it.

Before trusting a coding agent on your own codebase, look for a private-benchmark score, not just a public leaderboard rank.

Real-SWE's results show the best agents still fail roughly 6 in 10 tasks on real, proprietary code -- far below what public benchmark scores suggest. If you're evaluating a coding agent for enterprise use, ask whether it's been tested on anything the model couldn't have already seen.

VS
Varun Singla
Singapore · About · Learning in public