Evals Are the New PRDs: How Anthropic, Bridgewater, and SonderMind Actually Ship AI

A rare inside look at how Anthropic builds products before the models can support them, from the person running its labs teams. The most useful idea for builders is that evals have replaced PRDs: PMs now express user needs as evaluation frameworks researchers can act on directly. Penn also explains the co-dependency between Claude Code and Opus 4.5 — neither would have landed without the other.

• Evals are the new product spec: instead of writing requirements documents, PMs encode user needs into evaluation frameworks that model researchers can train against.
• 'Sweat the tokens as much as the pixels' — reading raw conversation transcripts is how you diagnose whether a failure is tool use, retrieval synthesis, or an alignment gap.
• Frontier products and frontier models need each other: Opus 4.5 succeeded partly because Claude Code existed as the vehicle to showcase it.
• Anthropic plans for forward compatibility by asking 'what changes when Claude 8 arrives?' so today's product decisions don't become obsolete.

Bridgewater explains the architecture of PAT, an AI analyst deployed to hundreds of internal investors that compresses hours of expert research into minutes. What makes this worth your time is the engineering specifics: a compiler-style split between a chat agent and a coding agent, correctness enforced in the architecture rather than by agent orchestration, and a 'Teach' button that turns user feedback into pull requests. It's one of the most concrete enterprise agent case studies I've seen.

• Separating the planning agent (natural language) from the coding agent (compiles plans to Python) makes programming an invisible implementation detail — and lets 20-task plans generate in parallel, roughly 4x faster than Claude Code.
• Search accuracy jumped from 50% to 90% by embedding human-like inspection logic: agents verify frequency, currency, and whether values match priors, not just series names.
• 95% reproducibility comes from enforcing correctness in the architecture (static analysis, DAG validation) rather than trusting agents to remember to validate.
• Every user interaction feeds a continuous-learning loop: autonomous agents review conversations, create benchmarks, and submit PRs to improve the system for everyone.

An unusually candid 18-minute map of the AI data market that most vendors would rather you not hear. Cai explains why most benchmark results are 'quietly fake,' introduces Verifier's Law to predict which domains (coding, then finance, then healthcare) AI conquers first, and shows how lab data-spending patterns predict product launches months ahead. If you're building in a vertical, this is a useful lens for timing.

• Type 1 data (real workflow captures like GitHub commits, Slack/Jira logs) is far more valuable than Type 2 (contrived expert-generated examples) — but most vendors sell Type 2 marketed as Type 1.
• Verifier's Law: domains mature in order of how cheaply and reliably outputs can be verified — coding won because GitHub provided verification at scale; robotics lags because the modality isn't settled.
• Most benchmarks test single isolated questions instead of sustained reasoning across dependent episodes, and cross-harness differences hide high false-positive/negative rates behind aggregate stats.
• Data market spending is a leading indicator: lab purchases of cybersecurity and biology data preceded the corresponding model launches by months.

A grounded case study in shipping an LLM product where mistakes have real human cost. SonderMind's team shows how licensed clinicians — not engineers — define correctness: their annotations of real conversation failures automatically become typed evals that gate every release. The counterintuitive lesson is that off-the-shelf guardrails were too conservative for mental health, blocking the very people seeking help.

• They disabled built-in frontier model guardrails on day one because over-triggering on legitimate mental health conversations 'feels like a door slam to the face' and increases isolation.
• Guardrails run as separate modular LLM-as-judge calls on input and output — more robust to jailbreaks, at a deliberate latency and cost trade-off.
• Clinician annotations of traced conversations (expected observations, turn index, category) auto-generate typed evals in CI/CD, so domain experts, not vibes, define edge-case correctness.
• They open-sourced 300 clinically reviewed guardrail scenarios (200 input, 100 output) as a shared baseline for mental health AI safety.

If your product touches PDFs or scanned documents, this 24 minutes will save you from an expensive vendor lock-in mistake. Barrow lays out a clean two-axis decision framework (text vs. full structure, API vs. self-hosted) with real per-page pricing across cloud providers, startups, and open VLMs. The core advice is refreshingly anti-benchmark: test on your own pages and look at the output.

• OCR is sticky infrastructure — once your app is built on a specific model's output quirks, switching vendors is painful, so choose deliberately.
• Pricing spread is huge: cloud providers charge $0.60–$1.50 per 1,000 pages for text but $10–$15 with tables/forms; self-hosted VLMs on an H100 run $0.20–$3 per 1,000 pages at ~10,000 pages/hour.
• Ignore public benchmarks — run 50–100 representative pages from your actual data through candidates and visually inspect the results.
• Preserving structure as markdown/HTML is a free downstream performance win, because LLMs are trained on structured text, not jumbled line output. Also check licenses: some models carry a $2M revenue cap.