How Anthropic ships with its own agents, evals in the trenches, and the new physics of small teams

This is a rare inside look at how Anthropic actually uses its own coding agents: their internal agent 'Claude Tag' now lands 65% of product PRs, and they explain the engineering decisions behind that number rather than just stating it. The discussion of shrinking system prompts by 80% for frontier models — because over-constraining hurts model judgment — is directly applicable if you maintain agent prompts today.

• Claude Tag lands 65% of Anthropic's product PRs; the team treats it as a coworker embedded in public Slack channels, which requires a public-by-default communication culture.
• System prompts were cut by 80% for newer models — detailed examples and hard rules that helped older models actively constrain frontier models' judgment.
• Auto mode runs a Sonnet classifier over every tool call and bash command in context, enabling dynamic permissions and defending against prompt injection.
• The PRD-to-implementation cycle has compressed from 6–12 months toward a week, shifting engineering value from execution to product taste.

A practitioner's honest account of building an evals pipeline at a 29-person edtech company with only 6 engineers — including the parts that failed. Lucas covers going from spreadsheets to a Claude + LangFuse workflow, discovering that inter-annotator agreement was worse than a coin flip, and eventually calibrating 4 LLM judges into production. If you're wondering what evals look like outside of course material, this is it.

• Initial inter-annotator agreement was worse than random for many criteria — the fix was months of rubric clarification and re-labeling, not better models.
• Biggest resource-saving mistake correction: stop labeling data for non-problems; only label failure modes actually discovered through error analysis.
• Calibrated judges serve double duty: CI gates on PRs to catch regressions, plus daily monitoring on ~80% of production output to track quality drift.
• They calibrated judges with GPT-4 while generating with a cheaper model — but once calibrated, you must keep the same judge model or your TPR/TNR metrics are invalid.

Tan backs a big thesis — AI agents change the input/output ratios of company building — with data from inside YC: the Winter 2025 batch where 25% of companies had 95% AI-generated codebases was the fastest-growing and most profitable in YC history. The most useful part for builders is the concrete organizational framework: skill files as employees, resolver tables as org charts, and the discipline of turning every one-off task into a reusable skill.

• The core discipline is 'never do one-off work' — after solving anything successfully, skillify it into a reusable file so the agent can repeat it.
• Put computation in the right place: latent space (LLM) for judgment and vague intent, deterministic code for execution and storage — most bugs come from mixing these up.
• Company knowledge bases need hygiene, not just memory: provenance on every fact, contradiction checks, and active pruning by human-agent teams.
• The leverage gap isn't the model — 2x and 100x people use the same Claude and same context window; the difference is how the work is wired.

Thirteen dense minutes on why authorization systems break when the entity acting isn't the entity that authenticated — which is exactly what agents do. If you're wiring agents into user accounts via OAuth today, this talk explains why broad scopes plus non-deterministic behavior is how production databases get deleted, and what a proper model looks like: separate agent identity, least-privilege defaults, just-in-time elevation, and full traceability.

• OAuth scopes are too coarse for agents: 'send email on your behalf' vs 'send email only to these recipients at this hour' is the difference between over-permissioned and safe.
• MCP servers often surface every available tool regardless of which user authorized the agent, exposing capabilities beyond the intended context.
• Agents need their own identity bound to the principal, with least-privilege defaults and just-in-time requests for elevated scopes.
• Every action must be traceable: which agent, on behalf of whom, authorized by whom, when, for what, and for how long.

The most novel idea I heard this week: use automated wet labs as the verifier in RL with verifiable rewards, turning physical experiments into training tokens. Lila claims 10 trillion experimentally-verified reasoning tokens across bio and materials, and backs the thesis with a case study where a 2-3 person team compressed six years of CAR-T work into six months. Long, but it's a genuine primary-source look at where AI-for-science infrastructure is heading.

• The thesis: after the internet corpus is exhausted, lab experiments become an 'infinite token generator' — verifiable rewards grounded in physical reality.
• In vivo CAR-T case study: 2-3 people reached preclinical non-human primate data in 6 months versus a typical 6 years and $100M.
• Cross-domain transfer works — general models trained across all sciences often beat domain-specific models sample-for-sample.
• Iteration speed (round-over-round learning cycles) matters more than raw experiment parallelization as the scaling dimension.