How Unify cut agent costs 95%, and why RL-trained agents break in the wild

A CTO walks through the actual engineering behind a 90-95% agent cost reduction in two weeks: collapsing many sub-agents into one code-writing main agent, working around OpenAI's 15 req/s prompt-cache limit by hashing users across 16 buckets, and building a zero-cost TypeScript sandbox. It's rare to get this level of production detail on agent economics, including why they optimize tool-call efficiency over per-token price.

• Tool-call efficiency dominates cost: sub-cent LLM calls are noise next to $1+ data vendor API calls, so a 10x-cheaper model that makes 3x more tool calls is a bad trade.
• OpenAI prompt caching caps at ~15 requests/second per cache key; Unify distributes load by hashing user IDs across 16 buckets.
• When using LLM-as-judge or user simulation, use a different model family than your main agent to avoid mode collapse in evals.
• A robust upfront planning step that scouts trajectories and picks high-precision APIs first was one of the biggest efficiency wins.

Finn presents Physical Intelligence's own results, and the headline is significant: their π₀7 foundation model now matches or beats specialist fine-tuned models across tasks out of the box, the same transition language models went through around GPT-2/GPT-3. She also details the concrete engineering that got them there — an RL recipe with human interventions, multi-timescale memory, and metadata prompting that lets low-quality data help rather than hurt.

• π₀7 out-of-the-box matches or exceeds RL post-trained specialist models across multiple tasks — the strongest evidence yet of generalist robot models.
• Their espresso robot hit 90%+ success and ran autonomously for 13 hours, the reliability bar Finn argues real-world robotics requires.
• Adding low-quality training data improved performance with metadata prompting but degraded it without — how you label data matters as much as what you add.
• The models are already deployed commercially (laundry folding at Ultra, warehouse packaging at Weave) and adapted to drones, surgical robots, and tractors.

An 18-minute researcher talk on why coding agents that ace RL benchmarks fail when deployed: real environments have session expiry, irreversible actions, adversarial content, and ambiguous success criteria that the training setup never modeled. The taxonomy of failure modes and the harness-guardrail checklist are directly usable if you're deploying computer-use or coding agents today.

• Real failures are mundane and dangerous: agents guessing passwords when sessions expire, clicking sponsored ads instead of buttons, filling personal details on the wrong site.
• Six deployment gaps break RL assumptions: partial observability, irreversible actions, non-determinism, session expiration, ambiguous success, and adversarial content.
• Process reward models should penalize dangerous actions along the trajectory, not just score final outcomes; calibrated confidence tells the agent when to escalate.
• Practical loop: deploy early behind strong harness guardrails, collect real failure modes, retrain on those patterns, then gradually thin the harness.

A genuinely novel architecture for self-modifying agents: split the system into a stateless executor, a protected harness holding secrets and history, and an isolated sandbox — then let the agent edit its own executor code, with a guardian process that tests changes for one step and auto-rolls back. The proof point is concrete: asked to cut spending, Exo rearchitected its own context assembly and reduced Discord adapter costs by 96%.

• Separating stateless compute from protected state is what makes self-modification safe: the executor can be rebuilt at runtime while secrets and history stay out of reach.
• Exo mounts its own executor code in the sandbox; a guardian tests each self-edit for one step and rolls back automatically if the agent breaks itself.
• Annotating conversation logs with per-message costs gives the agent runtime economics data — that's how it found and executed the 96% cost reduction.
• Krentsel argues recursive self-improvement is finally tractable because the medium matches the output: LLMs write code, and the harness is code.

A clear technical walkthrough of model cascades: route classification requests to a cheap model first and escalate to an expensive one only when confidence is low, with a concrete threshold-finding algorithm you can implement this week. Honest about limits too — it only works where confidence is calibrated (finite label sets), and the method fails loudly rather than silently when confidence scores are uninformative.

• Sample 200-500 items, run both proxy and oracle, then simulate thresholds to find the minimum-cost point that hits your target accuracy (e.g., 95% agreement).
• This works for classification with calibrated confidence, not open-ended generation where confidence scores are unreliable.
• If confidence is uncorrelated with accuracy, no threshold will meet your target — the method self-diagnoses rather than lying to you.
• For statistical guarantees beyond the sample-based estimate, use concentration inequalities or the Bargain Python package.