LLMs for judgment, code for counting: scaling to 500K sensors on 300x fewer tokens

Phaidra's engineers had to make an LLM reason over 500,000+ sensor names in gigawatt-scale data centers, and every naive approach — vector RAG, pure LLM, parallel sharding — failed on recall or hallucinated phantom equipment. Their fix is a clean architectural pattern: the LLM plans and judges, deterministic code does the lookup and set operations, and cost scales with the hierarchy's depth rather than the number of instances. The production numbers are the best argument: from 116M tokens and 30% accuracy down to 390K tokens and 100% accuracy across 66 real cases.

• Design the system to scale with the depth of the hierarchy (data center → hall → rack → GPU), not the number of instances — their path linearizer produces the same-size summary for 64 GPUs or 460,000.
• The LLM outputs structured plans (what to collect, scope, filters) instead of processing raw names, holding cost constant at ~9,000 tokens per query regardless of system size.
• Their old approach degraded from 80% accuracy at 64 GPUs to 30% at 460,000; the hybrid approach hit 100% with 300x fewer tokens.
• A useful inversion of Karpathy's framing: AI-native systems should start at Software 3.0 for fast demos, then mature toward 1.0 determinism for production.

A proper technical breakdown of where LLM latency actually comes from, with experiments instead of hand-waving. The core lesson: prefill (processing your prompt) is parallel and compute-bound, while decode (generating tokens) is sequential and memory-bound — and decode is almost always your bottleneck. If you're building agentic products, the cost-multiplication section alone justifies the time.

• Generating one output token costs roughly 300x more time than processing one input token — cap max_tokens before you spend weeks optimizing your retrieval pipeline.
• In a 5-step agent loop, cost multiplied 11.9x because each step resends the full prior history; context management is the difference between a viable product and one killed by its bill.
• Small requests waste GPU: 143 tokens/sec at 25 tokens vs 2,271 tokens/sec at 1,205 tokens — continuous batching exists to fix exactly this.
• GPU warm-up is routinely skipped in production and cold starts quietly destroy SLAs.

Litt names the problem most of us feel but haven't articulated: agents write code faster than we build understanding of it, and that gap — he calls it cognitive debt — eventually blocks you from contributing ideas at all. Rather than a lament, he offers three concrete techniques you can adopt this week: AI-generated explanation documents with quizzes, custom 'micro-worlds' that visualize your specific system, and shared spaces where humans and agents build collective understanding.

• Cognitive debt works like technical debt: let your understanding of the system degrade long enough and you lose the ability to participate creatively, not just to verify correctness.
• Have agents write throwaway code whose purpose isn't shipping — custom debuggers and ephemeral simulations built purely so you understand your own system better.
• Auto-generated quizzes at the end of AI explanations act as a speed governor: you can't move on until you've actually absorbed the material.
• Structure AI explanations deliberately: context first, intuition before detail, and 'literate code diffs' presented in reading order rather than file order.

A short, dense talk on a failure mode every agent builder hits: agents produce more work than anyone can review, and 'done' quietly becomes a lie the system tells itself. Dotta reframes completion as a set of assertions — artifact produced, evidence attached, verifier separate from author, named owner, defined next step — and shows how Paperclip's liveness model keeps work flowing without drowning humans in review queues. Ends with a checklist you can apply to your own agent pipeline regardless of harness.

• 'Done' isn't a boolean; it's a ladder — producer claims it, a reviewer checks it, it's validated against standards, an authorized person approves it, and it survives real-world conditions.
• The core trade-off: too much autonomy produces slop, too much review produces unprocessable queues. Design for liveness with explicit blockers, not blanket approvals.
• Separate the verifier from the author — ideally with a different model — and demand explicit evidence of completion rather than the agent's self-report.
• Give agents their own verification tools (browsers, screenshots, custom hooks) so they test their work before a human ever sees it.

Five techniques for reducing agent hallucinations that live in code, not prompts — which is exactly the right place for them. Fuentes demos each one: semantic tool selection via vector search, GraphRAG for queries plain RAG botches, a validator/critic agent chain, and business rules enforced as hooks that intercept tool calls so they can't be talked around. The final third is AWS product material; the first 50 minutes are transferable to any stack.

• Filtering tools with vector search before sending them to the model cuts token consumption from ~3,000 to ~300 per call — and swap_tools keeps long conversations lean.
• GraphRAG beats traditional RAG on aggregations, counts, and multi-hop reasoning because Cypher queries return computed results instead of text samples.
• Business rules as code-level hooks that intercept tool calls are unbreakable; the same rules as prompt instructions are merely suggestions.
• Runtime guardrails can steer instead of block: the agent self-corrects and completes the task (e.g. splitting a 60-guest booking across rooms) rather than failing outright.