The Software Factory + The Future

1. 1
   Keynote39 min

   Software Is Changing (Again)

   Andrej Karpathy

   .

2. 2
   Current case study20 min

   The self-improvement loop in a software factory

   Warp

   .

3. 3
   Critical counterargument19 min

   Harness Engineering Is Not Enough: Why Software Factories Fail

   Dex Horthy

   Brownfield maintainability, weak review, incidents, and architectural decay under high-volume agent output.

4. 4
   Post-deployment operations25 min

   Always-on agents run production without the on-call tax

   , and Google SRE, Postmortem Culture: Learning from Failure (30 min) — the operating discipline that agent-run systems still need.

5. 5
   Tracing and evals19 min

   How to Debug AI Agents: Tracing, Observability & Evals

   Arize AI

   And Arize Phoenix, Your First Traces (30 min hands-on).

6. 6
   System design

   Symphony article

   Revisit OpenAI's

   And specification with the factory build in mind (20 min).

• Evals course≈ 2 h

  Evaluating AI Agents

  DeepLearning.AI

  Tracing, component and trajectory evaluation, LLM judges, and production monitoring.

• Link

  Langfuse

  Observability

  (open source alternative to Phoenix) and the OpenTelemetry GenAI semantic conventions.

• Link

  DSPy

  Controlled improvement loops

  Optimize prompts against your eval set instead of editing them by hand.

• Link

  SWE-bench

  Benchmarks to borrow task design from

  And Terminal-Bench.

• Link

  Introduction

  Google SRE

  .

• Evals, in depth1 h 46 min

  Why AI evals are the hottest new skill for product builders

  Hamel Husain and Shreya Shankar on Lenny's Podcast

  Error analysis, LLM-as-judge validation, and the eval workflow.

• Software-factory thesis

  The Software Factory

  Eno Reyes

  A timestamped segment on agent readiness, deterministic systems, harnesses, feedback loops, and organizations as capital allocators for agent work.

• The decade of agents2 h 26 min

  “We're summoning ghosts, not building animals”

  Andrej Karpathy with Dwarkesh Patel

  Why agents still cannot plan, remember, or compound knowledge, and what that means for the next decade.

• Broad practitioner synthesis

  Future of Programming and Modern Software Engineering

  Revisit DHH's

  , especially 3:59:24 on the future of programming. Use it for perspective, not as a technical authority.

Connect the pieces into a minimal factory:

issue → aligned spec → isolated agent run → tests/checks → independent review → human merge

Trace every run. Create a fixed evaluation set of 10–20 representative tasks and record success, regressions, cost, latency, retries, and human interventions. Add one controlled improvement loop that may propose changes to prompts, skills, or tools, but cannot deploy those changes without evaluation and human approval. “Self-improving” must not mean “silently rewrites its own controls.”

Write one blameless postmortem for the worst failure the factory produced during the ten weeks.