Overview
The hardest part of deploying agents is not building a loop; it is translating messy operational knowledge into behavior that can be executed safely across fragmented systems. Drawing on global shipping operations, the speaker argues that routine work is often already automated, leaving agents to confront an expensive long tail of exceptions produced by drifting, incomplete, and country-specific processes. Traditional SOPs—frequently little more than ordered screenshots—describe what humans see and click, but they omit the preconditions, identifiers, decisions, backend calls, validation, recovery, and proof required by an executable agent workflow. The proposed production architecture therefore surrounds the runtime with a much larger system: an SOP corpus representing organizational process memory, a bounded execution environment, and a feedback mechanism that clusters failures and converts expert judgments into executable changes. Quality emerges through trace-based review, rights-disabled replay, targeted guardrails, and more than 100,000 incremental corrections—not through larger models or vague instructions to be careful. Critical actions remain gated by review and approval, while repeated successful sequences are consolidated into reusable composite tools. The central conclusion is that the agent itself is not the primary asset. The real advantage is a learning architecture that makes work representable, execution bounded, behavior observable, correction inexpensive, and improvement cumulative across countries and workflows.
Sections
Core Concepts
Key terms used to describe the production-agent methodology.
- Standard operating procedure (SOP): a documented procedure common in regulated industries that is intended to preserve how work should be performed.
- Tribal dungeons: the speaker's term for operational knowledge that exists inside an organization but is not represented in a form an agent can execute safely.
- Agent SOP: an executable process specification containing setup, preconditions, decisions, identifiers, backend calls, validation, recovery, and evidence of success.
- SOP corpus: the organization's process memory, including variations required by different countries and operating conditions.
- Trace: shared execution evidence that enables experts and engineers to inspect the same case and agree on what happened.
- Composite tool: a reusable capability formed by consolidating repeatable sequences from validated successful scenarios.
Strategic Implications
Higher-level conclusions implied by the production experience.
- The principal scaling constraint is organizational learning throughput: how quickly expert knowledge can be converted into executable, validated changes.
- The process corpus is likely to create more durable competitive advantage than model selection because it preserves proprietary operational decisions and continuously incorporates corrections.
- Agent evaluation should be organized around tracked exception scenarios rather than generic benchmark accuracy, because failures occur at specific workflow states and require specific controls.
- Cross-country deployment is fundamentally a process-variation problem; reusable tools become scalable only after local differences are explicitly represented instead of treated as noise.
- A production agent platform behaves less like a chatbot and more like a continuously refined operational control system with evidence, permissions, replay, and change management.
Production Risks and Controls
Failure modes highlighted by the speaker and the safeguards proposed for them.
- Treating screenshots or click instructions as a complete process can leave critical state, recovery, and validation requirements undefined.
- Using vague cautions such as 'please be careful' leaves unsafe actions available to the agent.
- Testing against production systems can create harmful side effects while attempting to improve behavior.
- Legacy dependencies can cause substantial latency variation regardless of the speed of the agent loop.
- Raw failure volume can overwhelm domain experts and make improvement reactive or arbitrary.
- Defaulting to broad or bloated system interfaces, including MCP-based integrations, may reduce response control and processing quality in this specific environment.
Architecture and Operating Data
Specific architecture components, control mechanisms, and reported production measurements.
- The architecture has three principal components: an SOP corpus, an execution runtime, and feedback capture.
- The SOP corpus is described as approximately 20 times larger than the runtime.
- The production system operates more than 200 instances.
- Observed latency ranges from a few minutes to as much as 10 minutes, mainly because execution depends on legacy systems.
- The system accumulated more than 100,000 corrections during nine months of development.
- Quality evaluation uses replay of real examples with production rights disabled.
- Failure controls are matched to failure type: wrong workflow leads to classifier evaluation, wrong write leads to a write gate, and a wrong assumption leads to review.
- The team favors distilled, tuned function-calling tools over MCP integrations when source systems return bloated responses or require tighter behavioral control.