Overview
AI labs are making a consequential bet: train agents on millions of verifiable tasks in reproducible environments, and the resulting problem-solving ability may generalize into AGI. The argument challenges that bet by separating verifiability from trainability. Coding and mathematics improve rapidly because researchers can run vast numbers of parallel, deterministic rollouts, while computer use and consequential real-world activities—building companies, practicing politics, litigating, or trading—are difficult to reset, replay, and evaluate. Their feedback is sparse, delayed, ambiguous, and embedded in changing environments. Longer context windows can temporarily absorb on-the-job knowledge, but ever-growing context is expensive and fails to consolidate experience into durable abstractions. Conventional gradient updates have the opposite weakness: they modify weights but require far more data than a single deployment supplies. The proposed bridge is continual learning through methods such as on-policy self-distillation, which transfers only outcome-relevant knowledge from an experienced in-context model into the base model, and the more speculative idea of “dreaming,” where agents construct simulations to rehearse deployment-specific skills. In the envisioned 2027–2028 pathway, RL on verifiable rewards creates deployable general agents, while subsequent real-world experience becomes the primary engine of capability growth.
Sections
Core Research Disputes
The principal disagreements over whether current scaling methods can produce durable, general intelligence.
- Can massive RL training overcome extreme training-time sample inefficiency because its cost is amortized across billions of later sessions?
- Can sufficiently long context windows replace continual weight updates?
- Will competence learned from short, reproducible white-collar tasks transfer to long-horizon real-world projects?
- Is RL's low information uptake per sample a defect or a safeguard?
Learning Regimes Compared
Contrasts among domains and methods that differ in reproducibility, supervision density, and durability.
- Coding improves faster than computer use because repositories can be cloned into deterministic containers, while live websites resist identical parallel rollouts.
- In-context learning extracts lessons efficiently from a session but stores them in expensive temporary state; gradient learning persists knowledge but requires far more samples.
- OPSD transfers outcome-relevant behavior from an experienced teacher, whereas naive supervised fine-tuning attempts to reproduce all observed tokens.
- RLVR depends on externally verifiable rewards, while OPSD can learn from differences between a context-rich teacher and the base model.
Technical Mechanisms and Constraints
Specific architectural, optimization, and data requirements underlying the argument.
- Scalable RL environments need deterministic replay, identical starting states, and large numbers of parallel rollouts.
- Attention creates session-specific fast weights that support sample-efficient adaptation, but retaining expanding KV caches imposes poor memory scaling.
- Approximately 30–50% of a laboratory's compute is attributed to inference, which currently does not directly improve model weights.
- Cursor Tab reportedly trains an online objective based on accepted edits across more than 400 million requests per day.
- OPSD minimizes per-token probability discrepancies between a base student model and a context-rich teacher model, yielding denser supervision than a single trajectory reward.
- Proposed intermediate-memory approaches include sparse attention and KV-cache compaction, although the speaker suspects the loss function is the deeper bottleneck.
- Dreaming or test-time training would spend compute constructing RL environments and training against them for skills relevant to a specific production deployment.
Higher-Order Implications
Synthesis derived from the relationship between environment design, deployment data, and learning architecture.
- The effective boundary of current AI progress may be environmental reproducibility rather than task difficulty. Capabilities advance fastest where reality can be cheaply copied, reset, and scored.
- Inference is potentially an underused training resource: deployment consumes enormous compute while exposing models to the most valuable organization-specific information, yet that experience is largely discarded.
- Continual learning is fundamentally a compression problem as much as an optimization problem. The goal is not perfect episodic recall but the extraction of durable abstractions that improve future decisions.
- RLVR and continual learning may be complementary stages rather than competing paradigms: the former creates an agent capable of productive deployment, while the latter expands its competence through deployment.
- If models learn across users as well as within individual relationships, deployment scale could create a compounding capability advantage that is difficult to reproduce through pre-release training alone.
Forecasts for Continual Learning
Explicit and conditional expectations about future capabilities and training practices.
- Computer-use performance may accelerate once coding agents can construct high-fidelity clones of common applications cheaply.
- Within a couple of years, effective context windows may feel practically unbounded for extended work sessions.
- By 2027 or 2028, agents may work alongside users for a full week, receive simple performance feedback, and distill successful session learning into the base model.
- If world-model simulation becomes sufficiently capable, dreaming or test-time training could become a fourth major scaling axis alongside pretraining, RL, and inference-time compute.
- AI capability improvement may eventually be driven primarily by accumulated economic deployment experience rather than training completed before public release.