Overview
Perceptron’s research thesis is that physical AI should move beyond separate vision-language models, vision-language-action systems, and world models toward a single embodied foundation model capable of perceiving, reasoning, and acting in real time. The talk develops this idea through two central bottlenecks. First, long video produces enormous token streams but only sparse, synthetic, or poorly targeted supervision; predicting every pixel supplies dense supervision but wastes attention on irrelevant background detail. Second, continuously operating cameras and robots create more visual context than conventional architectures can process efficiently. Perceptron’s response combines automatically learned perceptual objectives with a published mixture-of-experts method that lets the model select or skip tokens across layers according to task relevance. The speaker reports training on one petabyte of mixed text, image, video, and trajectory data, and describes interactive perception in which the model writes code, requests crops, changes contrast, and iteratively locates difficult objects. In robotics, the model combines embodied reasoning with control-token generation and can perform multistep perception-and-action tasks. The most consequential claimed scaling result is that ten times more video pretraining can reduce required teleoperation data by tenfold. However, the speaker explicitly acknowledges that temporal-spatial understanding and real-world robustness are not yet reliable enough for confident deployment.
Sections
Core Model Concepts
The principal model categories and mechanisms described in the talk.
- A vision-language model accepts combinations of images, video, and text but conventionally returns a textual output.
- A vision-language-action model extends the output space beyond text to include actions, while often retaining a conventional VLM-style architecture.
- A world model predicts video from inputs that may include images, video, actions, or mixtures of them.
- A semantic world model does not directly emit an observable output; it learns a representation expected to be useful later.
- An embodied foundation model is presented as one unified model for multimodal perception, embodied reasoning, and control.
- Embodied reasoning refers to reasoning grounded in spatial, temporal, sensory, and task context so that a system can decide what physical action or subtask should follow.
Contrasting Modeling Strategies
Explicit contrasts between model families, training targets, and robotic-control structures.
- VLMs primarily map multimodal inputs to text, whereas VLAs add actions to the output space.
- Sparse textual or synthetic supervision covers very little of a long video, while pixel prediction covers everything but allocates learning attention indiscriminately.
- A monolithic VLA can attempt an entire long-horizon task directly, whereas a coordinator can decompose the task for lower-level tactile policies.
- Traditional isolated perception and control are reportedly brittle to background variation, while joint perception-control modeling provides greater resilience.
Architecture, Data, and Scaling Details
Specific quantities, architectural mechanisms, training practices, and reported system properties.
- One hour of video may produce approximately one million visual tokens, while a conventional objective may calculate loss over only about 0.2% of them.
- Averaging representations across image or video segments can reportedly provide compression of up to roughly 10 times, although the speaker characterizes it as an interim solution.
- The mixture-of-experts method for heterogeneous data includes a router that predicts which tokens to process or skip, and performs this selection across multiple layers.
- The released model was reportedly trained on one petabyte of text, image, video, and trajectory data collected from web crawls, proprietary training recipes, and synthetic pipelines.
- The model can interactively manipulate visual inputs by writing code, requesting crops or zooms, adjusting contrast, and proposing successive candidate regions.
- Robotic training combines control, trajectory, perception, and embodied-reasoning objectives within a unified model that emits control tokens.
- Robustness training includes simulated camera shutdowns and sunlight entering from selected directions.
- The reported scaling law states that 10 times more video pretraining data can permit a 10-fold reduction in teleoperation data; teleoperation is estimated by the speaker at up to $100 per hour.
Strategic Implications
Higher-level conclusions derived from the architecture and experimental claims.
- The scarce resource in embodied AI may be relevant computation rather than raw sensory data: continuous cameras generate abundant inputs, but models need mechanisms that decide which portions deserve processing and supervision.
- Interactive perception blurs the boundary between seeing and acting because operations such as zooming, cropping, and changing contrast become actions taken to improve subsequent perception.
- General video becomes economically strategic when unified training allows it to substitute for expensive robot-specific demonstrations, potentially shifting robotics progress toward large-scale pretraining.
- The demonstrations imply that data-distribution design can be as important as raw compute for spatial reasoning, because internet-scale data does not naturally label relationships such as left, right, above, and below.
- The talk combines ambitious benchmark and cost claims with explicit admissions of unresolved robustness and deployment challenges, so independent evaluation is essential before treating the system as production-ready.