RC RANDOM CHAOS
ai

How Planning Strategies Beat Predictors in JEPA World Models

· via Hacker News

Original source

After 600 experiments, here's what I discovered about JEPA world models

Hacker News →

JEPA (Joint-Embedding Predictive Architecture) proposes an alternative to generative pixel-based world models by operating in learned embedding space. The encoder converts frames to embeddings, the predictor forecasts next-frame embeddings given actions, and the planner searches action sequences to reach goals. Yann LeCun’s team released LeWM, a JEPA model benchmarked on Push-T, where a robot pushes a block to a target position based on goal images.

After 600 experiments, the author found LeWM’s predictions nearly match perfect physics within its effective 25-step planning horizon. Baseline LeWM achieves only 22% success on 75-step long-horizon tasks. Two new planning methods combined improved success to 85%. Method 1 uses hierarchical planning to decompose 75-step tasks into 25-step segments, adding nearly 60 percentage points through task structure rather than predictor improvement. Method 2 builds a memory graph from demonstration embeddings, where embedding distance indicates task reachability with 87% accuracy. Penalizing jumps between demonstrations refined the method from 62% to 85%. These planning approaches substantially outperform improving the predictor: perfect physics added only 20 points, and four-fold increased search computation yielded less than one point of improvement.

Two additional methods extend performance to harder goal scenarios, with one modification to goal scoring producing a 60-point improvement.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.