Software · Learning Infrastructure

Simulation, Synthetic Data & World Models

The scaling layer that lets robot teams generate experience, rehearse rare failures, evaluate policies, and transfer behavior before paying for every lesson in the physical world.

The bottleneck

Physical data is expensive. A real robot needs hardware, a workcell, an operator, resets after failures, maintenance, and supervision. Rare safety events are especially difficult to collect at useful scale.

Simulation and synthetic data promise a cheaper experience factory:

real demonstrations
+ physics simulation
+ generated scenes and trajectories
+ world-model rollouts
→ policy training and evaluation
→ selective real-world validation

The objective is not to eliminate physical testing. It is to reduce how many lessons must be learned through slow, costly, and potentially damaging real-world trials.

Four jobs that should not be conflated

1. Environment simulation

A physics engine models geometry, contacts, sensors, actuators, and task environments. It is useful for reinforcement learning, motion planning, controller testing, and repeatable regression tests.

Key question: does the simulator preserve the contact dynamics and actuator constraints that matter for the target task?

2. Synthetic data generation

Rendered or generated images, depth maps, segmentations, trajectories, and action labels expand the training distribution. Domain randomization changes lighting, textures, object placement, camera parameters, and physical properties so a policy does not memorize one scene.

Key question: does synthetic variation cover real failure modes, or does it create a larger but still biased dataset?

3. World-model rollouts

A learned model predicts how the environment may evolve after an action. World models can generate candidate futures for planning, create additional training experience, or screen policies before hardware deployment.

Key question: is the model action-faithful over a long enough horizon, rather than merely producing plausible video?

4. Policy evaluation

A virtual evaluator ranks or stress-tests policies before expensive physical trials. This is a different product from training data generation: its value depends on whether virtual results correlate with real robot success and failure.

Key question: does the simulator preserve policy ordering in the real world?

Where value can accrue

LayerProductCustomer valueMain failure mode
Physics and renderingSimulator, digital twin, sensor modelsRepeatable training and test environmentsContact or sensor mismatch
Scenario generationSynthetic scenes, assets, trajectoriesMore coverage without more robot hoursUnrealistic or duplicated data
World modelsAction-conditioned future predictionPlanning, rollouts, and cheap experienceVisually plausible but action-inaccurate futures
EvaluationPolicy benchmarks and stress testsFewer unsafe or low-quality hardware trialsWeak correlation with real deployment
Sim-to-real toolingCalibration, system identification, domain randomizationFaster transfer to the target robotHidden hardware and control mismatch
Data operationsDataset curation, provenance, replay, versioningReproducible learning loopsPoor lineage and contaminated evaluation sets

The evidence ladder

Claims in this layer should be graded by transfer evidence, not visual quality.

  1. Generated demo — the output looks plausible.
  2. Closed-loop simulation — a policy can interact with the environment rather than replay a fixed sequence.
  3. Held-out virtual evaluation — the system is tested on unseen scenes or tasks.
  4. Real-robot transfer — a policy trained or selected virtually works on physical hardware.
  5. Cross-embodiment transfer — the method works across more than one robot configuration.
  6. Fleet evidence — the pipeline reduces real data requirements, failures, or deployment time at operational scale.

The gap between levels 1 and 4 is where most marketing risk sits.

Metrics worth tracking

Data efficiency

Transfer quality

Physics and control fidelity

Evaluation quality

Economics

Sim-to-real is an operations problem

A good simulator is not enough. Transfer depends on measuring and maintaining the target robot:

robot calibration
→ system identification
→ sensor and actuator models
→ domain randomization
→ policy training or evaluation
→ staged hardware rollout
→ failure replay and model update

This makes calibration tooling, telemetry, data lineage, and regression testing part of the simulation value chain. If the physical robot changes through wear, repair, or component substitutions, the digital representation must change with it.

Company and project map

Full-stack platforms

Open research and infrastructure

Robot foundation-model companies

Physical Intelligence, Skild AI, Figure, Tesla, Agility Robotics, and other robot-model teams may build substantial simulation and synthetic-data stacks internally. The investable exposure may therefore sit inside vertically integrated OEMs as often as in standalone software vendors.

Research checklist

Before treating a simulation or world-model announcement as a strong signal, ask:

Radar thesis

The strategic question is not which simulator produces the best-looking world. It is which pipeline converts virtual experience into reliable physical work with the lowest marginal robot-hour requirement.

The strongest platforms should eventually demonstrate a measurable loop:

more virtual coverage
→ fewer expensive physical trials
→ faster failure discovery
→ safer policy releases
→ more fleet data
→ better simulation and world models

That loop is the software counterpart to manufacturing scale.

Primary references