World Model Training Data: What Egocentric and Action Data World Models Need

Last updat­ed: Novem­ber 2026

World mod­el train­ing data is the video, action and sen­sor record a mod­el stud­ies to learn how the phys­i­cal world changes. It is not a labeled image set. A world mod­el has to see cause and effect: a hand push­es, a draw­er slides, a cup tips over.

This guide is for robot­ics, phys­i­cal-AI and embod­ied-AI teams in the US and EU who are scop­ing a world mod­el dataset. You will get the data types that mat­ter and the specs that make clips usable. You also get a cov­er­age plan, con­sent basics and the FRAMES scop­ing check­list.

Scop­ing a world mod­el dataset now? Book a 20-minute scop­ing call to talk through devices, actions and cov­er­age with our team.

What is a world model?

A world mod­el is a neur­al net­work that learns how an envi­ron­ment changes over time, so it can pre­dict what comes next. Video pre­dic­tion mod­els fore­cast future frames from past frames. Action-con­di­tioned world mod­els also take an action as input, so they can pre­dict the result of a move before a robot makes it.

NVIDI­A’s glos­sary describes world mod­els as AI sys­tems that under­stand real-world dynam­ics, includ­ing physics and spa­tio-tem­po­ral rela­tion­ships. It also notes that their train­ing data can include large-scale video, images, text, 3D data, sen­sor obser­va­tions and some­times actions or tra­jec­to­ries [1].

Three pub­lic research efforts show how this works in prac­tice:

• NVIDIA Cos­mos. The Cos­mos plat­form paper describes world foun­da­tion mod­els that are pre­trained on video and then post-trained for cam­era con­trol, robot­ic manip­u­la­tion and dri­ving. The team gath­ered about 20 mil­lion hours of raw video and extract­ed about 100 mil­lion clips of 2 to 60 sec­onds [2].
• Google Deep­Mind Genie 3. Genie 3 gen­er­ates inter­ac­tive envi­ron­ments from a text prompt. Users can nav­i­gate them in real time at 24 frames per sec­ond and 720p, with con­sis­ten­cy for a few min­utes. Each frame is gen­er­at­ed in sequence, based on the world descrip­tion and the user’s actions [3].
• Meta V‑JEPA 2. V‑JEPA 2 was pre­trained with­out actions on over 1 mil­lion hours of inter­net video. The team then post-trained an action-con­di­tioned ver­sion, V‑JEPA 2‑AC, on less than 62 hours of unla­beled robot video from the DROID dataset.

In short: large vol­umes of pas­sive video teach a mod­el how the world looks and moves. A much small­er, care­ful­ly aligned set of action data teach­es it what an agen­t’s actions do.

What training data does a world model need?

Most world mod­els need two lay­ers of data. A large pre­train­ing lay­er of var­ied video teach­es appear­ance and motion. A small­er action lay­er pairs video with the actions, robot states or con­trols that caused each change. Ego­cen­tric video, tele­op­er­a­tion demos, mul­ti-view rigs and sim­u­la­tion each fill a dif­fer­ent gap in that action lay­er.

Egocentric video

Ego­cen­tric video is first-per­son footage from a head- or chest-mount­ed cam­era. It shows hands, tools and objects from the actor’s view­point. That fram­ing is close to what a humanoid’s head cam­era or a wrist cam­era sees dur­ing manip­u­la­tion.

Diver­si­ty is planned, not acci­den­tal. The Ego4D research dataset, for exam­ple, holds 3,670 hours of dai­ly-life video from 931 cam­era wear­ers in 74 loca­tions across 9 coun­tries [5]. The lim­it is that actions are implic­it: you see the hand move, but not a motor com­mand. Teams add hand pose, step labels or nar­ra­tion as proxy actions.

Action and proprioception logs

Pro­pri­o­cep­tion logs record what the robot did and felt. Typ­i­cal streams are joint posi­tions and veloc­i­ties, grip­per state, end-effec­tor pose, force, torque and base odom­e­try. These streams run at their own rates. They are only use­ful once they are aligned to video frames on a shared clock.

Teleoperation demonstrations

In tele­op­er­a­tion, a human oper­a­tor dri­ves the robot through a task. You get video, the exact com­mands and the robot­’s state in one record. This is the strongest action sig­nal you can col­lect. It is also the slow­est to scale, because every hour needs a robot, an oper­a­tor and a reset. See our tele­op­er­a­tion data col­lec­tion ser­vices for how these ses­sions are run.

Multi-view and third-person video

Fixed cam­eras around a work­space, com­bined with wrist or head cam­eras, show the same event from sev­er­al angles. When the cam­eras are cal­i­brat­ed, mul­ti-view video gives the mod­el geom­e­try, depth cues and a way to rea­son about occlu­sion.

Simulation and synthetic data

Sim­u­la­tion gives per­fect labels and safe access to rare or risky events, such as dropped objects or col­li­sions. Its weak­ness is the gap between sim­u­lat­ed and real physics and appear­ance. Most teams use it to fill cov­er­age holes in world mod­el train­ing data, not to replace real cap­ture. Our sim­u­la­tion and syn­thet­ic data ser­vices cov­er this lay­er.

What specs should world model training data meet?

Usable world mod­el train­ing data has a steady frame rate and time­stamps from one shared clock. It also needs cal­i­brat­ed cam­eras with record­ed intrin­sics and action labels aligned to frames. Every clip also needs a fixed meta­da­ta record: device, scene, task, pseu­do­ny­mous oper­a­tor ID, con­sent sta­tus and QA result. With­out these, clips are hard to com­bine or trust.

Frame rate and resolution

Cap­ture at the rate your mod­el trains at, or cap­ture high­er and down­sam­ple lat­er. Nev­er mix vari­able and con­stant frame rates with­out flag­ging it. Keep res­o­lu­tion fixed per device fam­i­ly. Cos­mos’s raw pool ranged from 720p to 4K, and the team shard­ed clips by res­o­lu­tion and aspect ratio before train­ing [2].

Time sync across sensors

Put cam­eras, IMUs and the robot con­troller on one clock. Record known off­sets in the ses­sion log. Add a vis­i­ble sync event, such as an LED flash, at the start of each ses­sion so QA can check align­ment lat­er.

Camera intrinsics and extrinsics

Record focal length, prin­ci­pal point and lens dis­tor­tion for each device and each cap­ture mode. For mul­ti-view and wrist cam­eras, record extrin­sics too, mean­ing each cam­er­a’s pose rel­a­tive to the oth­ers. Recal­i­brate when­ev­er a mount moves.

Action labels

Define the action space before cap­ture. Com­mon choic­es are end-effec­tor deltas, joint tar­gets or dis­crete step labels. State the units, coor­di­nate frames and rates. For human ego­cen­tric video, use hand pose or step-lev­el nar­ra­tion as proxy actions and keep the label guide ver­sioned.

Metadata schema

• Ses­sion ID, device fam­i­ly, firmware and cap­ture mode
• Scene type, loca­tion class and light­ing con­di­tion
• Task name, objects used and out­come (suc­cess, fail­ure, recov­ery)
• Pseu­do­ny­mous par­tic­i­pant or oper­a­tor ID and con­sent record ref­er­ence
• Cal­i­bra­tion file ver­sion, sync off­sets and QA sta­tus

How much coverage and diversity does a world model dataset need?

There is no uni­ver­sal hour count. Cov­er­age should match where the mod­el will run: the scenes, objects, tasks, light­ing and embod­i­ments it must han­dle, plus fail­ures and recov­er­ies. Plan a cov­er­age matrix first, then fill its cells. Vol­ume with­out cov­er­age repeats the same dynam­ics and wastes bud­get.

The V‑JEPA 2 result is a use­ful sig­nal here. A small action-con­di­tioned set, under 62 hours of robot video, sat on top of a very large pas­sive video base [4]. That pat­tern sug­gests the action lay­er rewards care­ful cov­er­age more than raw vol­ume.

Build the matrix from these axes:

• Envi­ron­ments: homes, kitchens, ware­hous­es, retail floors, labs.
• Objects and mate­ri­als: rigid, deformable, trans­par­ent, reflec­tive, artic­u­lat­ed.
• Task verbs: pick, place, pour, open, fold, wipe, hand over.
• Con­di­tions: light­ing, clut­ter and cam­era motion.
• Embod­i­ments: human hands, par­al­lel grip­pers, dex­ter­ous hands.
• Out­comes: suc­cess, fail­ure and recov­ery, labeled as such.

Watch for redun­dan­cy as well as gaps. The Cos­mos pipeline includ­ed a seman­tic dedu­pli­ca­tion step for this rea­son [2]. For a wider view of humanoid data needs, see our guide to humanoid robot train­ing data.

How to build a world model dataset: a six-step process

In short: world mod­el train­ing data moves through scope, con­sent, cap­ture, sync, label­ing and QA, with a recap­ture loop when checks fail.

1. Scope. Name the mod­el’s job, tar­get embod­i­ment and deploy­ment scenes. Turn them into a cov­er­age matrix and an action space.
2. Con­sent. Write par­tic­i­pant infor­ma­tion, con­sent forms and a bystander pol­i­cy. In the EU, decide the legal basis and whether a data pro­tec­tion impact assess­ment is need­ed.
3. Cap­ture. Use cal­i­brat­ed devices with locked set­tings. Run a short pilot first and review it before scal­ing.
4. Sync. Align video, IMU and robot logs to one clock. Check every ses­sion’s sync event and record off­sets.
5. Label actions. Apply the action schema: robot com­mands, hand pose, step labels or nar­ra­tion. Keep the label guide ver­sioned.
6. QA and deliv­er. Check specs, sync, labels, pri­va­cy redac­tion and meta­da­ta. Deliv­er in the agreed for­mat with a man­i­fest, and send failed ses­sions back for recap­ture.

The FRAMES checklist for scoping world model training data

FRAMES is our six-part check­list for scop­ing a world mod­el dataset before any cap­ture starts. Each let­ter is a deci­sion you should write down and sign off.

• F, Fideli­ty. Which devices, frame rate, res­o­lu­tion and field of view? Are intrin­sics record­ed per device and mode?
• R, Range. Which scenes, objects, task verbs, embod­i­ments and out­comes must be cov­ered? What is the min­i­mum per cell?
• A, Actions. What is the action space? Robot com­mands, hand pose or step labels? At what rate, in which units and frames?
• M, Meta­da­ta. What is the clock source, the sync method and the fixed schema every clip must car­ry?
• E, Ethics. How are explic­it con­sent, bystanders, minors, screens and doc­u­ments han­dled? What is redact­ed, and who can with­draw?
• S, Scor­ing. What QA gates apply, what are the pass rules, and what trig­gers recap­ture?

If any let­ter has no writ­ten answer, the dataset is not ready to scale.

What drives the effort and cost of world model training data?

Effort in world mod­el train­ing data is dri­ven by action align­ment more than raw footage. Hours of pas­sive ego­cen­tric video scale fair­ly eas­i­ly. Syn­chro­nized tele­op­er­a­tion, mul­ti-cam­era cal­i­bra­tion, dense action labels, pri­va­cy redac­tion and recap­ture loops add most of the work. Scope these dri­vers before you com­pare quotes.

Dri­ver	Low­er effort	High­er effort	Decide ear­ly
Data source	Sin­gle-cam­era ego­cen­tric video	Tele­op­er­a­tion with robot state logs	Which lay­er you need now
Sen­sor set­up	One device fam­i­ly, fixed set­tings	Mul­ti-view rigs, wrist cam­eras, IMUs	Cal­i­bra­tion and sync method
Action labels	Clip-lev­el task tags	Frame-aligned actions or hand pose	Action space and label rate
Cov­er­age	Few scene types, com­mon objects	Many scenes, rare objects, fail­ures	Min­i­mum per matrix cell
Pri­va­cy	Con­trolled sets with no bystanders	Pub­lic or shared spaces	Con­sent and redac­tion rules
QA depth	Spot checks on sam­ples	Full review with recap­ture loop	Pass rules per gate

Most teams blend sources. Ego­cen­tric video adds scale and real­ism, tele­op­er­a­tion adds pre­cise actions, and sim­u­la­tion adds rare events. For more on how these lay­ers fit togeth­er, read our overview of phys­i­cal AI train­ing data for robot­ics.

Consent, privacy and common mistakes in world model training data

In short: treat every iden­ti­fi­able face, voice and home as per­son­al data. Get explic­it con­sent where the law expects it, and design cap­ture to col­lect less.

EU capture and the GDPR

Video of iden­ti­fi­able peo­ple is per­son­al data under the GDPR [6]. Arti­cle 9 pro­hibits pro­cess­ing bio­met­ric data to unique­ly iden­ti­fy a per­son. Excep­tions exist, such as explic­it con­sent for spec­i­fied pur­pos­es. Arti­cle 35 requires a data pro­tec­tion impact assess­ment before pro­cess­ing that is like­ly to result in high risk, espe­cial­ly with new tech­nolo­gies.

World mod­el datasets rarely aim to iden­ti­fy any­one. Even so, faces, voic­es, screens and home inte­ri­ors appear in ego­cen­tric footage. Plan data min­i­miza­tion, redac­tion and with­draw­al han­dling up front. This is gen­er­al infor­ma­tion, not legal advice; con­firm your approach with coun­sel.

US notes

US rules vary by state. Illi­nois’ Bio­met­ric Infor­ma­tion Pri­va­cy Act lists a scan of hand or face geom­e­try as a bio­met­ric iden­ti­fi­er. It requires a writ­ten release before a pri­vate enti­ty col­lects one [7]. Check the rules for every state where you cap­ture.

Common mistakes that weaken world model training data

• Cap­tur­ing at scale before a reviewed pilot.
• Let­ting devices drift to dif­fer­ent set­tings or firmware mid-project.
• Log­ging robot state on a dif­fer­ent clock from the cam­eras.
• Label­ing actions with­out a writ­ten, ver­sioned action schema.
• Record­ing only suc­cess­es, so the mod­el nev­er sees fail­ure or recov­ery.
• Treat­ing bystanders, screens and doc­u­ments as some­one else’s prob­lem.

How Graveiens AI approaches world model data collection

Graveiens AI runs an explic­it-con­sent col­lec­tion work­flow. It moves from par­tic­i­pant onboard­ing to record­ing and cap­ture man­age­ment, then meta­da­ta tag­ging, then QA. Our cat­a­logue includes 150,000+ ego­cen­tric videos across sev­en device fam­i­lies. Learn more about our ego­cen­tric video data col­lec­tion ser­vice.

Pilots fol­low a pay-for-approved-deliv­er­ables mod­el: you are billed only on files you approve. Graveiens AI is ISO 9001 cer­ti­fied. For tele­op­er­a­tion setups, mul­ti-view rigs, action schemas and sim­u­la­tion lay­ers, we scope this with you against your FRAMES answers. If you are com­par­ing part­ners, our guide to choos­ing an ego­cen­tric data col­lec­tion ven­dor lists the ques­tions to ask.

FAQ: world model training data

Sources

1. NVIDIA Glos­sary: What Is a World Mod­el?
2. NVIDIA et al., Cos­mos World Foun­da­tion Mod­el Plat­form for Phys­i­cal AI (arXiv:2501.03575)
3. Google Deep­Mind, Genie 3: A new fron­tier for world mod­els
4. Ass­ran et al., V‑JEPA 2: Self-Super­vised Video Mod­els Enable Under­stand­ing, Pre­dic­tion and Plan­ning (arXiv:2506.09985)
5. Grau­man et al., Ego4D: Around the World in 3,000 Hours of Ego­cen­tric Video (arXiv:2110.07058)
6. Reg­u­la­tion (EU) 2016/679, Gen­er­al Data Pro­tec­tion Reg­u­la­tion, EUR-Lex
7. Illi­nois Gen­er­al Assem­bly, Bio­met­ric Infor­ma­tion Pri­va­cy Act (740 ILCS 14)

About the author

Graveiens AI Edi­to­r­i­al Team. The Graveiens AI Edi­to­r­i­al Team writes about data col­lec­tion, anno­ta­tion and qual­i­ty review for teams build­ing speech, lan­guage and phys­i­cal-AI mod­els. The team draws on day-to-day work run­ning con­sent-based cap­ture and QA for AI train­ing datasets.