Key take\u00adaways<\/strong><\/p>\n\n\n\n World mod\u00adel train\u00ading data is the video, action and sen\u00adsor record a mod\u00adel stud\u00adies to learn how the phys\u00adi\u00adcal world changes. It is not a labeled image set. A world mod\u00adel has to see cause and effect: a hand push\u00ades, a draw\u00ader slides, a cup tips over.<\/p>\n\n\n\n This guide is for robot\u00adics, phys\u00adi\u00adcal-AI and embod\u00adied-AI teams in the US and EU who are scop\u00ading a world mod\u00adel dataset. You will get the data types that mat\u00adter and the specs that make clips usable. You also get a cov\u00ader\u00adage plan, con\u00adsent basics and the FRAMES scop\u00ading check\u00adlist.<\/p>\n\n\n\n Scop\u00ading a world mod\u00adel dataset now?<\/strong> Book a 20-minute scop\u00ading call<\/a> to talk through devices, actions and cov\u00ader\u00adage with our team.<\/p>\n\n\n\n A world mod\u00adel is a neur\u00adal net\u00adwork that learns how an envi\u00adron\u00adment changes over time, so it can pre\u00addict what comes next. Video pre\u00addic\u00adtion mod\u00adels fore\u00adcast future frames from past frames. Action-con\u00addi\u00adtioned world mod\u00adels also take an action as input, so they can pre\u00addict the result of a move before a robot makes it.<\/p>\n\n\n\n NVIDI\u00adA\u2019s glos\u00adsary describes world mod\u00adels as AI sys\u00adtems that under\u00adstand real-world dynam\u00adics, includ\u00ading physics and spa\u00adtio-tem\u00adpo\u00adral rela\u00adtion\u00adships. It also notes that their train\u00ading data can include large-scale video, images, text, 3D data, sen\u00adsor obser\u00adva\u00adtions and some\u00adtimes actions or tra\u00adjec\u00adto\u00adries [1].<\/p>\n\n\n\n Three pub\u00adlic research efforts show how this works in prac\u00adtice:<\/p>\n\n\n\n In short:<\/strong> large vol\u00adumes of pas\u00adsive video teach a mod\u00adel how the world looks and moves. A much small\u00ader, care\u00adful\u00adly aligned set of action data teach\u00ades it what an agen\u00adt\u2019s actions do.<\/p>\n\n\n\n Most world mod\u00adels need two lay\u00aders of data. A large pre\u00adtrain\u00ading lay\u00ader of var\u00adied video teach\u00ades appear\u00adance and motion. A small\u00ader action lay\u00ader pairs video with the actions, robot states or con\u00adtrols that caused each change. Ego\u00adcen\u00adtric video, tele\u00adop\u00ader\u00ada\u00adtion demos, mul\u00adti-view rigs and sim\u00adu\u00adla\u00adtion each fill a dif\u00adfer\u00adent gap in that action lay\u00ader.<\/p>\n\n\n\n Ego\u00adcen\u00adtric video is first-per\u00adson footage from a head- or chest-mount\u00aded cam\u00adera. It shows hands, tools and objects from the actor\u2019s view\u00adpoint. That fram\u00ading is close to what a humanoid\u2019s head cam\u00adera or a wrist cam\u00adera sees dur\u00ading manip\u00adu\u00adla\u00adtion.<\/p>\n\n\n\n Diver\u00adsi\u00adty is planned, not acci\u00adden\u00adtal. The Ego4D research dataset, for exam\u00adple, holds 3,670 hours of dai\u00adly-life video from 931 cam\u00adera wear\u00aders in 74 loca\u00adtions across 9 coun\u00adtries [5]. The lim\u00adit is that actions are implic\u00adit: you see the hand move, but not a motor com\u00admand. Teams add hand pose, step labels or nar\u00adra\u00adtion as proxy actions.<\/p>\n\n\n\n Pro\u00adpri\u00ado\u00adcep\u00adtion logs record what the robot did and felt. Typ\u00adi\u00adcal streams are joint posi\u00adtions and veloc\u00adi\u00adties, grip\u00adper state, end-effec\u00adtor pose, force, torque and base odom\u00ade\u00adtry. These streams run at their own rates. They are only use\u00adful once they are aligned to video frames on a shared clock.<\/p>\n\n\n\n In tele\u00adop\u00ader\u00ada\u00adtion, a human oper\u00ada\u00adtor dri\u00adves the robot through a task. You get video, the exact com\u00admands and the robot\u00ad\u2019s state in one record. This is the strongest action sig\u00adnal you can col\u00adlect. It is also the slow\u00adest to scale, because every hour needs a robot, an oper\u00ada\u00adtor and a reset. See our tele\u00adop\u00ader\u00ada\u00adtion data col\u00adlec\u00adtion ser\u00advices<\/a> for how these ses\u00adsions are run.<\/p>\n\n\n\n Fixed cam\u00aderas around a work\u00adspace, com\u00adbined with wrist or head cam\u00aderas, show the same event from sev\u00ader\u00adal angles. When the cam\u00aderas are cal\u00adi\u00adbrat\u00aded, mul\u00adti-view video gives the mod\u00adel geom\u00ade\u00adtry, depth cues and a way to rea\u00adson about occlu\u00adsion.<\/p>\n\n\n\n Sim\u00adu\u00adla\u00adtion gives per\u00adfect labels and safe access to rare or risky events, such as dropped objects or col\u00adli\u00adsions. Its weak\u00adness is the gap between sim\u00adu\u00adlat\u00aded and real physics and appear\u00adance. Most teams use it to fill cov\u00ader\u00adage holes in world mod\u00adel train\u00ading data, not to replace real cap\u00adture. Our sim\u00adu\u00adla\u00adtion and syn\u00adthet\u00adic data ser\u00advices<\/a> cov\u00ader this lay\u00ader.<\/p>\n\n\n\n Usable world mod\u00adel train\u00ading data has a steady frame rate and time\u00adstamps from one shared clock. It also needs cal\u00adi\u00adbrat\u00aded cam\u00aderas with record\u00aded intrin\u00adsics and action labels aligned to frames. Every clip also needs a fixed meta\u00adda\u00adta record: device, scene, task, pseu\u00addo\u00adny\u00admous oper\u00ada\u00adtor ID, con\u00adsent sta\u00adtus and QA result. With\u00adout these, clips are hard to com\u00adbine or trust.<\/p>\n\n\n\n Cap\u00adture at the rate your mod\u00adel trains at, or cap\u00adture high\u00ader and down\u00adsam\u00adple lat\u00ader. Nev\u00ader mix vari\u00adable and con\u00adstant frame rates with\u00adout flag\u00adging it. Keep res\u00ado\u00adlu\u00adtion fixed per device fam\u00adi\u00adly. Cos\u00admos\u2019s raw pool ranged from 720p to 4K, and the team shard\u00aded clips by res\u00ado\u00adlu\u00adtion and aspect ratio before train\u00ading [2].<\/p>\n\n\n\n Put cam\u00aderas, IMUs and the robot con\u00adtroller on one clock. Record known off\u00adsets in the ses\u00adsion log. Add a vis\u00adi\u00adble sync event, such as an LED flash, at the start of each ses\u00adsion so QA can check align\u00adment lat\u00ader.<\/p>\n\n\n\n Record focal length, prin\u00adci\u00adpal point and lens dis\u00adtor\u00adtion for each device and each cap\u00adture mode. For mul\u00adti-view and wrist cam\u00aderas, record extrin\u00adsics too, mean\u00ading each cam\u00ader\u00ada\u2019s pose rel\u00ada\u00adtive to the oth\u00aders. Recal\u00adi\u00adbrate when\u00adev\u00ader a mount moves.<\/p>\n\n\n\n Define the action space before cap\u00adture. Com\u00admon choic\u00ades are end-effec\u00adtor deltas, joint tar\u00adgets or dis\u00adcrete step labels. State the units, coor\u00addi\u00adnate frames and rates. For human ego\u00adcen\u00adtric video, use hand pose or step-lev\u00adel nar\u00adra\u00adtion as proxy actions and keep the label guide ver\u00adsioned.<\/p>\n\n\n\n There is no uni\u00adver\u00adsal hour count. Cov\u00ader\u00adage should match where the mod\u00adel will run: the scenes, objects, tasks, light\u00ading and embod\u00adi\u00adments it must han\u00addle, plus fail\u00adures and recov\u00ader\u00adies. Plan a cov\u00ader\u00adage matrix first, then fill its cells. Vol\u00adume with\u00adout cov\u00ader\u00adage repeats the same dynam\u00adics and wastes bud\u00adget.<\/p>\n\n\n\n The V\u2011JEPA 2 result is a use\u00adful sig\u00adnal here. A small action-con\u00addi\u00adtioned set, under 62 hours of robot video, sat on top of a very large pas\u00adsive video base [4]. That pat\u00adtern sug\u00adgests the action lay\u00ader rewards care\u00adful cov\u00ader\u00adage more than raw vol\u00adume.<\/p>\n\n\n\n Build the matrix from these axes:<\/p>\n\n\n\n Watch for redun\u00addan\u00adcy as well as gaps. The Cos\u00admos pipeline includ\u00aded a seman\u00adtic dedu\u00adpli\u00adca\u00adtion step for this rea\u00adson [2]. For a wider view of humanoid data needs, see our guide to humanoid robot train\u00ading data<\/a>.<\/p>\n\n\n\n In short:<\/strong> world mod\u00adel train\u00ading data moves through scope, con\u00adsent, cap\u00adture, sync, label\u00ading and QA, with a recap\u00adture loop when checks fail.<\/p>\n\n\n\n FRAMES is our six-part check\u00adlist for scop\u00ading a world mod\u00adel dataset before any cap\u00adture starts. Each let\u00adter is a deci\u00adsion you should write down and sign off.<\/p>\n\n\n\n If any let\u00adter has no writ\u00adten answer, the dataset is not ready to scale.<\/p>\n\n\n\n Want a sec\u00adond pair of eyes on your FRAMES answers?<\/strong> Scope a pilot billed only on approved files<\/a> and test your spec on real footage first.<\/p>\n<\/div><\/div>\n\n\n\n Effort in world mod\u00adel train\u00ading data is dri\u00adven by action align\u00adment more than raw footage. Hours of pas\u00adsive ego\u00adcen\u00adtric video scale fair\u00adly eas\u00adi\u00adly. Syn\u00adchro\u00adnized tele\u00adop\u00ader\u00ada\u00adtion, mul\u00adti-cam\u00adera cal\u00adi\u00adbra\u00adtion, dense action labels, pri\u00adva\u00adcy redac\u00adtion and recap\u00adture loops add most of the work. Scope these dri\u00advers before you com\u00adpare quotes.<\/p>\n\n\n\n\n
What is a world model?<\/h2>\n\n\n\n
\n
What training data does a world model need?<\/h2>\n\n\n\n
![\"Diagram](\"https:\/\/www.graveiensai.com\/blog\/wp-content\/uploads\/2026\/11\/world-model-data-types.webp\")
Egocentric video<\/h3>\n\n\n\n
Action and proprioception logs<\/h3>\n\n\n\n
Teleoperation demonstrations<\/h3>\n\n\n\n
Multi-view and third-person video<\/h3>\n\n\n\n
Simulation and synthetic data<\/h3>\n\n\n\n
What specs should world model training data meet?<\/h2>\n\n\n\n
Frame rate and resolution<\/h3>\n\n\n\n
Time sync across sensors<\/h3>\n\n\n\n
Camera intrinsics and extrinsics<\/h3>\n\n\n\n
Action labels<\/h3>\n\n\n\n
Metadata schema<\/h3>\n\n\n\n
\n
How much coverage and diversity does a world model dataset need?<\/h2>\n\n\n\n
\n
How to build a world model dataset: a six-step process<\/h2>\n\n\n\n
![\"Six-step](\"https:\/\/www.graveiensai.com\/blog\/wp-content\/uploads\/2026\/11\/world-model-data-pipeline.webp\")
\n
The FRAMES checklist for scoping world model training data<\/h2>\n\n\n\n
![\"The](\"https:\/\/www.graveiensai.com\/blog\/wp-content\/uploads\/2026\/11\/frames-framework-world-model-dataset.webp\")
\n
What drives the effort and cost of world model training data?<\/h2>\n\n\n\n
![\"Comparison](\"https:\/\/www.graveiensai.com\/blog\/wp-content\/uploads\/2026\/11\/world-model-data-sources-compared.webp\")