Isaac Sim Synthetic Data Generation: How the Replicator Workflow Works, Step by Step

Last updat­ed: Octo­ber 2026

Key takeaways

• Isaac Sim syn­thet­ic data gen­er­a­tion runs through Repli­ca­tor: you build a USD scene, attach ran­dom­iz­ers, choose anno­ta­tors and let a writer save labeled frames.
• The labels come free with every frame (box­es, seg­men­ta­tion, depth and more), but only for objects that car­ry seman­tic labels in the scene.
• Ran­dom­ize what varies in your deploy­ment site. Ran­dom­iz­ing every­thing adds noise, not bet­ter trans­fer.
• Pick the out­put writer before you ren­der, so the dataset loads straight into your train­ing code.
• Hold back a real, labeled test set from day one. It is the only hon­est mea­sure of whether the syn­thet­ic data helps.

Teams adopt­ing NVIDIA Isaac Sim usu­al­ly under­stand why sim­u­la­tion helps. The hard­er part is turn­ing a scene into a dataset that trains. This guide is for per­cep­tion and robot­ics engi­neers plan­ning Isaac Sim syn­thet­ic data gen­er­a­tion for detec­tion, seg­men­ta­tion or pose mod­els. It walks through the Repli­ca­tor work­flow, the labels you can export, how to write a ran­dom­iza­tion plan, how to ver­sion datasets and how to debug a dataset that does not trans­fer. For the fun­da­men­tals of sim­u­la­tion, domain ran­dom­iza­tion and the sim-to-real gap, start with our guide to robot­ics sim­u­la­tion.

Plan­ning a first dataset? Scope a pilot billed only on approved files.

What is Isaac Sim synthetic data generation?

Isaac Sim syn­thet­ic data gen­er­a­tion is the process of ren­der­ing labeled train­ing data from sim­u­lat­ed 3D scenes in NVIDIA Isaac Sim. Its Repli­ca­tor tools ran­dom­ize scene prop­er­ties such as light­ing, mate­ri­als, object pos­es and cam­era place­ment, then write images togeth­er with ground-truth labels like bound­ing box­es, seg­men­ta­tion masks and depth, with­out man­u­al anno­ta­tion.

Isaac Sim is built on NVIDIA Omni­verse and stores scenes in USD (Uni­ver­sal Scene Descrip­tion), an open 3D for­mat. NVIDI­A’s doc­u­men­ta­tion describes Repli­ca­tor as the frame­work for script­ing scene vari­a­tion and cap­tur­ing anno­tat­ed data; most pro­duc­tion teams dri­ve it from Python rather than the GUI, because script­ed pipelines are repeat­able.

How does the Replicator workflow work?

The Repli­ca­tor work­flow has five stages: assem­ble a labeled USD scene, define ran­dom­iz­ers, attach anno­ta­tors to one or more cam­eras, choose a writer and out­put for­mat, then run gen­er­a­tion and val­i­date the out­put against real data. Each stage has its own fail­ure modes, so check each one on a small batch before ren­der­ing at scale.

1. Assemble a labeled scene

Start with the envi­ron­ment, the tar­get objects and the cam­era rig that mir­rors your real sen­sor. Sim­Ready assets (sim­u­la­tion-ready 3D mod­els with physics and mate­r­i­al prop­er­ties) save mod­el­ing time. The step teams most often miss is seman­tic label­ing: every object you want in the labels needs a class label in the scene, or the anno­ta­tors will ignore it.

2. Define randomizers

Ran­dom­iz­ers vary the scene for each cap­tured frame: light inten­si­ty and direc­tion, tex­tures and mate­ri­als, object pos­es and scat­ter, dis­trac­tor objects and cam­era posi­tion. Research on domain ran­dom­iza­tion (Chen et al., ICLR 2022) and the review by Mura­tore et al. (2022) both treat vari­a­tion as the main lever for trans­fer to real con­di­tions. In prac­tice, start with the vari­a­tion your deploy­ment site actu­al­ly has, such as shelf clut­ter or shift light­ing, then widen it if val­i­da­tion shows gaps.

3. Attach annotators

Anno­ta­tors decide which labels are saved with each frame. Choose only the ones your mod­el uses, because every extra anno­ta­tor adds ren­der time and stor­age.

Label type	Typ­i­cal use	Watch out for
RGB image	Input for every vision mod­el	Match res­o­lu­tion and field of view to the real cam­era
2D bound­ing box­es	Object detec­tion	Tight vs loose box­es for occlud­ed objects
3D bound­ing box­es / pos­es	Pose esti­ma­tion, grasp plan­ning	Coor­di­nate frames must match your train­ing code
Seman­tic and instance seg­men­ta­tion	Seg­men­ta­tion, mask-based grasp­ing	Unla­beled objects become back­ground
Depth and sur­face nor­mals	Depth-aware mod­els, RGB‑D pipelines	Real depth sen­sors are nois­i­er than per­fect sim depth

4. Choose a writer and output format

Writ­ers save the frames and labels to disk. Isaac Sim ships gen­er­al-pur­pose writ­ers and lets you write cus­tom ones. Decide the tar­get for­mat (for exam­ple, a KITTI- or COCO-style lay­out, or your own schema) before you ren­der, and test that your data loader reads a small batch. NVIDI­A’s devel­op­er blog on Isaac Sim and TAO shows this pat­tern: gen­er­ate in Isaac Sim, then train in a sep­a­rate toolk­it that expects a defined for­mat.

5. Generate, then validate on real data

Ren­der a pilot batch, train a quick mod­el and test it on a held-out set of real, hand-labeled images from your deploy­ment site. Com­pare against a mod­el trained on real data alone and one trained on real plus syn­thet­ic. If syn­thet­ic data does not move the real-world met­ric, change the ran­dom­iza­tion or the scene before ren­der­ing more.

How do you write a randomization plan?

A ran­dom­iza­tion plan lists every para­me­ter you will vary, the range for each and where that range comes from. Base ranges on evi­dence from the deploy­ment site, such as pho­tos, light read­ings or shelf lay­outs, rather than guess­es. Write the plan before script­ing, so the dataset can be reviewed, repro­duced and changed one para­me­ter at a time.

Para­me­ter	Where the range comes from	Quick check
Light inten­si­ty and col­or	Site pho­tos at dif­fer­ent shifts, or sim­ple light-meter read­ings	Do dark­est and bright­est ren­ders look like the real extremes?
Object pose and place­ment	How items actu­al­ly sit on shelves, tables or con­vey­ors	Are impos­si­ble pos­es (float­ing, inter­sect­ing) fil­tered out?
Mate­ri­als and tex­tures	Real pack­ag­ing, sur­faces and fin­ish­es on site	Are shiny and trans­par­ent items rep­re­sent­ed?
Dis­trac­tors and clut­ter	Typ­i­cal non-tar­get items in the real scene	Is clut­ter den­si­ty close to the busiest real scenes?
Cam­era posi­tion and noise	Real mount­ing tol­er­ances and sen­sor specs	Do blur and noise lev­els match real frames?

Change one group of para­me­ters at a time when you iter­ate. If you widen light­ing and clut­ter togeth­er and the real-world met­ric moves, you will not know which change helped.

How much synthetic data should you generate in Isaac Sim?

There is no fixed num­ber. Gen­er­ate in small batch­es, mea­sure the effect on a real held-out test set after each one, and stop scal­ing when the real-world met­ric stops improv­ing. The use­ful amount depends on task dif­fi­cul­ty, how well the scene match­es real­i­ty and how much real labeled data you already have.

A prac­ti­cal rhythm is to ren­der a batch, retrain, test on the real held-out set and log the result next to the dataset ver­sion. Plot the real-world met­ric against the num­ber of syn­thet­ic frames. When the curve flat­tens, more frames of the same scene will not help; a bet­ter scene, wider or nar­row­er ran­dom­iza­tion, or more real data prob­a­bly will.

Ren­der­ing more frames is cheap once a pipeline runs, so the temp­ta­tion is to gen­er­ate mil­lions. The cost that mat­ters is engi­neer­ing time spent fix­ing a scene that does not trans­fer, which is why val­i­da­tion belongs inside the loop rather than at the end.

Versioning and reproducibility

A syn­thet­ic dataset is only use­ful if you can rebuild it. Treat it like code.

• Fix ran­dom seeds. Record the seed for each batch, so a frame can be regen­er­at­ed exact­ly.
• Ver­sion scenes and assets. Keep USD scenes, asset lists and mate­r­i­al libraries under ver­sion con­trol, with a tag per dataset release.
• Store the ran­dom­iza­tion plan with the data. The plan and the gen­er­a­tion script belong next to the frames they pro­duced.
• Write a short dataset card. Note scene ver­sion, seed range, anno­ta­tors, writer for­mat, frame count and the real-data test results.

The SCENE pilot check

Before your first full ren­der, score each item from 1 to 5. Any­thing below 3 is a fix-first item.

• S: Sen­sor match. The vir­tu­al cam­era match­es the real one in res­o­lu­tion, field of view, mount­ing and frame rate.
• C: Class­es labeled. Every tar­get object car­ries a seman­tic label, and dis­trac­tors are labeled or delib­er­ate­ly left as back­ground.
• E: Envi­ron­ment vari­a­tion scoped. Ran­dom­iz­ers cov­er the light­ing, clut­ter and mate­ri­als of the real site, writ­ten down as a list.
• N: Native for­mat. The writer out­puts the exact for­mat your data loader reads, test­ed on a small batch.
• E: Eval­u­a­tion on real data. A real, labeled test set from the deploy­ment site exists and is nev­er used for train­ing.

Want this as a work­sheet? Ask us for the Isaac Sim pilot scop­ing work­sheet.

Debugging a synthetic dataset that does not transfer

When a mod­el trained with Isaac Sim data scores well on syn­thet­ic test frames but poor­ly on real ones, look at the errors before chang­ing any­thing. The pat­tern of fail­ures usu­al­ly points to the cause.

Symp­tom on real data	Like­ly cause	First fix
Miss­es objects in dim or harsh light	Light­ing range too nar­row	Widen light ran­dom­iza­tion using site read­ings
Detects back­ground items as tar­gets	Too few or unre­al­is­tic dis­trac­tors	Add labeled dis­trac­tors that look like real clut­ter
Box­es off­set or wrong size	Cam­era intrin­sics or field of view mis­match	Copy real cal­i­bra­tion into the vir­tu­al cam­era
Fails on shiny or trans­par­ent items	Mate­ri­als too sim­ple	Add real­is­tic mate­ri­als and test them alone
Depth-based mod­el unsta­ble	Per­fect sim­u­lat­ed depth	Add sen­sor noise or mix in real depth frames

Common mistakes in Isaac Sim synthetic data generation

• For­get­ting seman­tic labels. Objects with­out labels silent­ly drop out of box­es and masks.
• Sen­sor mis­match. A vir­tu­al cam­era with a dif­fer­ent field of view or res­o­lu­tion teach­es the wrong geom­e­try.
• Ran­dom­iz­ing the wrong things. Vari­a­tion that nev­er occurs on site adds noise; miss­ing vari­a­tion that does occur leaves gaps.
• Per­fect depth. Sim­u­lat­ed depth has none of the noise of real RGB‑D sen­sors; add sen­sor noise or expect a gap.
• No real test set. Val­i­da­tion on syn­thet­ic data only mea­sures how well the mod­el learned the sim­u­la­tor.
• For­mat sur­pris­es. Ren­der­ing a large batch before test­ing the loader wastes GPU hours.

For a wider view of where sim­u­la­tion sits next to ego­cen­tric video, tele­op­er­a­tion and world-mod­el data, see the phys­i­cal AI train­ing data guide.

How Graveiens AI approaches Isaac Sim synthetic data generation

Graveiens AI builds sim­u­la­tion and syn­thet­ic data pipelines on NVIDIA Isaac Sim and Isaac Lab, MuJo­Co and open-source physics engines, so scenes and out­puts stay portable to the clien­t’s own stack. The work cov­ers domain-ran­dom­ized aug­men­ta­tion of footage we have already cap­tured, dig­i­tal twins of a clien­t’s work­space and val­i­da­tion against real data. We can sup­ply the real side of that val­i­da­tion through con­sent-backed ego­cen­tric video data col­lec­tion.

This is a new­er offer­ing, and we are onboard­ing a small num­ber of pilot part­ners to prove the com­bined real-plus-syn­thet­ic work­flow before scal­ing it. Engage­ments fol­low our explic­it-con­sent data prac­tices, and clients are invoiced only for approved deliv­er­ables. Graveiens AI is ISO 9001 cer­ti­fied.

Frequently asked questions

Scope your Isaac Sim pilot

Send us your tar­get objects, cam­era spec and deploy­ment envi­ron­ment, and we will help you plan a first Isaac Sim syn­thet­ic data gen­er­a­tion batch with a real-data val­i­da­tion step. Explore sim­u­la­tion and syn­thet­ic data ser­vices or book a 20-minute scop­ing call.

Sources

1. NVIDIA. Isaac Sim doc­u­men­ta­tion (Repli­ca­tor and syn­thet­ic data gen­er­a­tion). docs.isaacsim.omniverse.nvidia.com
2. NVIDIA Devel­op­er Blog. “Devel­op­ing and Deploy­ing AI-Pow­ered Robots with NVIDIA Isaac Sim and NVIDIA TAO.” developer.nvidia.com
3. Chen, X. et al. “Under­stand­ing Domain Ran­dom­iza­tion for Sim-to-real Trans­fer.” ICLR 2022. arxiv.org/abs/2110.03239
4. Mura­tore, F. et al. “Robot Learn­ing From Ran­dom­ized Sim­u­la­tions: A Review.” Fron­tiers in Robot­ics and AI, 2022. frontiersin.org

About the author

Graveiens AI Edi­to­r­i­al Team. The team writes from day-to-day work run­ning con­sent-backed data pro­grams for AI and robot­ics teams, includ­ing sim­u­la­tion, first-per­son video cap­ture and anno­ta­tion.