Hand Pose Annotation for Robot Learning: Keypoints, Hand-Object Labels and QA

Last updat­ed: Octo­ber 2026

Key takeaways

• Hand pose anno­ta­tion marks the joints of the hand in each frame, so a mod­el can learn how peo­ple shape and move their hands dur­ing a task.
• For robot learn­ing, key­points alone are rarely enough: add con­tact, grasp type and object state to cap­ture the hand-object inter­ac­tion.
• Choose 2D key­points for image-space mod­els and 3D key­points when the pol­i­cy needs depth; 3D usu­al­ly needs cal­i­brat­ed mul­ti-view or depth cap­ture.
• Occlu­sion is the main qual­i­ty prob­lem, so write rules for hid­den joints before label­ing starts.
• Good cap­ture makes label­ing cheap­er: hands in frame, steady light­ing and syn­chro­nized cam­eras reduce rework.

Robots that manip­u­late objects need to learn from hands. Ego­cen­tric and mul­ti-view video shows those hands at work, but a pol­i­cy can only use what the labels describe. This guide is for robot­ics and com­put­er vision teams plan­ning hand pose anno­ta­tion for manip­u­la­tion, grasp­ing or imi­ta­tion learn­ing. Most search results on the top­ic are research papers and pub­lic datasets; this guide turns that work into a prac­ti­cal spec. It cov­ers 2D vs 3D key­points, hand-object inter­ac­tion labels, label tax­onomies by task, tools and for­mats, QA checks and cap­ture tips that make label­ing faster.

Want to see the out­put first? Request a sam­ple anno­tat­ed clip set.

What is hand pose annotation?

Hand pose anno­ta­tion is the label­ing of hand joint posi­tions, usu­al­ly the wrist and fin­ger joints, in images or video frames. Each key­point gets a posi­tion and often a vis­i­bil­i­ty flag. For robot learn­ing, teams usu­al­ly add inter­ac­tion labels such as con­tact, grasp type and object state, so the data shows not just where the hand is but what it is doing.

Most projects use a fixed key­point lay­out with the wrist plus joints along each fin­ger, com­mon­ly 21 points per hand. Keep the same lay­out, joint order and nam­ing across the whole dataset, because down­stream code depends on them.

2D vs 3D hand keypoints: which do you need?

Choose 2D key­points when your mod­el works in image space, such as hand detec­tion or action recog­ni­tion from a sin­gle cam­era. Choose 3D key­points when the pol­i­cy rea­sons about depth, reach or grasp geom­e­try. 3D labels usu­al­ly require cal­i­brat­ed mul­ti-view or depth cap­ture, which rais­es cap­ture and label­ing effort.

Option	Best for	Cap­ture needs	Effort
2D key­points	Detec­tion, track­ing, action recog­ni­tion	Sin­gle RGB cam­era	Low­er
2D key­points + vis­i­bil­i­ty	Ego­cen­tric video with fre­quent occlu­sion	Sin­gle RGB cam­era	Mod­er­ate
3D key­points	Grasp geom­e­try, retar­get­ing to robot hands	Cal­i­brat­ed mul­ti-view or RGB‑D	High­er
3D key­points + object pose	Hand-object recon­struc­tion, han­dover	Mul­ti-view or RGB‑D plus object mod­els	High­est

Mul­ti-view cap­ture also helps label­ing itself. Simon et al. showed that detec­tions from sev­er­al views can be tri­an­gu­lat­ed in 3D and repro­ject­ed to cre­ate bet­ter labels for occlud­ed joints, an approach they called mul­ti­view boot­strap­ping.

Which hand-object interaction labels matter for robot learning?

The hand-object labels that mat­ter most for robot learn­ing are con­tact (which hand touch­es which object, and when), grasp type, the objec­t’s iden­ti­ty and state, and action seg­ments with start and end times. Togeth­er they turn hand motion into a descrip­tion of the task that a manip­u­la­tion pol­i­cy can learn from.

• Con­tact. Per-frame or per-seg­ment flags for hand-object con­tact, ide­al­ly per hand.
• Grasp type. A short, fixed list (for exam­ple pow­er, pre­ci­sion, pinch) that your team can apply con­sis­tent­ly.
• Object iden­ti­ty and state. Which object, and whether it is open, closed, full, emp­ty, cut or assem­bled.
• Action seg­ments. Start and end times for steps such as reach, grasp, move, place and release.
• Object pose (option­al). 6D pose when the pol­i­cy needs exact geom­e­try.

Pub­lic datasets show how these lay­ers com­bine. DexY­CB bench­marks 2D key­point detec­tion, 6D object pose and 3D hand pose for hands grasp­ing objects, and adds a human-to-robot han­dover task. HOI4D pro­vides ego­cen­tric RGB‑D sequences with frame-wise 3D hand pose, object pose, seg­men­ta­tion and hand action labels. Pub­lic data is a good ref­er­ence; com­mer­cial robots usu­al­ly need cus­tom cap­ture of their own tasks and set­tings.

Label taxonomy examples by robot task

A tax­on­o­my should be small enough to label con­sis­tent­ly and spe­cif­ic enough to teach the task. The exam­ples below show how the same four lay­ers change with the job the robot will do.

Task fam­i­ly	Grasp types to use	Object states to track	Action seg­ments
Kitchen prepa­ra­tion	Pow­er, pre­ci­sion, pinch, tool grip	Whole, cut, peeled, open, closed	Reach, grasp, cut, trans­fer, release
Small-part assem­bly	Pinch, tri­pod, pre­ci­sion	Loose, insert­ed, fas­tened	Pick, align, insert, fas­ten, inspect
Ware­house pick­ing	Pow­er, wrap, two-hand lift	On shelf, in hand, in tote	Reach, grasp, lift, place, release
House­hold tidy­ing	Pow­er, pinch, flat-hand push	Fold­ed, unfold­ed, open, closed	Pick, fold, push, place, close

Start with the small­est list that cov­ers your tar­get tasks, and add a label only when the pilot shows anno­ta­tors need it. Every new class adds train­ing time for anno­ta­tors and dis­agree­ment in QA.

The GRIP label spec

Write a one-page spec before label­ing. We use four head­ings, GRIP, to keep it short and com­plete.

• G: Geom­e­try. Key­point lay­out, joint order, 2D or 3D, coor­di­nate frames and units.
• R: Rules for occlu­sion. How to label joints that are hid­den, out of frame or blurred: esti­mate, mark as not vis­i­ble, or skip.
• I: Inter­ac­tion labels. Con­tact, grasp types, object states and action seg­ments, with def­i­n­i­tions and exam­ples.
• P: Pass cri­te­ria. QA thresh­olds, sam­pling rate for review and what trig­gers rework.

Need a start­ing tem­plate? Ask us for the hand-label spec tem­plate our team uses on scop­ing calls.

Which tools and formats are used for hand pose annotation?

Hand pose anno­ta­tion is done in key­point label­ing tools that sup­port skele­ton tem­plates, video inter­po­la­tion and, for 3D work, mul­ti-view or point-cloud views. Out­put is usu­al­ly COCO-style JSON key­points with vis­i­bil­i­ty flags for 2D, and per-frame 3D joint coor­di­nates or hand-mod­el para­me­ters for 3D. Agree the for­mat with your train­ing code before label­ing.

• Skele­ton tem­plates. The tool should enforce the joint order and con­nec­tions from your spec, so anno­ta­tors can­not swap fin­gers.
• Inter­po­la­tion and track­ing. Label­ing keyframes and inter­po­lat­ing between them saves time on slow motion; fast fin­ger motion still needs dense frames.
• Mul­ti-view sup­port. For 3D, anno­ta­tors should see syn­chro­nized views side by side, with repro­ject­ed points to check con­sis­ten­cy.
• Attribute fields. Con­tact, grasp type and object state should be fields on the same time­line, not a sep­a­rate project.
• Export. Test a small export in your data loader first: joint order, image coor­di­nates vs nor­mal­ized val­ues, and frame index­ing.

How does a hand pose annotation project run?

A hand pose anno­ta­tion project runs in five steps: write the label spec, label a pilot batch, cal­i­brate anno­ta­tors against gold exam­ples, label at scale with lay­ered review, then deliv­er in your train­ing for­mat with a QA report. Most rework comes from vague occlu­sion rules, so test them on hard clips dur­ing the pilot.

1. Spec. Agree the GRIP spec and the deliv­ery for­mat.
2. Pilot batch. Label a small set that includes hard cas­es: occlu­sion, fast motion, both hands busy.
3. Cal­i­brate. Build gold clips, mea­sure agree­ment between anno­ta­tors and tight­en the guide­lines.
4. Label and review. Label at scale with a review­er pass and spot audits by a lead.
5. Deliv­er. Export key­points and inter­ac­tion labels in your schema (for exam­ple, COCO-style JSON key­points or a cus­tom per-frame for­mat) with a QA sum­ma­ry.

Bimanual and multi-person scenes

Two hands work­ing togeth­er, or two peo­ple in frame, cre­ate their own label­ing prob­lems. Fix a left/right con­ven­tion from the cam­era wear­er’s point of view, give each hand a per­sis­tent track ID, and decide in the spec whether hands that do not belong to the cam­era wear­er are labeled, skipped or blurred. Record the rule in the GRIP spec so every anno­ta­tor applies it the same way.

How do you measure hand pose annotation quality?

Mea­sure hand pose anno­ta­tion qual­i­ty by com­par­ing labels with gold exam­ples and with a sec­ond anno­ta­tor: key­point dis­tance error in pix­els or mil­lime­ters, agree­ment on vis­i­bil­i­ty flags, and agree­ment on con­tact, grasp type and seg­ment bound­aries. Track results per anno­ta­tor and per clip type, so hard cas­es such as occlu­sion get their own thresh­old.

• Key­point error. Mean dis­tance between labeled and gold key­points, report­ed per joint.
• Vis­i­bil­i­ty agree­ment. Whether anno­ta­tors agree on which joints are hid­den.
• Inter­ac­tion agree­ment. Agree­ment on con­tact flags, grasp types and seg­ment start and end times.
• Tem­po­ral smooth­ness. Jumps between frames that a real hand could not make usu­al­ly sig­nal errors.

How much hand pose data does a robot learning project need?

There is no stan­dard num­ber of labeled frames. The amount depends on how many tasks, objects and set­tings the pol­i­cy must cov­er, whether labels are 2D or 3D, and how much pub­lic data you can reuse for pre­train­ing. Plan in stages: a pilot batch to set­tle the spec, a first train­ing set sized to show whether the labels help, then growth dri­ven by mea­sured gaps.

Cov­er­age usu­al­ly mat­ters more than raw vol­ume. A few thou­sand frames spread across many hands, objects, light­ing con­di­tions and cam­era angles tend to teach more than many more frames from one par­tic­i­pant at one table. Track cov­er­age in meta­da­ta (task, object, set­ting, par­tic­i­pant and cam­era) so you can see where the next batch should go, and keep a fixed test set that is nev­er used for train­ing.

Capture tips that make hand labeling easier

• Keep hands in frame. Set head-mount­ed cam­era angle and height so both hands stay vis­i­ble dur­ing the task.
• Light the work area. Low light caus­es motion blur on fin­gers, the hard­est joints to label.
• Use a high­er frame rate for fast tasks. Fast fin­ger motion between frames makes inter­po­la­tion unre­li­able.
• Cal­i­brate and sync cam­eras. 3D labels depend on intrin­sics, extrin­sics and a shared clock.
• Script the tasks. Clear task steps make action seg­ments and con­tact labels eas­i­er to define.

For rigs, cal­i­bra­tion files and con­sent, see our guide to choos­ing an ego­cen­tric data col­lec­tion ven­dor. For oth­er label types such as box­es and track­ing, see our video anno­ta­tion guide, and for where hand data fits in a robot data mix, our humanoid robot train­ing data guide.

How Graveiens AI approaches hand pose annotation

Graveiens AI’s data anno­ta­tion team labels hand key­points, con­tact, grasp types, object states and action seg­ments to the clien­t’s spec, with lay­ered review and QA against agreed pass cri­te­ria. When a pro­gram also needs footage, we cap­ture it through con­sent-backed ego­cen­tric video data col­lec­tion and keep cap­ture and label­ing under one QA process.

Every pro­gram fol­lows our explic­it-con­sent work­flow (par­tic­i­pant onboard­ing, cap­ture man­age­ment, meta­da­ta tag­ging, QA), and clients are invoiced only for approved deliv­er­ables. Graveiens AI is ISO 9001 cer­ti­fied.

Frequently asked questions

Scope your hand annotation pilot

Send us a few sam­ple clips and your tar­get labels, and we will return a spec and a pilot plan. Explore data anno­ta­tion ser­vices or scope a pilot billed only on approved files.

Sources

1. Simon, T., Joo, H., Matthews, I., Sheikh, Y. “Hand Key­point Detec­tion in Sin­gle Images using Mul­ti­view Boot­strap­ping.” arXiv:1704.07809. arxiv.org/abs/1704.07809
2. Chao, Y.-W. et al. “DexY­CB: A Bench­mark for Cap­tur­ing Hand Grasp­ing of Objects.” arXiv:2104.04631. arxiv.org/abs/2104.04631
3. Liu, Y. et al. “HOI4D: A 4D Ego­cen­tric Dataset for Cat­e­go­ry-Lev­el Human-Object Inter­ac­tion.” arXiv:2203.01577. arxiv.org/abs/2203.01577

About the author

Graveiens AI Edi­to­r­i­al Team. The team writes from day-to-day work run­ning con­sent-backed data pro­grams for AI and robot­ics teams, includ­ing first-per­son video cap­ture, hand and action anno­ta­tion, and mul­ti­lin­gual voice data.