Skip to content

ReKep

Keypoint constraints for robot manipulation

Visit official site

What rekep is

The system represents a manipulation task as constraints that relate the robot to its environment, then optimizes them into multi-stage, bimanual and reactive behavior.

What you can do with it

  • Generate keypoint candidates from RGB-D data and language
  • Optimize actions with an off-the-shelf solver
  • Replan within and across stages in real time

Getting started

  • Prepare RGB-D observations and a free-form instruction
  • Identify scene keypoints with DINOv2
  • Feed them to GPT-4o, then solve the actions

Pros & cons

✓ What we like

  • No manual labels or task-specific training are needed
  • Real-time closed-loop replanning covers stages

! What to watch out for

  • RGB-D observations and a language instruction are required inputs
  • Keypoint candidates depend on the observed scene

FAQ

What generates the constraints?

The material names DINOv2 for keypoints and GPT-4o for turning them into constraints.

Does it work in the wild?

The material describes manipulation in in-the-wild environments with changing object positions.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review