Domino Hands

Overview

Domino Hands is a ROS2 manipulation system that uses a Franka Emika FER arm and an Intel RealSense camera to detect loose dominoes, pick them up, reorient them, place them into configurable patterns, and then topple the first domino. The project was built for Northwestern ME450 by Gregory Aiosa, Daniel Augustin, Michael Jenz, and Chenyu Zhu.

System Architecture

The system is organized as three ROS2 packages: a high-level domino task package, a reusable MoveIt motion-planning package, and a custom service-interface package. The main launch file brings up MoveIt/RViz, RealSense, OpenCV domino detection, AprilTag/easy-handeye calibration, and the placement orchestrator.

• place_dominoes: Orchestrates the full task, loads goal patterns, requests camera detections, manages domino collision objects, and exposes /place_dominoes and /topple_dominoes services.
• find_dominoes: Subscribes to RealSense color/depth topics, detects dominoes with OpenCV, publishes annotated debug images, and broadcasts corrected domino TFs.
• motion_planning: Wraps MoveIt2 pose goals, Cartesian paths, named joint configurations, gripper commands, planning-scene updates, and force-limited descent.
• motion_planning_interfaces: Defines custom services for gripper, box, pose, and Cartesian planning requests.

RealSense RGB-D camera
  |
  v
find_dominoes
  |  HSV + depth + contour orientation
  v
domino TFs in base frame
  |
  v
place_dominoes
  |  pattern YAML + current domino poses
  v
MotionPlanningInterface
  |
  v
MoveIt2 + PlanningScene + Franka gripper
  |
  v
pick -> stage upright -> place -> topple

Launch and Runtime Flow

The same launch file supports simulation and real hardware. In demo mode it starts the Franka MoveIt demo stack. On the real robot it assumes the station is running and launches RViz, RealSense, calibration publishers, perception, and task control.

# Real robot workflow
ros2 launch place_dominoes domino.launch.xml \
  open_cv:=true demo:=false pattern:=straight_line.yaml

ros2 service call /place_dominoes std_srvs/srv/Empty
ros2 service call /topple_dominoes std_srvs/srv/Empty

# Simulation workflow
ros2 launch place_dominoes domino.launch.xml \
  demo:=true scene:=jig_laying.yaml pattern:=straight_line.yaml

ros2 service call /place_dominoes std_srvs/srv/Empty

Perception Pipeline

The perception node is intentionally fast and inspectable. It uses depth slicing to isolate the table region, HSV filtering to segment domino-colored pixels, contour geometry to reject non-domino shapes, and RealSense intrinsics to convert the center pixel into a 3D point.

• Depth clipping rejects background pixels outside the useful workspace before color segmentation.
• HSV thresholds are exposed as ROS parameters so color tuning can happen live without changing code.
• OpenCV minAreaRect gives both the center and planar orientation of each domino.
• Duplicate detections are clustered so one physical domino does not publish multiple active TFs.

RGB image + aligned depth image + camera intrinsics
  |
  v
depth slice: keep points within table workspace
  |
  v
HSV threshold + blur + morphology
  |
  v
find contours and min-area rectangles
  |
  v
filter by area and long/short side ratio
  |
  v
center pixel + depth -> 3D camera-frame point
  |
  v
rectangle angle -> yaw quaternion
  |
  v
camera TF -> base TF -> domino0, domino1, ...

Perception Tradeoffs

OpenCV RGB-D Detection

OpenCV was the right tool for this project because dominoes are small, close to the camera, and visually distinct from the table under controlled lighting. The detector is simple enough to debug from intermediate image topics.

Hand-Eye and AprilTag Calibration

The camera pose is published through easy_handeye2 calibration, with AprilTags available during calibration. Accurate extrinsics are critical because the manipulation node consumes domino TFs in the robot base frame, not image coordinates.

Pattern Generation

Goal patterns are loaded from YAML files as named domino poses. The project includes straight-line, circle, squiggle, spiral, and dense generated patterns. Each goal domino has a size, position, and quaternion orientation, which allows the planner to treat the future layout as a concrete set of target poses.

• straight_line.yaml: 8 dominoes in a simple line for reliable hardware demos.
• circle.yaml: 15 dominoes arranged along a circular arc.
• squiggle.yaml: 10 dominoes generated with turn-angle logic.
• spiral.yaml and domino_positions.yaml: larger generated patterns for simulation and planning experiments.

pattern YAML
  |
  v
goal_domino_0 ... goal_domino_n
  |
  v
publish translucent RViz markers
  |
  v
pair start_domino_i with goal_domino_i
  |
  v
move each physical domino into the selected pattern

Manipulation Pipeline

The core challenge is that a domino is too small to simply pick up flat and place into a tight final pattern. The gripper can collide with neighboring dominoes, and a flat grasp does not give the arm a good final placement posture. The solution is a staged regrasp pipeline.

1. 1Initial Pickup: Align the gripper using the detected domino TF and close around the domino.
2. 2Staging: Move to a separate staging area and convert laying dominoes into upright dominoes.
3. 3Final Regrasp: Grab the upright domino in a placement-friendly orientation.
4. 4Final Placement: Lower into the selected pattern with collision-aware planning and force confirmation.

For each domino:
  1. Move above the table and collect current domino TFs.
  2. Move to the next detected domino and align the gripper to its long axis.
  3. Pick the domino from its start pose.
  4. If laying down, move to a staging area and stand it upright.
  5. Regrasp the upright domino with a placement-friendly top-down grip.
  6. Move above the matching goal pose.
  7. Lower through a Cartesian path when possible.
  8. Detach, force-seat on the table, respawn the collision object, and open the gripper.
  9. Return to ready and repeat.

Planning Strategy

MoveIt Pose Goals

Pose goals are used for most large moves: picture pose, ready pose, above-start pose, staging pose, and above-goal pose. Each plan is retried repeatedly because manipulation around a dense table scene can fail due to IK, orientation constraints, or collision geometry.

Cartesian Descent

For short vertical moves, the system prefers Cartesian waypoints. This is especially important when lowering a domino into a goal pose because the end-effector should descend predictably instead of approaching from a strange IK path.

Force-Sensing Placement

Hard-coded table heights were not reliable because the real workspace surface was not perfectly flat. The fix was a force-limited descent loop: the arm lowers in 1 mm increments at slow velocity while monitoring Franka controller effort. When the effort crosses a threshold, the robot treats that as table contact.

target placement pose
  |
  v
temporarily remove table collision object
  |
  v
lower 1 mm at a time
  |
  v
monitor fer_joint2 effort
  |
  +-- effort below threshold --> continue lowering
  |
  +-- effort above threshold --> contact confirmed
  |
  v
restore table collision object

Dynamic Planning Scene

The planning scene is not static. Dominoes are added as boxes, attached to the gripper during transport, detached at the goal, removed during force placement, and respawned afterward. This keeps MoveIt aware of carried objects while still allowing intentional table contact.

• Table and dominoes are loaded from YAML into the MoveIt planning scene.
• Domino collision objects are attached to fer_hand_tcp while the robot is carrying them.
• Objects are removed before force-seating to prevent planning-scene conflicts with real contact.
• After placement, the domino is respawned in RViz/MoveIt at the corrected final pose.

Toppling Routine

After placement, the /topple_dominoes service removes the first domino collision object, computes a point behind the first goal domino based on its orientation, moves the gripper behind it, and pushes through the domino to start the chain reaction.

first goal domino pose
  |
  v
compute domino local axis in world frame
  |
  v
choose approach point behind domino
  |
  v
move gripper to hover point
  |
  v
move into domino to topple pattern

Key Technical Challenges

• Tiny object scale: a few millimeters of perception or calibration error can cause failed grasps or poor placement.
• Camera calibration: hand-eye calibration quality directly affects every detected domino TF.
• Dense placement: the Franka gripper is large relative to domino spacing, requiring staging and top-down regrasping.
• MoveIt reliability: Cartesian, pose, and named-configuration plans are combined with retries and joint-limit checks.
• Reality gap: the table and domino collision objects are useful for planning, but must be removed during intentional force contact.

Quickstart

# Real robot demo with camera-based detection
ros2 launch place_dominoes domino.launch.xml \
  open_cv:=true demo:=false pattern:=straight_line.yaml

ros2 service call /place_dominoes std_srvs/srv/Empty
ros2 service call /topple_dominoes std_srvs/srv/Empty

# Simulation demo with a predefined starting scene
ros2 launch place_dominoes domino.launch.xml \
  demo:=true scene:=jig_laying.yaml pattern:=straight_line.yaml

ros2 service call /place_dominoes std_srvs/srv/Empty