Imported from robotics-playground/skills (skills/cv-detection/SKILL.md). Install upstream with npx skills add robotics-playground/skills --skill cv-detection. Copyright stays with the author.

Computer Vision & Object Detection for ROS 2

Quick navigation

• OpenCV integration, cv_bridge, image_transport, calibration -> references/opencv-ros2.md
• YOLO inference, training, annotation, defect detection -> references/yolo-inference.md
• OAK-D, depth images, point clouds, stereo vision -> references/depth-sensing.md
• Jetson optimization, TensorRT, Isaac ROS, profiling -> references/jetson-optimization.md

Who This Skill Is For

This skill is written for developers new to computer vision. If you have general programming experience but have never worked with image tensors, camera intrinsics, or bounding boxes, this is your starting point.

An inference pipeline is just a data pipeline. A trained model is just a function that takes pixels and returns structured data. Annotations are just labeled training examples.

Vision Pipeline Architecture

Every vision system in ROS 2 follows this pipeline:

Camera Hardware          ROS 2 Transport         Processing              Action
+-------------+    +--------------------+    +----------------+    +-------------+
| USB Camera  | -> | /camera/image_raw  | -> | OpenCV resize, | -> | Publish     |
| OAK-D Lite  |    | sensor_msgs/Image  |    | color convert  |    | detections  |
| RealSense   |    +--------------------+    +-------+--------+    | to Nav2     |
+-------------+                                      |             | or operator |
                                                     v             +-------------+
                                              +----------------+
                                              | YOLO inference |
                                              | or classical   |
                                              | CV detection   |
                                              +----------------+

ROS 2 topics work as pub/sub message channels. The camera publishes images to a topic. Your detection node subscribes, processes each frame, and publishes results to another topic. Downstream nodes (navigation, alerts, UI) subscribe to your detection results.

Pipeline Stages

Which Detection Approach?

Not every problem needs deep learning. This table helps you choose:

Decision Flowchart

Is the target visually simple? (solid color, straight edge, known shape)
+-- YES --> Classical CV with OpenCV
|           Fast, no training data, runs on any CPU
|
+-- NO  --> Do you need to detect multiple defect types?
            +-- YES --> YOLO (YOLOv8 or YOLOv11)
            |           Best balance of accuracy and speed
            |           See references/yolo-inference.md
            |
            +-- NO  --> Do you need pixel-level segmentation?
                        +-- YES --> YOLO-Seg or custom segmentation model
                        +-- NO  --> Start with YOLO, it handles most cases

For inspection and defect detection: Start with YOLOv8 or YOLOv11. Real-world defects (cracks, corrosion, deformations, foreign objects) are visually complex and vary widely. Classical CV cannot reliably detect them. YOLO gives you multi-class detection with bounding boxes, which is exactly what an operator or autonomous system needs.

ROS 2 Image Format Guide

Images in ROS 2 are structured messages. Understanding the formats prevents hours of debugging.

sensor_msgs/msg/Image

The primary image message. Fields that matter:

Common Encodings

The #1 image encoding bug: OpenCV uses BGR channel order. If your image looks blue-orange instead of normal colors, you are displaying RGB data as BGR (or vice versa). Use cv2.cvtColor(img, cv2.COLOR_RGB2BGR) to convert.

sensor_msgs/msg/CompressedImage

Smaller than raw images. Use for bandwidth-constrained links (WiFi, remote monitoring).

When to use compressed: Always for remote viewing or recording. JPEG at quality 80 is roughly 10x smaller than raw. For local processing between nodes on the same machine, raw is fine and avoids decode overhead.

sensor_msgs/msg/CameraInfo

Camera intrinsic parameters. Published alongside images. You need this for:

• Undistorting images (removing lens distortion)
• Converting pixel coordinates to 3D rays
• Projecting 3D points onto the image

Common Vision Failure Diagnosis

When something goes wrong (and it will), start here:

Camera Selection Guide

Choosing the right camera depends on your use case:

Recommendation for Inspection Robots

Primary: OAK-D Pro ($300). The active IR illumination is valuable for low-light environments. You get:

• Color camera for visual inspection and defect detection
• Stereo depth for obstacle avoidance and dimensional measurement
• On-device neural inference (run YOLO directly on the camera's VPU)
• Built-in IR illumination for low-light conditions

Budget alternative: OAK-D Lite ($150) with external lighting. You lose the IR depth assist but can add your own lighting. Good for prototyping before committing to the Pro.

For initial prototyping: A USB webcam ($60) is fine for developing your detection pipeline. You can train and test your YOLO model on webcam footage before deploying to the OAK-D.

Image Processing Performance Rules

These rules prevent the most common performance problems:

1. Never Queue Images

# BAD: default QoS queues images, subscriber falls behind
self.sub = self.create_subscription(Image, '/camera/image_raw', self.callback, 10)

# GOOD: only process the latest frame, drop old ones
from rclpy.qos import QoSProfile, QoSReliabilityPolicy, QoSHistoryPolicy
qos = QoSProfile(
    reliability=QoSReliabilityPolicy.BEST_EFFORT,
    history=QoSHistoryPolicy.KEEP_LAST,
    depth=1
)
self.sub = self.create_subscription(Image, '/camera/image_raw', self.callback, qos)

2. Resize Before Processing

Inference on a 1920x1080 image is 4x slower than 640x480. YOLO resizes internally, but you save memory and transfer time by resizing early.

3. Use Compressed Transport for Remote Viewing

# Remap to use compressed transport
ros2 run image_transport republish raw compressed \
  --ros-args -r in:=/camera/image_raw -r out/compressed:=/camera/image_raw/compressed

4. Profile Before Optimizing

Measure actual FPS before changing anything. The bottleneck might not be where you think.

import time

class DetectionNode(Node):
    def __init__(self):
        super().__init__('detection_node')
        self.last_time = time.time()
        self.frame_count = 0

    def callback(self, msg):
        self.frame_count += 1
        if self.frame_count % 30 == 0:
            now = time.time()
            fps = 30.0 / (now - self.last_time)
            self.get_logger().info(f'Processing at {fps:.1f} FPS')
            self.last_time = now

Defect Detection: Quick Start Path

Here is the recommended progression for building an inspection/detection system:

Phase 1: Camera Integration (Week 1)

1. Get a USB webcam publishing images in ROS 2 (usb_cam package)
2. Write a node that subscribes to images and displays them with OpenCV
3. Learn cv_bridge and image encodings
4. See references/opencv-ros2.md

Phase 2: Classical CV Experiments (Week 2)

1. Implement edge detection on camera feed (Canny, find contours)
2. Implement color-based detection (HSV thresholding for water/sediment)
3. Learn image preprocessing (resize, blur, threshold)
4. See references/opencv-ros2.md

Phase 3: YOLO Training and Inference (Weeks 3-4)

1. Collect or download domain-specific images (Roboflow, public datasets)
2. Annotate images with defect bounding boxes
3. Train YOLOv8 on your dataset
4. Create a ROS 2 inference node
5. See references/yolo-inference.md

Phase 4: Depth Integration (Week 5)

1. Set up OAK-D Lite with depthai_ros
2. Combine color detection with depth measurement
3. Publish defect detections with 3D positions
4. See references/depth-sensing.md

Phase 5: Jetson Deployment (Week 6)

1. Export YOLO model to TensorRT
2. Deploy on Jetson Orin NX/Nano
3. Optimize for real-time inference (15+ FPS target)
4. See references/jetson-optimization.md

Key Vocabulary

Common ROS 2 Vision Packages

Reference File Index