PTZ Camera + LiDAR Multi-Domain Sensor Fusion Node
Spatial perception and tracking architecture integrating 3D LiDAR point clouds with PTZ optical cameras via ROS 2 and Extended Kalman Filtering.
01. Problem & Operational Context
Optical PTZ cameras provide detailed semantic classification but lack direct metric depth, while LiDAR point clouds offer accurate range but lack dense semantic labels; calibrating moving PTZ optical axes with LiDAR presents spatial-temporal synchronization challenges.
02. Technical Approach
Developed a modular ROS 2 sensor fusion node that resolves range and bearing accuracy issues by projecting calibrated 3D LiDAR point clusters onto dynamic PTZ camera image planes and tracking targets via an Extended Kalman Filter.
03. System Architecture & Pipeline
End-to-End System Pipeline & Computation Flow
ROS 2 message passing graph synchronizing LiDAR point cloud clusters with dynamic PTZ camera zoom and pan/tilt coordinate transformations.
Point Cloud Preprocessing
Voxel filtering, passthrough boundaries, and obstacle Euclidean cluster extraction.
Dynamic PTZ Frame Transformation
Real-time tf2 coordinate transform computation accounting for PTZ pan, tilt, and optical zoom factor.
Spatial Sensor Association
Geometric projection associating 3D LiDAR spatial centroids with 2D camera bounding boxes.
State Estimation & Filtering
Extended Kalman Filter estimating 3D position coordinates and velocity vectors.
04. Key Implementation Details
- Developed ROS 2 composition nodes with time-synchronized message filters to temporally align high-frequency LiDAR scans with camera frames.
- Resolved range and bearing estimation discrepancies between sensors for multi-domain target tracking.
- Integrated dynamic PTZ coordinate transforms updating camera projection matrices in real time as pan/tilt motors actuate.
- Implemented Extended Kalman Filter tracking state space variables $(x, y, z, dot{x}, dot{y}, dot{z})$ under measurement noise.
05. Architectural Tradeoffs & Decisions
Enables independent sensor diagnostics, distributed processing across compute nodes, and hot-swapping sensors.
Monolithic single-process executable which risked complete pipeline crash on sensor disconnect.
06. Engineering Challenges & Failures
07. Evaluation & Observations
Tested range and bearing accuracy across multi-domain target approaches and dynamic PTZ pan/tilt maneuvers.
Sensor fusion maintained target lock during visual dropouts by propagating LiDAR kinematic state through the EKF.
Publisher/subscriber architecture linking LiDAR PCL preprocessing, PTZ tf2 transforms, and EKF node.