Efficient dataset labeling for robotic object perception exploiting eye-in-hand RGB-D camera

Robotic perception faces significant challenges in accurately handling diverse objects with complex geometries, particularly in dynamic environments. Traditional dataset labeling methods struggle with varying object geometries, occlusions, and inefficiencies in annotation. To address these issues, this study proposes an eye-in-hand RGB-D camera setup for efficient dataset labeling, integrating stereo vision and data fusion techniques to enhance perception accuracy. The methodology employs a Universal Robots UR5 robotic arm equipped with a Luxonis OAK-D RGB-D camera to capture multi-view data along an ellipsoidal trajectory. A semi-automated annotation strategy is introduced, leveraging the High-Quality Segment Anything Model (HQ-SAM) for adaptive segmentation. The process begins with sparse manual annotations in the first frame, which are iteratively propagated across subsequent frames using a randomized point selection method. This method refines key points based on depth covariance constraints, ensuring robust adaptation to occlusions and object variations. The proposed approach significantly improves dataset annotation efficiency, reduces manual labeling effort, and enhances segmentation accuracy. By dynamically generating annotation points across frames, the system effectively mitigates occlusion-related errors and ensures consistent labeling across complex object geometries. Experimental results demonstrate the effectiveness of this approach in capturing high-quality RGB-D datasets for robotic perception, particularly for diverse objects with complex structures.