Open-Vocabulary Object Detection for Robots Using Vision-Language Models

Open-Vocabulary Object Detection for Robots Using Vision-Language Models Traditional object detectors are trained on a fixed set of classes. For example: person car chair dog But robots often encounter objects that were not explicitly included in their original training labels. Open-vocabulary perception allows a robot to query concepts using natural language. From Fixed Classes to Natural Language Traditional: Image --> Detector --> {person, car, chair} Open vocabulary: Image + "find a red toolbox" | v Vision-Language Model | v Candidate Regions Robot Perception Pipeline Camera | v Image Preprocessing | v Vision-Language Model | +--> "red toolbox" +--> "safety helmet" +--> "door handle" | v Detected Regions | v 3D Localization | v Robot Planner Why This Matters A robot deployed in the real world may receive commands such as: Find the nearest orange package. The system should not require a new fixed detector class for every possible object. Connecting 2D and 3D A VLM may identify an object in an image. Depth or LiDAR can then estimate its 3D location. RGB Image | v 2D Object Region | +---- Depth | +---- LiDAR | v 3D Object Position This transforms semantic understanding into spatial information. Safety and Verification Open-vocabulary models can produce uncertain or incorrect detections. For robot control, add verification: VLM Detection | v Confidence Check | v Geometric Validation | v Temporal Consistency | v Planner Never assume that a language model's output is automatically safe for direct actuation. ROS 2 Architecture A modular implementation might use: /camera/image | v /vlm_detector | v /detections | v /3d_projection | v /object_tracker | v /planner This makes it possible to replace the VLM without redesigning the rest of the robot stack. Latency Management Large models may be expensive. Possible strategies include: Run perception at a lower frequency. Track detected objects between VLM calls. Resize images. Use hardware acceleration. Cache repeated queries.