标签：#vlm

Building Autonomous Robot Decision Systems with Vision-Language-Action Models A traditional robot pipeline often separates perception, planning, and control. Vision-Language-Action (VLA) systems aim to connect visual observations and language instructions with actions. Vision + Language | v VLA Model | v Robot Actions From Perception to Action Traditional architecture: Camera -> Detector -> Planner -> Controller A VLA-oriented architecture can be: Camera | v Visual Representation | +------ Language Instruction | v VLA Model | v Action Proposal | v Safety Layer | v Robot Example Instruction: "Pick up the blue box and place it on the table." The system needs to connect: "blue box" to a visual object. "table" to a destination. "pick up" to manipulation. "place" to a sequence of actions. Action Abstraction Do not expose raw motor commands directly to a language model. Instead use an action interface: PICK(object_id) MOVE_TO(location_id) PLACE(object_id, location_id) STOP() This creates a safer boundary between AI reasoning and robot control. ROS 2 Architecture /camera | v /perception | v /vla_agent <--- /task_instruction | v /action_server | v /navigation /manipulation ROS 2 actions are useful for long-running operations such as navigation and manipulation. Safety Layer A robust system should validate AI-generated actions. VLA Proposal | v Schema Validation | v Capability Check | v Collision / Safety Check | v Execution The model should not be able to bypass safety constraints. Handling Uncertainty The robot may need to ask for clarification: Model: "I found two blue boxes." Robot: "Which box should I pick?" This is preferable to silently choosing an unsafe action. Real-Time Architecture Keep high-frequency control loops independent from the VLA model. Fast Loop: Sensors -> Controller -> Motors Slow Loop: Camera -> VLA -> Task Planning A language model should not be placed directly inside a millisecond-level motor-control loop unless the entire system is specifically de

Open-Vocabulary Object Detection for Robots Using Vision-Language Models Traditional object detectors are trained on a fixed set of classes. For example: person car chair dog But robots often encounter objects that were not explicitly included in their original training labels. Open-vocabulary perception allows a robot to query concepts using natural language. From Fixed Classes to Natural Language Traditional: Image --> Detector --> {person, car, chair} Open vocabulary: Image + "find a red toolbox" | v Vision-Language Model | v Candidate Regions Robot Perception Pipeline Camera | v Image Preprocessing | v Vision-Language Model | +--> "red toolbox" +--> "safety helmet" +--> "door handle" | v Detected Regions | v 3D Localization | v Robot Planner Why This Matters A robot deployed in the real world may receive commands such as: Find the nearest orange package. The system should not require a new fixed detector class for every possible object. Connecting 2D and 3D A VLM may identify an object in an image. Depth or LiDAR can then estimate its 3D location. RGB Image | v 2D Object Region | +---- Depth | +---- LiDAR | v 3D Object Position This transforms semantic understanding into spatial information. Safety and Verification Open-vocabulary models can produce uncertain or incorrect detections. For robot control, add verification: VLM Detection | v Confidence Check | v Geometric Validation | v Temporal Consistency | v Planner Never assume that a language model's output is automatically safe for direct actuation. ROS 2 Architecture A modular implementation might use: /camera/image | v /vlm_detector | v /detections | v /3d_projection | v /object_tracker | v /planner This makes it possible to replace the VLM without redesigning the rest of the robot stack. Latency Management Large models may be expensive. Possible strategies include: Run perception at a lower frequency. Track detected objects between VLM calls. Resize images. Use hardware acceleration. Cache repeated queries.