标签：#Robot

Waymo argued that fully autonomous vehicles aren't possible without using a mix of sensors and warned that pure end-to-end AI systems aren't safe enough.

From Natural Language to Robot Actions with Physical Foundation Models Physical AI aims to connect intelligence with real-world action. A user might say: "Bring me the bottle from the kitchen." A robot must turn that high-level instruction into a sequence of grounded actions. The Full Pipeline Natural Language | v Task Understanding | v World Model | v Task Planning | v Motion Planning | v Control | v Physical Robot The important insight is that language understanding alone is not enough. Grounding Language in the World Consider: "Pick up the bottle." The system must identify: Which bottle? Where is it? Can the robot reach it? Is the gripper suitable? Is the route collision-free? Therefore: Language + Vision + Robot State + Environment Model | v Grounded Action Action Representation A foundation model can produce structured actions rather than motor commands: { "action" : "pick" , "object" : "bottle" , "location" : "kitchen_counter" } The robotics stack then translates this into navigation and manipulation primitives. Hierarchical Planning A high-level instruction can be decomposed: Bring bottle | +--> Navigate to kitchen | +--> Find bottle | +--> Reach bottle | +--> Grasp bottle | +--> Navigate to user | +--> Release bottle Each subtask can be executed and verified independently. Connecting to ROS 2 /natural_language_task | v /task_planner | v /world_model | v /action_executor / v v /navigation /manipulation Verification Loop Physical AI should use closed-loop execution: Plan | v Execute | v Observe | v Verify | +---- success ---> Next Step | +---- failure ---> Replan This is critical because the physical world is uncertain. A grasp may fail. An obstacle may move. A door may be closed. Safety Boundaries Foundation models should operate behind explicit constraints: Allowed actions Workspace limits Collision checking Velocity limits Force limits Emergency stop Human approval for sensitive actions Production Architecture Separate responsibilities: Foundation Model | |

Building Autonomous Robot Decision Systems with Vision-Language-Action Models A traditional robot pipeline often separates perception, planning, and control. Vision-Language-Action (VLA) systems aim to connect visual observations and language instructions with actions. Vision + Language | v VLA Model | v Robot Actions From Perception to Action Traditional architecture: Camera -> Detector -> Planner -> Controller A VLA-oriented architecture can be: Camera | v Visual Representation | +------ Language Instruction | v VLA Model | v Action Proposal | v Safety Layer | v Robot Example Instruction: "Pick up the blue box and place it on the table." The system needs to connect: "blue box" to a visual object. "table" to a destination. "pick up" to manipulation. "place" to a sequence of actions. Action Abstraction Do not expose raw motor commands directly to a language model. Instead use an action interface: PICK(object_id) MOVE_TO(location_id) PLACE(object_id, location_id) STOP() This creates a safer boundary between AI reasoning and robot control. ROS 2 Architecture /camera | v /perception | v /vla_agent <--- /task_instruction | v /action_server | v /navigation /manipulation ROS 2 actions are useful for long-running operations such as navigation and manipulation. Safety Layer A robust system should validate AI-generated actions. VLA Proposal | v Schema Validation | v Capability Check | v Collision / Safety Check | v Execution The model should not be able to bypass safety constraints. Handling Uncertainty The robot may need to ask for clarification: Model: "I found two blue boxes." Robot: "Which box should I pick?" This is preferable to silently choosing an unsafe action. Real-Time Architecture Keep high-frequency control loops independent from the VLA model. Fast Loop: Sensors -> Controller -> Motors Slow Loop: Camera -> VLA -> Task Planning A language model should not be placed directly inside a millisecond-level motor-control loop unless the entire system is specifically de

Model Predictive Control for Real-Time Robot Navigation A path planner tells a robot where it should go. A controller determines how the robot should move to follow that path. Model Predictive Control (MPC) repeatedly predicts future behavior and chooses control inputs that optimize a short horizon. MPC Concept Current State | v Predict future states | v Optimize control sequence | v Apply first control | v Measure new state | +----> Repeat The key idea is that the entire control sequence is not executed at once. Only the first action is applied before the problem is solved again. Robot Model For a simple differential-drive robot: x_dot = v cos(theta) y_dot = v sin(theta) theta_dot = omega The controller can predict where the robot will be after applying candidate velocity commands. Optimization Objective A typical objective might penalize: Distance from reference path Heading error Excessive control effort Rapid control changes Collision proximity Conceptually: Cost = tracking_error + control_effort + smoothness_penalty + obstacle_penalty Prediction Horizon Suppose the controller predicts: t0 -> t1 -> t2 -> t3 -> t4 For each candidate control sequence it estimates the resulting trajectory. The optimizer selects the best feasible sequence. Obstacle Handling A cost function can strongly penalize trajectories near obstacles: Obstacle ### ##### ### \ predicted trajectories \---- safe \--- unsafe Hard constraints can also be used when collision avoidance must be guaranteed by the optimization formulation. ROS 2 Architecture /global_plan | v /mpc ^ | /odom /imu /local_costmap | v /cmd_vel Real-Time Requirements MPC is computationally heavier than simple feedback controllers. Monitor: Optimization time Control frequency Solver failures CPU utilization Prediction horizon Sensor latency If optimization misses its deadline, the system needs a safe fallback. Practical Implementation Strategy Start simple: Define a robot model. Implement trajectory prediction. Define tracking

Implementing A* and RRT Motion Planning for Robotics Two classic planning approaches are A * and RRT (Rapidly-exploring Random Tree) . A* is particularly useful when the environment can be represented as a graph or grid. RRT is useful when planning in continuous or high-dimensional configuration spaces. A* Planning A* combines the cost already traveled with an estimate of the remaining cost. Conceptually: f(n) = g(n) + h(n) Where: g(n) is the cost from the start. h(n) estimates the cost to the goal. f(n) ranks candidate nodes. Grid Example S . . # . . . . . . # . . . . . . . . # . . # # # . # . . . . . . . G The planner explores promising cells while avoiding blocked cells. Python Implementation Skeleton import heapq def astar ( graph , start , goal , heuristic ): queue = [( 0 , start )] cost = { start : 0 } parent = { start : None } while queue : _ , current = heapq . heappop ( queue ) if current == goal : break for neighbor in graph [ current ]: new_cost = cost [ current ] + 1 if neighbor not in cost or new_cost < cost [ neighbor ]: cost [ neighbor ] = new_cost priority = new_cost + heuristic ( neighbor , goal ) heapq . heappush ( queue , ( priority , neighbor )) parent [ neighbor ] = current return parent RRT Planning RRT works differently. Instead of systematically exploring grid cells, it samples points and gradually grows a tree. x / x------x / S-----x x----x------G A typical loop is: Sample a random configuration. Find the nearest existing node. Steer toward the sample. Check collision. Add the new node if valid. Repeat until the goal is reached. RRT Skeleton for _ in range ( max_iterations ): sample = random_configuration () nearest = nearest_node ( tree , sample ) new_node = steer ( nearest , sample ) if collision_free ( nearest , new_node ): tree . add ( new_node ) tree . connect ( nearest , new_node ) if reached_goal ( new_node ): return extract_path ( tree , new_node ) A* vs RRT Property A* RRT Representation Grid/graph Continuous space Search Determinis

Building Global and Local Path Planners for Autonomous Robots Autonomous navigation is not just about finding a route from A to B. A robot must plan a useful route through a map and continuously adapt that route to obstacles, other robots, people, and changes in its environment. A practical navigation system therefore separates global planning from local planning . Global vs Local Planning Global Map | v +------------------+ | Global Planner | +------------------+ | v Global Path | v +------------------+ Sensors>| Local Planner | +------------------+ | v Velocity Commands | v Robot Global Planner The global planner considers the larger environment. Its job is typically to find a route such as: Start ---> Corridor ---> Door ---> Room ---> Goal Common approaches include: A* Dijkstra Graph search Grid-based planning Sampling-based planning Local Planner The local planner operates closer to the robot and reacts to current observations. It considers: Nearby obstacles Robot velocity Robot footprint Dynamic objects Current trajectory Short-term goal direction Why Both Are Needed Suppose the global path is: Robot -----> Hallway -----> Goal A person suddenly walks into the hallway. The global route may still be valid, but the robot needs to slow down, stop, or temporarily move around the person. That is the local planner's job. Grid-Based Global Planning Represent the environment as a costmap: . . . . . . . . . # # . . . . . # # . . . . . . . . . . . . . . . G . S . . . . . . A planner searches through free cells while assigning higher costs to undesirable regions. Local Planning A local planner can generate multiple candidate trajectories: obstacle ### Robot --> / | \ / | / | candidate trajectories Each trajectory can be scored based on: Collision risk Distance to path Distance to goal Smoothness Velocity Clearance ROS 2 Architecture /map | v /global_planner | v /global_plan | v /local_planner <--- /scan /pointcloud | v /cmd_vel Keep the global and local planners modular so

Open-Vocabulary Object Detection for Robots Using Vision-Language Models Traditional object detectors are trained on a fixed set of classes. For example: person car chair dog But robots often encounter objects that were not explicitly included in their original training labels. Open-vocabulary perception allows a robot to query concepts using natural language. From Fixed Classes to Natural Language Traditional: Image --> Detector --> {person, car, chair} Open vocabulary: Image + "find a red toolbox" | v Vision-Language Model | v Candidate Regions Robot Perception Pipeline Camera | v Image Preprocessing | v Vision-Language Model | +--> "red toolbox" +--> "safety helmet" +--> "door handle" | v Detected Regions | v 3D Localization | v Robot Planner Why This Matters A robot deployed in the real world may receive commands such as: Find the nearest orange package. The system should not require a new fixed detector class for every possible object. Connecting 2D and 3D A VLM may identify an object in an image. Depth or LiDAR can then estimate its 3D location. RGB Image | v 2D Object Region | +---- Depth | +---- LiDAR | v 3D Object Position This transforms semantic understanding into spatial information. Safety and Verification Open-vocabulary models can produce uncertain or incorrect detections. For robot control, add verification: VLM Detection | v Confidence Check | v Geometric Validation | v Temporal Consistency | v Planner Never assume that a language model's output is automatically safe for direct actuation. ROS 2 Architecture A modular implementation might use: /camera/image | v /vlm_detector | v /detections | v /3d_projection | v /object_tracker | v /planner This makes it possible to replace the VLM without redesigning the rest of the robot stack. Latency Management Large models may be expensive. Possible strategies include: Run perception at a lower frequency. Track detected objects between VLM calls. Resize images. Use hardware acceleration. Cache repeated queries.

3D Object Detection for Physical AI Applications A robot needs more than image classification. It needs to know: What object is present? Where is it? How large is it? How is it oriented? 3D object detection answers these questions in physical space. 3D Detection Pipeline Camera / LiDAR | v Preprocessing | v Feature Extraction | v 3D Detection Model | v 3D Bounding Boxes | v Tracking / Planning A 3D bounding box can contain: (x, y, z) (width, height, depth) (rotation) (class) (confidence) LiDAR-Based Detection LiDAR naturally provides 3D geometry. A typical pipeline is: Point Cloud | v Filtering | v Voxelization / Features | v Neural Network | v 3D Boxes Challenges include sparse points and computational cost. Camera-Based Detection A camera provides dense visual information. Monocular 3D detection tries to infer depth from a single image, while stereo systems can estimate depth geometrically. Multi-Modal Detection Combining cameras and LiDAR can provide both semantics and geometry: Camera ---> Visual Features --+ | LiDAR ----> 3D Features ------+--> Fusion --> 3D Detection This is useful for autonomous robots operating around people, vehicles, and objects. Post-Processing Raw detections are often filtered using: Confidence thresholds Non-maximum suppression Geometric constraints Temporal tracking Tracking can stabilize detections across frames. ROS 2 Integration A practical architecture: /sensors/camera /sensors/lidar | v /3d_detector | v /objects_3d | +--> /tracker | +--> /planner Use standardized message structures where practical so perception remains decoupled from planning. Measuring Performance Evaluate: Precision Recall 3D IoU Position error Orientation error Inference latency FPS For physical AI, latency matters almost as much as accuracy. A detector that is accurate but too slow can still be unsuitable for a moving robot. Production Considerations Test across: Day/night conditions Different sensor placements Partial occlusion Different object sizes Dynamic

Building a Real-Time SLAM System for Mobile Robots SLAM means Simultaneous Localization and Mapping . A mobile robot must answer two questions: Where am I? What does the environment look like? The challenge is that the robot needs the map to localize while also needing localization to build the map. SLAM Architecture Sensors | +--> Frontend | | | +--> Odometry | +------------------+ v State Estimator | v Map Builder | v Map Sensor Options Typical systems use: 2D LiDAR 3D LiDAR Cameras IMUs Wheel encoders The right sensor combination depends on the environment. SLAM Frontend The frontend extracts motion constraints. For LiDAR: Scan | v Feature / Point Processing | v Scan Matching | v Relative Motion For visual SLAM: Image | v Feature Extraction | v Feature Matching | v Relative Pose Backend Optimization The backend can represent the robot trajectory as a graph: Pose 1 ---- Pose 2 ---- Pose 3 ---- Pose 4 \ / +------ Loop Closure ---+ Loop closure recognizes that the robot has returned to a previously observed location. This can significantly reduce accumulated drift. Real-Time Constraints SLAM is not useful if it produces excellent maps several seconds too late. Monitor: Sensor processing latency Pose estimation latency Map update time CPU/GPU utilization Queue sizes Frame/scan drops Map Resolution Higher resolution gives more detail but costs more memory and computation. Choose resolution based on: Robot size Environment Navigation requirements Available compute Failure Modes SLAM can struggle with: Repetitive environments Dynamic objects Feature-poor walls Rapid motion Poor sensor calibration Incorrect timestamps A robust system should monitor confidence and detect tracking failures. Production Pipeline Camera / LiDAR / IMU | v Sensor Calibration | v Odometry Frontend | v Pose Estimation | v Loop Detection | v Graph Optimization | v Map Server | v Navigation The goal of production SLAM is not just map quality. It is stable localization, predictable latency, and grac

Visual-Inertial Odometry for Autonomous Robots A robot needs to estimate how it moves through the world. GPS is unavailable indoors, wheel odometry can slip, and LiDAR may not always be available. Visual-Inertial Odometry (VIO) combines cameras and IMUs to estimate motion. Basic Idea Camera ---> Visual Features ---+ | v State Estimator ^ | IMU ----> Motion Information ---+ | v Robot Trajectory The camera provides visual constraints. The IMU provides high-frequency motion measurements. Why Combine Them? A camera gives rich spatial information but can suffer from: Motion blur Low texture Poor lighting Slow frame rate An IMU operates at much higher rates but accumulates drift when integrated over time. Their weaknesses are complementary. Feature Tracking A simple visual pipeline might be: Image | v Feature Detection | v Feature Tracking | v Motion Estimation Feature types may include corners or learned visual features. IMU Prediction The IMU can predict how the robot's state changes between camera frames. Conceptually: Previous State | +--> IMU measurements | v Predicted State | +--> Camera observation | v Corrected State This is the prediction/correction pattern used by many estimators. Initialization VIO initialization is important because the estimator must determine quantities such as: Initial orientation Gravity direction Velocity Scale for monocular systems Sensor biases Poor initialization can cause instability later. ROS 2 Architecture /camera/image | v Visual Frontend ----+ | /imu/data ----------+--> VIO Estimator --> /odometry | +--> /tf Use consistent timestamps and calibrated camera-IMU extrinsics. Improving Robustness Useful techniques include: Rejecting outlier feature matches Monitoring IMU saturation Handling dropped frames Estimating sensor biases Detecting low-texture scenes Monitoring estimator health Evaluation Evaluate against a trusted trajectory where available. Useful metrics include: Absolute trajectory error Relative pose error Drift per dista

ROS 2 QoS Profiles: Reliable vs Best-Effort Robot Communication Robot systems continuously exchange data with very different requirements. A dropped camera frame is usually acceptable. A dropped emergency command may not be. ROS 2 Quality of Service (QoS) lets you express these requirements. The Two Common Reliability Modes Reliable Reliable communication attempts to ensure that samples reach compatible subscribers. Useful for: Commands Configuration Important state transitions Critical application data Best Effort Best effort prioritizes timely delivery and may tolerate lost samples. Useful for: Cameras LiDAR High-frequency IMU streams Other continuously refreshed sensor data Example Imagine a camera producing 30 frames per second. If frame 100 is lost, the system can often process frame 101 immediately. For a command: MOVE_FORWARD losing the message may be unacceptable. Therefore: Camera -> Best Effort Command -> Reliable is often a sensible starting point. QoS Dimensions Reliability is only one QoS policy. Important policies include: Reliability Durability History Depth Deadline Lifespan Liveliness C++ Example auto sensor_qos = rclcpp :: SensorDataQoS (); auto publisher = create_publisher < sensor_msgs :: msg :: Image > ( "/camera/image" , sensor_qos ); For important application data, you might explicitly configure reliable communication: auto qos = rclcpp :: QoS ( rclcpp :: KeepLast ( 10 )) . reliable (); auto publisher = create_publisher < std_msgs :: msg :: String > ( "/robot/status" , qos ); QoS Compatibility A publisher and subscriber need compatible QoS settings. A common mistake is: Publisher: Best Effort Subscriber: Reliable and then wondering why messages are not received as expected. Always inspect the effective QoS of both endpoints. A Practical Decision Table Topic Suggested Starting Point Camera image Best Effort Point cloud Best Effort IMU Best Effort Navigation command Reliable Configuration Reliable Robot state Reliable Diagnostics Reliable These

Building a High-Performance Robot Communication System with DDS Modern robots may have dozens of processes distributed across CPUs, edge computers, and embedded devices. ROS 2 uses DDS (Data Distribution Service) as its underlying communication technology. Understanding DDS helps you design robot systems that remain responsive as message traffic grows. The Communication Model Instead of connecting every process directly: Camera ---> Perception LiDAR ---> Perception IMU ---> Localization | v Planning | v Control ROS 2 nodes communicate through DDS topics and discovery. A simplified model is: Publisher | v DDS DataWriter | v Topic | v DDS DataReader | v Subscriber Why DDS Is Useful for Robotics DDS provides mechanisms for: Discovery Reliability Durability Deadline management History Resource limits Data delivery policies These features are important because different robot data has different requirements. A camera stream may prioritize low latency. A configuration message may prioritize reliability. High-Performance Design Avoid treating every topic identically. For example: Data Typical Priority Camera frames Low latency LiDAR scans High throughput IMU Low latency Robot commands Reliability Configuration Reliability + durability Diagnostics Reliability Reduce Copying Large sensor messages can consume substantial CPU and memory bandwidth. Good practices include: Avoid unnecessary serialization/deserialization. Reuse buffers where possible. Keep image resolution appropriate for the workload. Compress only when bandwidth savings justify CPU cost. Separate high-rate sensor topics from low-rate metadata. Separate Data Paths A useful architecture is: +--> Vision Camera ----------+ | LiDAR -----------+--> Perception --> Planning --> Control | IMU -------------+ Diagnostics ---------------------> Monitoring Configuration ------------------> Lifecycle Manager Not all traffic needs the same QoS or processing path. Measuring Performance Do not optimize based on intuition alone.

The U.S. is shutting out more foreign-made drones and robots. China’s scale means the global competition may simply move elsewhere.

Welcome back to TechCrunch Mobility, your hub for the future of transportation and now, more than ever, the role AI is playing in it.

This is The Stepback, a weekly newsletter breaking down one essential story from the tech world. For more on falling robots and the US-China AI race, follow Robert Hart. The Stepback arrives in our subscribers' inboxes on Sunday at 8AM ET. Opt in for The Stepback here. How it started I've admitted my fondness for […]

The company is testing robots on tasks that can performed by technicians.

I just wrote about the billion-dollar rounds flooding into humanoid robotics. Here is the story from the other end of the scale, and I find it more encouraging. Hugging Face's open-source robot, a 25-centimeter bipedal machine with fifteen actuators and a sensor kit that includes a camera, speaker, LiDAR, NFC, Bluetooth, and WiFi, just passed a million dollars in sales. Fully open hardware, openly documented, quietly making real money. One of these robotics stories is funded like an industrial giant. The other is a small, open, shippable thing that people are actually buying. They are both true, and the small one is the one most builders can learn from. Open hardware turned out to be a business The reflexive assumption about open-source hardware is that you cannot make money on it, because anyone can copy the design. Hugging Face's robot is a live counterexample. The plans are open, the software stack is open through their LeRobot ecosystem, and it crossed a million in sales anyway. That is worth sitting with, because it means openness and revenue are not the opposites people assume. The reason it works is the same reason open-source software companies work. Most buyers do not want to source fifteen actuators, fabricate a chassis, and debug a sensor stack to save money on a robot that already exists and is affordable. They want the finished thing, they want it to work out of the box, and they are happy to pay the people who designed it. Openness is not the giveaway that kills the business. It is the trust and the ecosystem that make the business, because you can see exactly what you are buying, modify it, and build on a platform other people are also building on. Why this is the better story for builders The mega-funded humanoid companies are placing a bet only a handful of players can place: billions of dollars, years of runway, factories. That is a real path, and it is not your path or mine. The Hugging Face robot is the other path, and it is copyable. Small, open

Technical progress has encouraged a new batch of companies to jump in on the promise of profits from humanoid robots. And they're all Chinese automakers.

The company is testing robots that can swap cables, reset servers, and take on other tasks performed by technicians, fueling concerns among some workers that their jobs could be at risk.

The potential for AI to automate scientific research and manufacturing must be balanced with new risks, Anthropic says.