DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Manipulation and Grasping AI — Ch.09: General Purpose Manipulation [EN]

STATERA Library · Series 08 · Volume 17 · Chapter 09 · EN
Source: https://github.com/Statera-Library/08_Vol_17_Manipulation_and_Grasping_AI/blob/main/Chapter_09_General_Purpose_Manipulation/Chapter_09_General_Purpose_Manipulation_EN.md

Volume 17 Manipulation and Grasping AI

Chapter 09. General Purpose Manipulation

09.01. General Purpose Manipulation Vision and Requirements

General-purpose manipulation aims to create robotic systems that can interact competently with a broad range of objects, tools, environments, and tasks without requiring a separately engineered manipulation pipeline for every situation. Unlike conventional industrial automation, where geometry, object location, motion sequence, and operating conditions are tightly controlled, a general-purpose manipulator must operate under uncertainty and adapt its behavior as conditions change.

The central vision is a manipulation system that can receive a high-level objective, interpret the surrounding scene, identify relevant objects and their functional relationships, determine appropriate physical interactions, execute them safely, and evaluate whether the intended result has been achieved. Such a robot should move beyond executing predefined trajectories toward generating actions from perception, context, task knowledge, and continuously updated estimates of the physical world.

General-purpose capability does not imply that one robot must perform every conceivable manipulation task. Instead, it describes an architecture whose skills can be reused, composed, transferred, and extended across task families. Picking, placing, pushing, pulling, inserting, opening, closing, pouring, wiping, fastening, tool use, and object handover can therefore be treated as reusable interaction primitives rather than isolated applications requiring independent control systems.

A fundamental requirement is broad perceptual competence. The robot must estimate object identity, pose, geometry, scale, articulation, material characteristics, affordances, free space, obstacles, and relationships between objects. RGB cameras, depth sensors, force sensing, tactile sensing, proprioception, and other modalities provide complementary observations. Their integration produces a richer state representation than any individual sensing modality can provide.

Perception must also support manipulation rather than merely object recognition. Knowing that an object is a cup is insufficient if the robot cannot determine where it can be grasped, whether it contains liquid, how it is oriented, what obstacles surround it, or which region should remain unobstructed during use. Manipulation-oriented perception therefore emphasizes affordances, contact opportunities, functional parts, physical constraints, and action-conditioned scene understanding.

General-purpose manipulation additionally requires robust pose estimation and spatial reasoning under partial observability. Objects may be occluded, deformable, reflective, transparent, previously unseen, or moving during interaction. The robot must maintain beliefs about uncertain states and update them as new observations become available. Active perception becomes important because the manipulator can deliberately move its camera, arm, or object to obtain information needed for subsequent actions.

Another requirement is adaptable grasp generation. Fixed grasp libraries work effectively in structured production environments but become insufficient when objects vary significantly in shape, orientation, material, and accessibility. A general-purpose system should generate and rank grasp candidates according to geometry, task intent, collision risk, force requirements, reachability, and downstream actions. The best grasp for lifting an object may differ substantially from the best grasp for using it as a tool.

Manipulation must therefore be task-conditioned. The robot should reason not only about whether an object can be grasped, but also about why it is being grasped and what interaction follows. A screwdriver intended for fastening requires an orientation and grasp configuration that preserve access to its functional end. Similarly, a container intended for pouring must be grasped in a way that supports controlled rotation while maintaining stability and avoiding unintended contact.

Motion generation must connect these semantic intentions to physically executable trajectories. Planning must account for kinematic limits, singularities, joint limits, self-collision, environmental collision, payload, dynamic constraints, contact states, and end-effector orientation. In changing environments, planning cannot remain purely offline. Trajectories may need to be continuously revised using updated perception and feedback while preserving safety and task objectives.

Contact-rich interaction introduces requirements beyond free-space motion planning. In insertion, wiping, opening, assembly, or tool-use tasks, successful behavior depends on forces and constraints created through contact. Position control alone is often inadequate because geometric errors can generate excessive forces or prevent completion. Force control, impedance control, compliance, tactile feedback, and hybrid position-force strategies therefore become essential components of a general manipulation architecture.

A general-purpose manipulator also requires a representation of skills at multiple levels of abstraction. Low-level controllers regulate torque, velocity, position, and contact forces, while reusable motor skills implement behaviors such as grasping or insertion. Higher layers organize these skills into task sequences and reason about goals, objects, constraints, and failures. This hierarchy allows semantic instructions to be translated progressively into executable physical actions.

Learning expands this architecture by allowing behavior to improve from demonstrations, interaction data, simulation, teleoperation, reinforcement learning, and large-scale robot datasets. Rather than explicitly programming every variation, models can learn reusable relationships between observations, language, actions, and outcomes. Imitation learning can provide initial competence, while reinforcement learning and online adaptation can refine behaviors that require interaction with uncertain physical dynamics.

Foundation models and vision-language-action models introduce an additional route toward generalization. Large multimodal models can connect language instructions with visual observations and action representations, potentially allowing knowledge acquired across many tasks to support unfamiliar situations. However, semantic generalization alone does not guarantee reliable physical execution. Manipulation still requires accurate geometry, timing, contact reasoning, control stability, and mechanisms for detecting when model predictions are unsafe or physically infeasible.

World models can provide another important capability by representing how the environment is expected to evolve after an action. Instead of selecting actions only from the current observation, the robot can predict possible future states and evaluate alternative manipulation strategies before execution. Such predictive reasoning is particularly valuable when actions are irreversible, when objects interact dynamically, or when intermediate configurations strongly influence later stages of the task.

General-purpose operation also demands systematic failure detection and recovery. Real environments inevitably produce missed grasps, object slips, unexpected collisions, blocked trajectories, incorrect detections, and unsuccessful insertions. A robust system must distinguish between intended and actual outcomes, diagnose discrepancies, and choose an appropriate recovery strategy. Recovery may involve re-observation, re-grasping, replanning, changing the manipulation strategy, or requesting human assistance.

Safety must be integrated throughout the manipulation stack rather than treated as a final protective layer. Manipulators operating near humans must constrain velocity, force, workspace, and contact behavior according to context. Safety monitoring should remain active even when learned policies or high-level AI models generate commands. Independent limits, collision monitoring, emergency stopping, uncertainty estimation, and runtime validation provide safeguards against both physical failures and erroneous decisions.

Hardware versatility is another major requirement. General-purpose manipulation benefits from sufficient reach, payload, dexterity, repeatability, bandwidth, and compliant interaction capability. End effectors may include parallel grippers, multi-finger hands, suction devices, or interchangeable tools. No single mechanism is optimal for every task, so hardware design must balance mechanical simplicity and reliability against the dexterity required for diverse physical interactions.

The computational architecture must support tight coordination between perception, planning, learning, and control. Some processes operate at high frequency and require deterministic local execution, while semantic reasoning or large-model inference may tolerate longer latency. A practical system therefore separates fast physical control loops from slower deliberative processes while maintaining consistent state, synchronization, command validation, and feedback between the different computational layers.

Data diversity becomes a critical requirement as manipulation moves toward generality. Training data should span different objects, viewpoints, environments, robot configurations, task instructions, failure modes, and interaction dynamics. Simulation can increase coverage and reduce physical collection cost, but differences between simulated and real contact behavior remain significant. Domain randomization, system identification, real-world fine-tuning, and cross-robot datasets help reduce this simulation-to-reality gap.

Evaluation must consequently measure more than success on a small set of predefined demonstrations. General-purpose manipulation should be tested for robustness to unseen objects, altered object positions, environmental changes, instruction variations, disturbances, and combinations of previously learned skills. Metrics should include task success, completion time, intervention rate, collision frequency, grasp stability, recovery capability, safety violations, and performance degradation under distribution shift.

Scalability also depends on reducing task-specific engineering. If every new application requires extensive perception tuning, manually authored trajectories, custom reward functions, and repeated controller redesign, the system remains specialized despite using advanced AI. A stronger measure of generality is therefore the amount of additional engineering, data, and training required when the robot encounters a new task, object category, workspace, or embodiment.

Ultimately, general-purpose manipulation is best understood as an integrated intelligence problem rather than a single algorithmic capability. Perception identifies actionable structure, world models estimate physical consequences, planning selects feasible interactions, learning supplies adaptable skills, and feedback control realizes those interactions in the physical world. Reliability emerges from continuous coordination among these functions rather than from the sophistication of any individual model.

The long-term objective is a robotic manipulation platform capable of converting human-level intentions into safe, adaptive, and physically grounded behavior across diverse environments. Achieving this vision requires progress not only in AI models but also in sensing, actuation, control, data, system architecture, verification, and human-robot interaction. General-purpose manipulation therefore represents a convergence point where embodied intelligence must connect semantic understanding with dependable physical execution.

09.02. Mobile Manipulation Platform Loco Manip Integration [w/Code]

Mobile manipulation integrates locomotion and manipulation so that a robot can move through an environment, position its body relative to a target, and physically interact with objects using one coordinated system. Unlike a fixed manipulator whose workspace is determined by its mounting location, a mobile manipulation platform extends its effective workspace through base motion, allowing manipulation tasks to be performed across rooms, production areas, warehouses, laboratories, and other large environments.

A typical mobile manipulation platform combines a mobile base, one or more robotic arms, end effectors, perception sensors, onboard computing, power systems, and communication interfaces. The base may use differential drive, omnidirectional wheels, steering mechanisms, tracks, or legged locomotion. The manipulation subsystem may consist of a collaborative arm, industrial manipulator, lightweight arm, or dual-arm configuration selected according to payload, reach, dexterity, and interaction requirements.

The central integration challenge is that locomotion and manipulation cannot always be treated as independent functions. A conventional architecture may navigate to a predefined pose, stop the base, and then activate the arm. This sequential strategy is practical for structured tasks, but it restricts reachability and flexibility. Loco-manipulation instead treats base motion and arm motion as coordinated degrees of freedom that can contribute simultaneously to achieving the task objective.

The combined configuration space of a mobile manipulator is considerably larger than that of either subsystem alone. The planner must reason about base position, orientation, arm joint configuration, end-effector pose, obstacles, joint limits, stability, and task constraints. Multiple configurations may produce the same end-effector pose, creating redundancy that can be exploited to improve manipulability, avoid singularities, increase clearance, or maintain a safer distance from surrounding objects.

Base placement is therefore a fundamental component of mobile manipulation. Before reaching for an object, the robot must determine where the mobile platform should be positioned so that the target is reachable with sufficient manipulation margin. An apparently short navigation route may produce a poor arm configuration, while a slightly different base pose may greatly improve reachability. Navigation and manipulation planning should consequently share geometric and task-related information.

Perception provides the common spatial foundation required for this integration. Cameras, depth sensors, LiDAR, force sensors, tactile sensors, wheel odometry, inertial measurement units, and joint encoders contribute information about the robot and its surroundings. These observations must be transformed into consistent coordinate frames so that the system can estimate the relationship among the mobile base, manipulator, end effector, target object, obstacles, and global environment.

Localization accuracy directly affects manipulation performance because errors in mobile base pose propagate into the estimated position of the arm relative to the target. Navigation-level localization may be adequate for approaching a workstation but insufficient for precise grasping or insertion. Practical systems therefore combine global localization with local visual or geometric refinement near the manipulation target, allowing the robot to correct residual positioning errors before physical interaction.

A mobile manipulation system benefits from layered representations of the environment. A global map supports navigation through large spaces, while local occupancy maps or distance fields support collision avoidance near the robot. Object-level representations describe manipulation targets and semantic landmarks. Close to contact, detailed geometry, visual features, force measurements, and tactile observations become increasingly important because manipulation accuracy requirements are much tighter than ordinary navigation requirements.

Whole-body planning treats the mobile base and manipulator as a unified kinematic system. Instead of planning a base trajectory first and an arm trajectory afterward, the planner can search for coordinated motions that satisfy the end-effector objective while respecting collisions and mechanical constraints. This approach is especially valuable when reaching into shelves, operating along extended surfaces, handling large objects, or performing tasks whose required workspace exceeds the static reach of the arm.

Whole-body control extends this principle into real-time execution. Desired end-effector motion can be distributed among arm joints and base motion according to task priorities and constraints. The arm may perform fast, precise local adjustments while the base provides slower large-scale repositioning. Optimization-based controllers can regulate this coordination while considering joint limits, manipulability, obstacle clearance, velocity limits, and secondary objectives such as maintaining favorable sensor visibility.

Simultaneous base and arm motion introduces dynamic coupling that becomes increasingly important at higher speeds or with heavier payloads. Manipulator acceleration can disturb the mobile platform, while base acceleration changes the motion experienced by the arm and payload. Even when the base is nominally stable, structural compliance, wheel suspension, floor irregularities, backlash, and vibration can reduce end-effector accuracy. Dynamic models and feedback control must account for these effects when precision is required.

Stability becomes a critical constraint when the manipulator reaches far from the platform or carries a significant payload. The arm changes the system center of mass and can generate overturning moments during acceleration or contact. Platform dimensions, wheel arrangement, suspension, payload distribution, manipulator mounting position, and acceleration limits influence the available stability margin. Controllers should restrict motions that approach unsafe configurations even when they are kinematically reachable.

Contact-rich mobile manipulation introduces further complexity because interaction forces can propagate through the arm into the base. Tasks such as pushing a door, wiping a wall, operating a valve, drilling, or manipulating heavy equipment may require sustained force while the robot moves. Force control, impedance control, compliant behavior, traction estimation, and base stabilization must work together so that contact forces remain controlled without causing wheel slip, oscillation, or loss of localization.

Navigation around people and equipment must also consider the geometry of the manipulator. A mobile base may have a collision-free path while an extended arm collides with shelves, walls, machinery, or humans. The navigation system must therefore represent the complete robot footprint, including configuration-dependent swept volume. During transport, the arm can be moved into a compact safe posture, while manipulation-aware navigation can modify the posture when approaching the target workspace.

Task planning determines when locomotion, repositioning, manipulation, sensing, and recovery actions should occur. A command such as retrieving an object from another room may require navigation, target search, approach, base alignment, grasping, verification, retreat, transport, and placement. These behaviors should be represented as coordinated skills rather than isolated software functions so that execution can adapt when the environment or object state differs from the original plan.

Learning can improve loco-manipulation by discovering coordination strategies that are difficult to engineer manually. Demonstration learning can capture how humans or teleoperators combine body positioning with reaching, while reinforcement learning can optimize coordinated motion under complex constraints. Learned policies may also estimate promising base poses, grasp configurations, or recovery actions. However, learned behaviors should remain bounded by collision, stability, force, and safety constraints during deployment.

World models can further support integration by predicting how robot and environment states evolve under coordinated actions. The system may compare alternative strategies such as repositioning the base before grasping, moving while reaching, or manipulating an object from several successive poses. Predictive models are particularly useful for long-horizon tasks in which an early base placement decision can strongly affect later reachability, visibility, collision risk, and task completion.

Mobile manipulation requires careful management of computation and communication latency. Low-level motor control, force regulation, collision response, and emergency functions require fast deterministic execution, whereas semantic perception, global planning, or large-model reasoning can operate at slower rates. A hierarchical architecture allows these processes to coexist while ensuring that delayed high-level decisions cannot override immediate physical safety requirements or destabilize local controllers.

Energy management is also more important than in stationary manipulation because locomotion, perception, computing, and arm actuation share a limited onboard power source. Repeated acceleration of a heavy platform or manipulator can significantly reduce operating time. Planning can therefore consider energy alongside distance and execution time, selecting efficient base positions, reducing unnecessary repositioning, scheduling charging, and adapting performance according to the remaining battery state.

Safety requirements increase because the robot combines a moving platform with an articulated mechanism whose reachable volume changes continuously. Speed and separation monitoring, collision detection, force limits, protective stopping, human tracking, safe arm postures, and emergency stop functions must cover both subsystems. Safety supervision should evaluate the combined robot state rather than assuming that individually safe base and arm commands automatically produce safe whole-body behavior.

Calibration and synchronization are essential for maintaining integration quality over long-term operation. Transformations between the base, arm, cameras, LiDAR, end effector, and other sensors must remain accurate, while sensor timestamps and control states must be sufficiently synchronized. Mechanical impacts, payload changes, maintenance, sensor replacement, or mounting drift can degrade calibration, so automated validation and recalibration procedures are valuable for reliable deployment.

Failure recovery should exploit the mobility of the platform rather than repeatedly attempting the same arm motion. If a grasp is unreachable, the robot can reposition the base; if an object is occluded, it can move to another viewpoint; if an approach is blocked, it can select a different side of the workspace. This ability to modify the physical observation and manipulation geometry is one of the strongest advantages of mobile manipulation over fixed robotic systems.

Evaluation should measure both subsystem performance and integrated task performance. Navigation accuracy or arm precision alone cannot demonstrate effective loco-manipulation. Tests should examine base placement quality, manipulation success, whole-body collision avoidance, stability, localization during interaction, recovery behavior, task completion time, intervention frequency, energy consumption, and robustness to environmental variation. Integrated benchmarks should include tasks requiring genuine coordination rather than simple navigation followed by stationary manipulation.

Applications include warehouse picking, machine tending, laboratory automation, hospital logistics, retail handling, facility inspection, maintenance, household assistance, and flexible manufacturing. In these domains, mobility allows a smaller number of robots to service multiple workstations instead of dedicating one manipulator to each location. The economic value therefore comes not only from autonomous navigation or manipulation individually, but from the ability to deliver manipulation capability wherever physical work is required.

Ultimately, loco-manipulation transforms a robotic arm from a device operating inside a fixed workspace into a mobile physical agent capable of creating and modifying its own workspace through movement. Successful integration requires shared perception, manipulation-aware navigation, whole-body planning and control, stability management, contact reasoning, safety supervision, and adaptive recovery. When these capabilities operate as one coordinated system, mobile manipulation becomes a foundation for scalable general-purpose robotic autonomy.

09.03. Open World Object Grasping Novel Objects [w/Code]

Open-world object grasping addresses the ability of a robot to detect, reason about, and physically acquire objects that were not explicitly represented during system development or model training. Unlike closed-set manipulation, where object categories, geometries, and grasp templates are known in advance, open-world grasping assumes that unfamiliar objects will routinely appear. The system must therefore generalize from learned physical principles rather than depend on memorized object identities.

A novel object may differ from training examples in geometry, scale, material, texture, articulation, transparency, reflectivity, or functional purpose. The robot may encounter an unfamiliar container, tool, package, component, household item, or irregular natural object without possessing a predefined CAD model or grasp library. Successful grasping consequently requires estimating actionable properties directly from observations while explicitly representing uncertainty about what cannot yet be determined.

The perception pipeline begins by separating potentially manipulable objects from the surrounding environment. RGB cameras provide appearance and semantic cues, while depth cameras and LiDAR can recover geometric structure. Segmentation models identify object regions, and three-dimensional reconstruction converts observations into point clouds, surfaces, occupancy representations, or implicit geometry. Open-world systems must perform these operations even when conventional category classifiers cannot confidently assign a known label.

Object identity and graspability should therefore be partially decoupled. A robot does not necessarily need to know that an unfamiliar object belongs to a specific semantic category before lifting it. It may instead infer that a visible region provides sufficient clearance for a gripper, has approximately parallel contact surfaces, supports stable force closure, and is connected to the object body. Geometry-driven reasoning allows manipulation to proceed even when semantic recognition remains uncertain.

Semantic knowledge nevertheless improves grasp selection when it is available. Vision-language models and multimodal foundation models can relate visual observations to broad concepts, object functions, and likely usage patterns. An unfamiliar object may resemble a bottle, handle, tool, or container sufficiently to suggest useful interaction hypotheses. Open-world manipulation therefore benefits from combining geometric evidence with semantic priors rather than depending exclusively on either representation.

Affordance reasoning extends this process from recognizing what an object might be to estimating how it can be physically used. Different regions may afford holding, pulling, pressing, pouring, supporting, cutting, or insertion. The grasp should preserve the functional region required by the downstream task. An unknown tool should not automatically be grasped at the most geometrically convenient location if that contact blocks the portion likely to be needed during subsequent manipulation.

Grasp generation for novel objects commonly relies on local three-dimensional geometry rather than stored object-specific poses. Candidate contacts can be sampled from visible surfaces and evaluated according to gripper width, surface normals, antipodal relationships, collision clearance, force closure, and reachable approach directions. Data-driven grasp networks can learn these relationships from large collections of objects and then predict grasp quality directly from depth images, point clouds, or reconstructed scenes.

Partial observability creates a major difficulty because the robot typically sees only a fraction of an object before grasping it. The hidden side may contain an unexpected protrusion, cavity, support structure, or connection to another object. Shape completion models can estimate missing geometry, but predictions should be treated probabilistically. Conservative planning may avoid uncertain regions, while active perception can acquire additional viewpoints when uncertainty significantly affects grasp safety or feasibility.

Active perception is particularly important in open-world environments because the robot can deliberately reduce uncertainty before committing to contact. It may move its camera, reposition its mobile base, rotate its wrist, or manipulate surrounding objects to reveal hidden surfaces. The decision to gather another observation should depend on expected information gain and task cost. A difficult grasp may become straightforward after a small viewpoint change exposes the object\'s true geometry.

Material properties introduce another source of uncertainty. A visually similar object may be rigid, compliant, fragile, slippery, porous, heavy, or hollow. Visual appearance can provide prior estimates, but reliable interaction often requires physical feedback. Force sensing, tactile sensing, motor current, deformation measurements, and incipient slip detection allow the robot to update its assumptions after contact. Grasping thus becomes an information-gathering interaction rather than a single irreversible command.

Grasp force should adapt to this estimated physical state. Excessive force can damage fragile or deformable objects, while insufficient force causes heavy or slippery objects to fall. A general grasp controller can begin with conservative contact, estimate stiffness and friction from the response, and progressively adjust force while monitoring stability. Tactile arrays are especially valuable because they reveal contact distribution, local pressure, shear, and slip that may not be visible to external cameras.

Clutter further complicates novel-object grasping because target surfaces may be occluded or physically blocked by neighboring items. The robot must reason about collision-free approach corridors, gripper geometry, object-object contacts, and the likelihood that surrounding objects will move during extraction. In dense scenes, the optimal strategy may involve pushing, rearranging, or removing another object before attempting the final grasp rather than repeatedly selecting low-clearance grasp candidates.

Task context changes what constitutes a good grasp. A grasp optimized only for immediate lifting may leave the object in an unsuitable orientation for placement, insertion, handover, inspection, or tool use. Task-oriented grasping therefore evaluates downstream feasibility as part of grasp selection. The robot may intentionally choose a slightly less robust initial contact if it produces a configuration that significantly improves subsequent manipulation while remaining above an acceptable stability threshold.

Motion planning must jointly evaluate the selected grasp and the trajectory required to realize it. A high-quality grasp candidate is useless if the arm cannot reach the corresponding pre-grasp pose or if the approach collides with the environment. Candidate ranking should consequently include inverse kinematics, joint limits, singularity proximity, obstacle clearance, self-collision, and expected post-grasp motion. Mobile manipulators can additionally reposition their base to improve access to difficult novel objects.

Learning-based grasp models require diverse training distributions to generalize beyond familiar objects. Synthetic object collections, procedurally generated geometry, scanned objects, CAD repositories, simulation, and real grasp demonstrations can expose the model to broad shape variation. Domain randomization can vary scale, pose, texture, lighting, friction, mass, and sensor noise. The objective is not to reproduce every future object but to cover enough physical variation that useful grasp principles emerge.

Simulation is especially useful because millions of candidate grasps can be generated and evaluated without damaging hardware or manually labeling every interaction. Physics engines can estimate collision, contact, force closure, and lifting success across large object libraries. However, simulation cannot perfectly reproduce friction, compliance, tactile response, sensor artifacts, or complex contact dynamics. Real-world validation and fine-tuning remain necessary to prevent simulated grasp quality from becoming an unreliable proxy for physical success.

Self-supervised learning offers a complementary route to scalable improvement. A robot can attempt grasps, observe whether objects were lifted and retained, and automatically use the outcomes as training signals. Failed attempts are particularly informative because they reveal weaknesses in perception, candidate generation, force selection, or execution. Over time, autonomous interaction can produce a continually expanding dataset that reflects the actual sensors, gripper, environment, and failure modes of the deployed platform.

Uncertainty estimation is essential when models operate outside their training distribution. A grasp network may assign a high score to an unfamiliar geometry even when its prediction is poorly supported by experience. Ensembles, probabilistic models, confidence calibration, out-of-distribution detection, and geometric consistency checks can help identify such situations. The robot can respond by selecting a more conservative grasp, collecting additional observations, reducing speed, or requesting assistance instead of executing an overconfident action.

Failure detection must continue after the fingers close. The system should determine whether contact occurred at the expected locations, whether the object moved as predicted, whether sufficient grasp force exists, and whether slip begins during lifting. Vision, force, tactile feedback, joint sensing, and gripper state can be fused to verify grasp success. A binary command-completion signal is insufficient because the gripper may close successfully while holding nothing or while capturing an unstable object.

Recovery behavior converts individual grasp failures into robust task execution. If a candidate fails, the robot can place the object back, change the approach direction, adjust gripper width or force, select another contact region, obtain a new viewpoint, reposition its body, or manipulate clutter. The recovery strategy should use evidence from the previous attempt rather than simply repeating the same prediction. Each failure can therefore become an observation that improves the next decision.

Safety is particularly important because novel objects may contain properties that cannot be inferred from appearance. The robot should avoid aggressive interaction with uncertain sharp, fragile, hot, liquid-filled, electrically connected, or mechanically constrained objects. Semantic reasoning can identify possible hazards, while geometric and force limits provide independent protection. High uncertainty should reduce allowable interaction intensity rather than being interpreted as permission to explore without constraint.

Evaluation of open-world grasping must distinguish memorization from genuine generalization. Test objects should therefore be separated from training objects not only by instance but, where appropriate, by geometry, category, material, or functional structure. Metrics can include grasp success rate, retained-object rate, damage frequency, planning time, number of attempts, uncertainty calibration, recovery success, and downstream task completion. Performance should also be measured under clutter, occlusion, sensor noise, and distribution shift.

A strong benchmark should include both familiar and genuinely novel objects because generalization must not come at the expense of reliability on ordinary cases. Evaluation can progressively increase novelty from unseen instances of known categories to unseen categories and unusual geometries. This structure reveals where performance degrades and whether semantic, geometric, tactile, or active-perception mechanisms successfully compensate as the object distribution moves farther from training experience.

Open-world grasping ultimately changes the design objective from recognizing every possible object to acquiring sufficient physical understanding to interact safely with objects that cannot all be known in advance. Geometry, semantic priors, affordances, uncertainty, tactile feedback, active perception, planning, and learning must cooperate throughout the grasp cycle. Generalization emerges from this integration rather than from any single recognition or grasp-prediction model.

For general-purpose manipulation, this capability is foundational because real environments cannot be reduced to a permanent catalog of predefined objects. A useful robot must encounter novelty, determine what information is available, identify what remains uncertain, gather additional evidence when necessary, and execute a physically appropriate grasp with continuous verification. Open-world object grasping therefore provides a critical bridge between learned manipulation skills and dependable operation in unconstrained physical environments.

09.04. Language Conditioned Manipulation SayCan [w/Code]

Language-conditioned manipulation enables a robot to convert natural-language instructions into physically executable actions while considering the current environment and its own capabilities. Instead of requiring users to specify coordinates, trajectories, or low-level commands, the robot interprets goals such as bringing an object, placing an item, opening a container, or cleaning an area and determines which manipulation skills should be executed to accomplish the requested objective.

This problem connects two fundamentally different forms of intelligence. Language models provide broad semantic knowledge about objects, actions, relationships, and common task sequences, whereas robotic controllers operate under geometric, dynamic, and safety constraints. A language model may understand that a cup must be grasped before it can be delivered, but this semantic knowledge alone cannot determine whether the cup is visible, reachable, stable, or physically graspable by the particular robot.

SayCan introduced an influential framework for connecting these domains by combining language-model reasoning with learned robotic affordances. The central idea can be summarized as selecting actions according to both what the language model considers useful for accomplishing the instruction and what the robot estimates it can successfully execute. This prevents task planning from relying exclusively on linguistic plausibility while also avoiding purely reactive skill selection without semantic task understanding.

The language component evaluates how appropriate each available skill is given the instruction and the current sequence of completed actions. For a request such as bringing a drink to a user, actions involving finding, approaching, grasping, transporting, and placing the drink receive stronger semantic support than unrelated actions. The language model effectively provides a task-level prior over possible next actions based on linguistic and commonsense knowledge.

The affordance component evaluates whether each candidate skill is feasible in the current physical state. A grasp skill may receive a low value if the target is unreachable, while navigation toward the object may receive a high value. These estimates can be learned through robotic interaction and represented using value functions or success probabilities. The robot therefore distinguishes between an action that is conceptually appropriate and one that is currently executable.

Action selection combines these two signals. Conceptually, the preferred skill is one with both high semantic relevance and high physical feasibility. A linguistically reasonable action with poor affordance should not be executed, while a physically possible action that does not advance the requested task should also receive low priority. This grounding mechanism provides a practical bridge between high-level language reasoning and embodied robotic competence.

The skill library is consequently a critical part of the architecture. Skills may include navigation, object search, grasping, placing, opening, closing, pushing, pulling, manipulation of drawers, handover, or other reusable behaviors. Each skill exposes an interface that can be referenced by the high-level planner while internally handling perception, motion generation, feedback control, and termination conditions. Language reasoning therefore operates over meaningful robotic capabilities rather than raw actuator commands.

Skill parameterization extends this approach beyond a small set of fixed actions. A generic grasp skill may accept an object identity or spatial target, while a placement skill may accept a destination. Language grounding must map phrases such as "the red cup," "the table near the door," or "the empty shelf" into entities represented by the robot\'s perception system. The quality of manipulation therefore depends on reliable correspondence between linguistic references and physical scene representations.

Perception provides the state grounding required for this correspondence. Cameras, depth sensors, object detectors, segmentation models, spatial maps, and pose estimators identify objects and their relationships in the environment. The system must associate language concepts with perceived entities and update these associations as the robot moves. If several similar cups exist, contextual descriptions and spatial relationships may be necessary to resolve which object the instruction actually refers to.

Long-horizon instructions make language-conditioned manipulation particularly valuable. A command such as "bring me something to clean the table and put it back afterward" may require identifying a suitable cleaning object, navigating to it, grasping it, returning to the table, performing the cleaning action, locating its storage position, and replacing it. Language models can provide semantic decomposition, while robot skills and affordance estimates determine whether each step can actually be performed.

Hierarchical planning helps manage this complexity by separating goals, skills, and low-level controls. The highest layer interprets the human objective and decomposes it into subgoals. A skill layer chooses executable behaviors for each subgoal, while motion and control layers generate trajectories, contact forces, and actuator commands. Feedback flows upward so that high-level reasoning can respond when the physical execution produces an unexpected result.

Execution should therefore operate as a closed loop rather than generate a complete action sequence once and blindly follow it. After each skill, the robot observes the resulting state and determines whether the expected condition has been achieved. A successful grasp changes the available next actions, while a failed grasp should trigger re-observation or replanning. Repeated language reasoning over updated physical states improves robustness in dynamic and uncertain environments.

Failure recovery is especially important for language-conditioned systems because semantic plans often assume conditions that do not hold physically. An instructed object may be missing, blocked, too heavy, already in use, or outside the robot\'s reachable workspace. Instead of repeatedly executing an impossible skill, the system should identify the violated assumption, select an alternative action, ask the user for clarification, or report that the requested objective cannot currently be completed.

Language can itself support recovery by enabling interaction with the user. When an instruction is ambiguous, the robot can ask whether "the cup" refers to one of several candidates. When a requested object cannot be found, it may ask whether an alternative is acceptable. This transforms language from a one-way command interface into a bidirectional coordination mechanism in which the robot communicates uncertainty, constraints, progress, and possible alternatives.

Large language models can extend SayCan-like architectures by providing stronger task decomposition, commonsense reasoning, contextual interpretation, and flexible generation of candidate plans. They can infer that a container should normally be opened before retrieving its contents or that fragile objects require careful handling. However, linguistic competence should remain separated from authority over physical execution because plausible generated actions may still be unsafe, impossible, or unsupported by available robot skills.

Vision-language models further improve grounding by jointly reasoning about images and text. Instead of depending only on symbolic object labels, multimodal models can interpret visual scenes, identify objects described in flexible language, and relate spatial or functional descriptions to observed regions. This capability is useful in open-world environments where users may refer to unfamiliar objects by appearance, function, relative location, or contextual meaning rather than by predefined category names.

Vision-language-action models move integration one step further by learning mappings among observations, instructions, and robot actions. Such models can potentially reduce the separation between language planning and manipulation policy by learning reusable action representations from large robot datasets. Nevertheless, explicit skill constraints, geometric validation, collision checking, force limits, and safety supervision remain valuable because learned action generation does not guarantee physical correctness.

World models can complement language reasoning by predicting the consequences of candidate actions before execution. A language model may propose opening a drawer and retrieving an object, while a world model evaluates how the scene is expected to change and whether subsequent actions remain feasible. Combining semantic prediction with physical prediction allows the system to compare alternative plans not only according to linguistic coherence but also according to anticipated embodied outcomes.

Uncertainty should be represented at several levels. The robot may be uncertain about the meaning of an instruction, the identity of an object, the success probability of a skill, or the future outcome of an interaction. Treating these uncertainties explicitly enables conservative behavior. Low confidence can trigger additional perception, clarification, slower execution, alternative planning, or human assistance rather than forcing the system to commit to a poorly supported interpretation.

Safety requires an independent layer that constrains language-generated behavior. Natural-language instructions should never directly bypass collision avoidance, workspace limits, force constraints, human detection, or emergency stopping. A user may unintentionally request an unsafe action, and a language model may produce an inappropriate intermediate step. Runtime monitors and validated skill interfaces ensure that semantic reasoning operates only within the physical and operational limits authorized for the robot.

Training language-conditioned manipulation requires data connecting instructions, observations, skills, and outcomes. Sources may include teleoperation demonstrations, human-labeled trajectories, robot interaction logs, simulation, video-language datasets, and automatically generated task variations. Language augmentation can express equivalent objectives in many forms, while environmental randomization exposes the system to different object arrangements. Diversity is essential because real users rarely issue commands using standardized wording.

Evaluation should measure more than whether the robot follows an instruction in a familiar demonstration environment. Tests should vary linguistic phrasing, object instances, scene layouts, task order, ambiguity, missing objects, and unexpected failures. Important metrics include instruction completion rate, skill success, planning efficiency, intervention frequency, clarification quality, recovery success, safety violations, and generalization to combinations of tasks that were not explicitly represented during training.

A useful benchmark should also separate semantic errors from physical execution errors. The robot may fail because it misunderstood the instruction, selected the wrong object, chose an inappropriate skill, generated an unreachable motion, or lost the object during manipulation. Identifying these failure classes reveals whether improvements are needed in language reasoning, grounding, affordance estimation, planning, perception, or control rather than reducing all failures to a single task-success number.

The broader significance of SayCan is not limited to one specific implementation. Its important architectural principle is that semantic intelligence and physical competence provide complementary evidence for action selection. Language models answer what action makes sense in the context of the goal, while embodied affordances estimate what the robot can successfully do in the current state. Their intersection defines a much more useful action space than either component can provide independently.

For general-purpose manipulation, language conditioning offers a scalable interface between human intent and reusable robotic skills. Users can describe objectives at a natural level of abstraction while the robot handles task decomposition, grounding, feasibility checking, execution, verification, and recovery. As foundation models, world models, perception, and manipulation policies improve, this architecture can evolve from selecting predefined skills toward composing and adapting increasingly general physical behaviors.

The long-term objective is not merely a robot that obeys verbal commands, but an embodied agent that understands what a person intends, recognizes what the environment permits, knows what its own body can accomplish, and continuously reconciles these factors during execution. Language-conditioned manipulation therefore provides a key pathway from semantic reasoning to dependable physical intelligence, with SayCan representing an important foundation for capability-aware, grounded robotic decision making.

09.05. VLA Based General Manipulation RT 2 Pi0 [w/Code]

Vision-Language-Action models, commonly abbreviated as VLA models, aim to connect visual perception, natural-language understanding, and physical robot control within a unified learning framework. Instead of constructing separate modules for object recognition, task interpretation, skill selection, and manipulation policy, a VLA model learns relationships among observations, instructions, and actions so that semantic knowledge can directly influence embodied behavior.

The basic input to a VLA system typically includes one or more camera observations together with a language instruction describing the desired task. Additional information may include robot proprioception, joint states, gripper state, depth, force, or previous actions. The model processes these signals to generate an action representation that can ultimately be converted into joint commands, end-effector motion, gripper operations, or other robot control variables.

This architecture changes the traditional manipulation pipeline. Conventional systems often transform perception into explicit object poses, use symbolic or geometric planning to select a task sequence, and then execute separately engineered controllers. VLA systems attempt to learn more of this mapping from data. The model can associate visual patterns and linguistic concepts with useful physical behavior without requiring every intermediate representation to be manually specified.

RT-2 demonstrated an important direction for VLA research by transferring knowledge from large-scale vision-language pretraining into robotic control. Its central contribution was to represent robot actions in a form compatible with the token-based output of a vision-language model. By treating actions as token sequences, knowledge learned from web-scale visual and linguistic data could participate in reasoning about robotic tasks while robot demonstration data grounded those predictions in physical execution.

This transfer is valuable because robot datasets are much smaller than general image-text datasets. A robot may never have physically interacted with every object or concept it will encounter, but a pretrained vision-language model may already possess semantic knowledge about many of them. RT-2 showed how such knowledge can support generalization to objects, instructions, and reasoning situations that were not explicitly represented in the available robotic demonstrations.

The resulting capability is sometimes described as web knowledge transferred to robotics. If a model understands visual concepts such as tools, containers, symbols, colors, or relationships from broad pretraining, it can potentially use that knowledge when interpreting a manipulation instruction. The important step is grounding semantic understanding in the robot\'s action space so that recognizing a concept can influence what the robot physically does rather than remaining only a descriptive capability.

Action tokenization provides one method for creating this bridge. Continuous quantities such as end-effector translation, rotation, and gripper commands can be discretized or encoded into tokens that a transformer predicts autoregressively. These tokens are subsequently decoded into executable commands. This allows language tokens and action tokens to share a common modeling framework, although the physical controller must still respect timing, calibration, kinematic constraints, and safety limits.

RT-2 also highlighted emergent forms of robotic reasoning. A model may interpret an instruction involving a semantic property rather than a directly memorized object-action pair and choose an appropriate target based on knowledge acquired outside robot demonstrations. This does not imply unrestricted reasoning capability, but it demonstrates that broad semantic pretraining can improve manipulation generalization when combined with embodied data and an action representation suitable for robot control.

The π0, or pi-zero, direction represents another important stage in general-purpose robot policy development. Rather than relying only on discrete autoregressive action tokens, π0 combines pretrained vision-language representations with a generative action mechanism designed for continuous robot control. This approach addresses the fact that manipulation requires smooth, high-dimensional, temporally coordinated trajectories rather than isolated symbolic decisions.

A key concept associated with π0 is the use of flow matching for action generation. Instead of predicting one discrete action token after another, the policy can generate continuous action chunks through a learned generative process. Action chunking allows the model to predict a short horizon of coordinated robot behavior, reducing the burden of independently deciding every control step while preserving the continuous structure required for dexterous manipulation.

Continuous action generation is especially relevant for tasks involving coordinated arm motion, bimanual manipulation, deformable objects, or precise contact. Small errors accumulated across independently generated actions can destabilize a manipulation sequence. Predicting structured action trajectories allows temporal relationships among movements to be represented more directly. Low-level feedback control can then track these commands while compensating for disturbances and modeling errors.

General-purpose VLA training depends heavily on heterogeneous robot data. Demonstrations may come from different robots, camera configurations, action spaces, environments, and task families. A scalable model must normalize or transform these heterogeneous observations and actions into representations that permit shared learning. Cross-embodiment training is important because collecting enough demonstrations on every individual robot platform is expensive and limits the breadth of experience available to the model.

Robot datasets provide physical grounding that internet-scale visual and language data cannot supply. Images and text can describe how objects are used, but they do not directly encode actuator dynamics, contact forces, grasp stability, or the consequences of control errors. Robot trajectories supply this missing embodied information by linking observations and instructions to actions and outcomes. Effective VLA models therefore combine semantic breadth with physically grounded interaction experience.

Data diversity is as important as dataset size. Training should cover different objects, backgrounds, viewpoints, lighting conditions, instructions, robot configurations, task stages, successes, and failures. If a model repeatedly sees only one workspace or one phrasing of a command, it may learn superficial correlations rather than reusable manipulation principles. Diverse training distributions improve the probability that learned representations remain useful when deployment conditions change.

Language provides a powerful mechanism for sharing structure across tasks. Instructions such as pick, place, move, open, close, insert, fold, wipe, or hand over describe reusable action concepts that appear in many environments. A VLA model can associate these linguistic concepts with visual situations and physical trajectories. More complex instructions can then be interpreted as combinations or variations of behaviors encountered during training, supporting compositional generalization.

Visual grounding remains essential because language alone does not determine where or how an action should occur. The model must identify relevant objects, spatial relationships, free space, functional regions, and changes caused by previous actions. Transformer-based visual representations can integrate global scene context with task instructions, but manipulation often requires fine spatial precision. High-resolution features, multiple camera views, depth information, and proprioception can therefore significantly improve execution quality.

Closed-loop execution is critical for deploying VLA policies. A model may generate an action chunk, execute part of it, observe the resulting state, and then generate a revised continuation. This receding-horizon structure allows the policy to respond to object displacement, grasp errors, human motion, or unexpected contact. Pure open-loop execution becomes increasingly unreliable as task duration grows because small physical errors accumulate and change the state from what the model originally predicted.

VLA models can also be integrated with explicit world models or planning mechanisms. A policy primarily maps current observations to actions, whereas a world model predicts how candidate actions may change future states. Combining these functions can support longer-horizon reasoning by allowing alternative strategies to be evaluated before physical execution. Language models can contribute task decomposition while learned action policies provide the detailed embodied behavior required for each stage.

Generalization should be considered along several dimensions rather than as a single property. A system may generalize to unseen object instances while failing on new task combinations, or handle new language expressions while remaining sensitive to workspace geometry. Evaluation should therefore separate object generalization, scene generalization, instruction generalization, task compositionality, embodiment transfer, and robustness to physical disturbances.

Failure recovery provides another demanding test of general manipulation. Training demonstrations often emphasize successful trajectories, yet deployment inevitably includes missed grasps, slips, occlusions, blocked paths, and partially completed tasks. VLA policies should recognize these states and generate corrective actions rather than continue an obsolete trajectory. Including failure examples, corrective demonstrations, and autonomous interaction data can improve the model\'s ability to recover.

Uncertainty estimation remains important because large models can produce confident actions in unfamiliar situations. A VLA policy may encounter an object, mechanism, or geometry far outside its training distribution. Confidence estimation, ensemble methods, out-of-distribution detection, action consistency checks, and external geometric validation can help determine when model output should be trusted. High uncertainty can trigger slower motion, additional perception, replanning, or human assistance.

Safety should remain architecturally independent from learned policy generation. Even a highly capable VLA model should not directly bypass joint limits, collision detection, velocity restrictions, force limits, workspace boundaries, or emergency stopping. Learned policies can propose behavior, while deterministic safety controllers and runtime monitors determine whether those commands are permissible. This separation is particularly important when models inherit broad but imperfect knowledge from heterogeneous datasets.

Real-time performance introduces additional engineering constraints. Large multimodal transformers can require substantial computation, while robot control demands predictable latency. Practical systems may separate slower semantic inference from faster action generation and low-level servo control. Model compression, efficient attention, action chunking, hardware acceleration, and asynchronous processing can reduce latency while preserving the responsiveness required for physical interaction.

Cross-embodiment generalization is one of the most important long-term opportunities. Different robots may have different arm lengths, joint arrangements, grippers, sensor placements, and control frequencies, yet many tasks share semantic and geometric structure. Common representations can allow knowledge learned on one embodiment to benefit another. Robot-specific adapters or action transformations can then map shared policy representations into commands appropriate for each physical platform.

Fine-tuning and adaptation remain necessary even for broadly pretrained models. A factory, laboratory, hospital, or household may contain specialized objects and operational requirements absent from general training data. Small amounts of site-specific demonstration data can adapt a general VLA model to these environments. Parameter-efficient adaptation, policy heads, embodiment adapters, or targeted post-training can preserve general knowledge while improving performance on local tasks.

Evaluation must ultimately focus on physical task completion rather than only model prediction accuracy. Relevant metrics include success rate, completion time, intervention frequency, collision rate, grasp stability, recovery success, instruction compliance, and performance on unseen objects or tasks. Long-horizon benchmarks are especially important because they reveal whether perception, semantic understanding, action generation, and feedback remain coherent over extended sequences.

RT-2 and π0 illustrate two complementary milestones in the evolution of VLA-based manipulation. RT-2 demonstrated that large-scale vision-language knowledge can be connected to robot actions through tokenized representations, enabling semantic knowledge to influence physical behavior. π0 emphasizes continuous generative action policies capable of producing rich robot trajectories from multimodal context, pushing VLA systems toward more dexterous and general manipulation.

The broader trajectory is toward robot foundation models that combine semantic knowledge, diverse embodied experience, multimodal perception, and scalable action generation within reusable pretrained systems. Such models will not eliminate geometric planning, feedback control, world modeling, or safety engineering. Instead, they can provide a shared intelligence layer that reduces the amount of task-specific programming required when robots encounter new objects, instructions, environments, and embodiments.

Ultimately, VLA-based general manipulation seeks to transform robot programming from manually defining every behavior into training systems that can interpret intent and generate appropriate physical actions from broad prior experience. RT-2 and π0 represent important steps along this path, linking foundation-model reasoning with embodied control through different action representations. Their significance lies in demonstrating how semantic intelligence can progressively become reusable physical intelligence across increasingly diverse manipulation tasks.

09.06. Tool Use and Instrument Manipulation [w/Code]

Tool use extends robotic manipulation beyond directly grasping and moving objects by allowing a robot to employ an intermediate object to modify the environment or achieve a task. A tool changes the robot\'s effective capabilities: a screwdriver transfers torque, a spatula extends contact beneath an object, a brush distributes force across a surface, and a probe allows interaction with locations that the robot\'s gripper cannot directly reach.

Instrument manipulation requires reasoning about function rather than object identity alone. Recognizing an object as a hammer, wrench, brush, pipette, or screwdriver is useful, but successful operation depends on understanding which part should be grasped, which part interacts with the environment, and how forces and motions should be transmitted through the object. The robot must therefore represent functional regions, interaction geometry, and expected physical effects.

A useful representation separates the grasp region from the action region of a tool. The grasp region should provide stable attachment to the robot while preserving access to the functional end. The action region is the portion that contacts or influences the target. Between them, the tool body establishes a geometric and mechanical transformation that changes the relationship between end-effector motion and the contact generated at the task location.

Affordance reasoning is central to this process because tools are defined strongly by what they enable. A long rigid object may afford reaching or pushing even if it was not manufactured as a dedicated tool. A flat object may support scraping, while a compliant surface may support wiping or polishing. General-purpose manipulation should therefore infer tool affordances from geometry, material, context, and task requirements rather than depend exclusively on predefined semantic categories.

Language and semantic knowledge can provide useful priors for selecting appropriate tools. Given an instruction to tighten a screw, clean a surface, retrieve an inaccessible object, or transfer liquid, a language model can propose objects whose functions are compatible with the goal. However, semantic compatibility is insufficient by itself. The robot must verify that the proposed tool exists, can be grasped, fits the target geometry, and can withstand the required physical interaction.

Perception must identify both the tool and the task-relevant features of the target. For a screwdriver task, the robot may need to estimate the handle, shaft, tip, screw head, insertion axis, and available approach direction. For wiping, it may need to identify the cleaning surface, contaminated region, tool contact area, and obstacle boundaries. Tool manipulation therefore demands finer functional perception than ordinary pick-and-place behavior.

Pose estimation becomes especially important because errors at the robot wrist can be amplified at the tool tip. A long instrument introduces a lever arm, so small orientation errors may produce significant positional displacement at the contact point. Calibration between the robot, gripper, tool, cameras, and environment must therefore be accurate. Online visual or tactile correction can compensate for residual errors when precise alignment is required.

Tool acquisition itself is a task-oriented grasping problem. The robot should select a grasp that not only secures the object but also enables its intended use. Grasping a hammer near its head may be mechanically stable yet provide poor leverage, while holding a screwdriver too close to its tip may obstruct insertion. Candidate grasps should therefore be evaluated according to stability, reachability, manipulability, functional clearance, and downstream task performance.

After grasping, the system must estimate the effective tool center point and its transformation relative to the robot. Industrial systems often define a calibrated tool center point explicitly, but general-purpose robots may encounter previously unseen instruments. The robot can estimate this relationship from vision, known geometry, tactile contact, probing, or exploratory motion. Accurate tool-frame estimation allows motion planning to reason about the functional tip rather than only the gripper.

Motion planning with a tool changes the collision geometry of the complete robotic system. A path that is safe for an empty gripper may cause the attached instrument to strike the environment. The planner must include the tool body, functional tip, swept volume, target contact geometry, and post-contact motion. Long or irregular tools may substantially constrain feasible approach directions and require repositioning of the arm or mobile base.

Many tool-use tasks are contact-rich and cannot be solved reliably with position control alone. Tightening, scraping, wiping, polishing, cutting, probing, and insertion depend on forces generated between the tool and target. Hybrid position-force control, impedance control, admittance control, and compliant motion allow the robot to maintain desired contact while adapting to geometric errors and surface variations. Force limits also protect the tool, workpiece, and robot.

The mechanical characteristics of the tool alter the dynamics of manipulation. Tool mass, center of gravity, inertia, stiffness, compliance, and friction influence control behavior. A heavy drill changes the arm\'s payload distribution, while a flexible brush transmits forces differently from a rigid probe. The robot should estimate or identify these properties sufficiently well to adjust acceleration, force commands, trajectory speed, and stability margins.

Tactile and force sensing provide information that vision cannot easily observe during contact. A screwdriver tip may appear aligned visually while failing to seat correctly in the screw head. Force and torque patterns can reveal insertion, slippage, jamming, or successful engagement. Similarly, tactile sensing can detect whether a tool is shifting inside the gripper. Multimodal feedback is therefore essential for verifying that the intended mechanical relationship has actually been established.

Instrument manipulation often requires constrained motion. Turning a valve, rotating a handle, operating a lever, or inserting a probe restricts the tool to a particular path determined by the environment. Rather than imposing a free-space trajectory, the controller should estimate and follow these constraints. Constraint-aware manipulation can combine geometric models with measured forces to adapt when the exact mechanism geometry is unknown or differs from prior assumptions.

Tool use also introduces a transformation of action semantics. A robot that normally controls its gripper position must reason about the effect produced at the remote tool contact point. Moving the wrist downward may generate scraping, pressing, cutting, or stirring depending on the instrument and contact state. Effective planning therefore requires predicting not merely robot motion but the physical consequence of robot-tool-environment interaction.

World models can support this prediction by estimating how candidate tool actions will change the environment. The robot can evaluate whether pushing with a rod will move an object within reach, whether rotating a screwdriver will advance a screw, or whether a wiping trajectory will cover the required surface. Predictive models become especially valuable when the task contains irreversible actions or when several tool-use strategies are physically possible.

Learning from demonstration provides a practical way to acquire complex tool-use behavior. A human teleoperator can demonstrate how to grasp an instrument, establish contact, regulate force, and execute the required trajectory. Imitation learning can capture correlations that are difficult to specify analytically, while subsequent reinforcement learning or optimization can improve efficiency and robustness. Demonstrations should include variations in tool pose, target location, and contact conditions.

Large-scale robot datasets can support transfer across related tools and tasks. Experience with one brush may help another wiping tool, while manipulation of different handles can reveal common rotational constraints. Representations based on geometry, affordances, contact modes, and functional effects are more transferable than representations based only on object identity. This supports generalization to tools that differ visually but serve similar mechanical roles.

Vision-language-action models can further connect semantic instructions with tool selection and execution. A model may receive an instruction, observe available objects, identify a plausible instrument, and generate manipulation behavior conditioned on both visual context and language. However, tool use magnifies the consequences of geometric or force errors, so learned policies should be combined with explicit collision checking, contact monitoring, and safety-constrained low-level control.

Novel-tool use is an important measure of general-purpose intelligence. A robot should ideally infer that an unfamiliar rigid object can extend reach, that a flat edge can scrape, or that a hook-shaped object can pull something closer. This capability requires analogical reasoning based on geometry and physical effects. Instead of asking only whether the robot has seen the exact tool before, the system asks whether the object\'s properties satisfy the functional requirements of the task.

Exploratory interaction can help when these properties are uncertain. The robot may gently lift a tool to estimate mass, touch it against a surface to infer stiffness, or perform a small motion to observe how it transmits force. These experiments convert unknown physical properties into measurable evidence. Active physical exploration therefore complements visual and semantic reasoning, particularly for unfamiliar instruments or objects with ambiguous material characteristics.

Bimanual manipulation expands the range of tool-use tasks. One hand may stabilize a workpiece while the other operates a tool, or both hands may manipulate a large instrument together. Such behavior requires coordination of contact forces, relative poses, task constraints, and collision avoidance across both arms. The workpiece, tool, and robot can form a closed kinematic chain whose stability must be maintained throughout execution.

Mobile manipulation adds another layer by allowing the robot to reposition its body while using a tool. A mobile manipulator may wipe a large wall, inspect equipment with a probe, or operate controls distributed across a facility. Base motion expands the effective tool workspace but introduces localization, stability, and coordination challenges. Whole-body planning can distribute movement among the mobile base, arm, wrist, and tool according to task requirements.

Failure detection should distinguish among grasp failure, tool slippage, missed contact, incorrect alignment, insufficient force, excessive force, target motion, and unsuccessful task effect. Each failure requires a different response. The robot may regrasp the instrument, recalibrate the tool frame, modify the approach, adjust force, repeat a trajectory, select another tool, or request human assistance. Recovery should be driven by observed evidence rather than blind repetition.

Safety becomes particularly important because tools can amplify force, extend the robot\'s reachable region, introduce sharp or hot surfaces, or create hazards not present during bare-hand manipulation. Safety monitoring must account for the complete robot-tool geometry and the potential energy transmitted through the instrument. Speed limits, force thresholds, restricted zones, human detection, collision monitoring, and emergency stopping should remain active throughout tool acquisition and use.

Evaluation should measure more than whether the robot successfully grasps the instrument. Relevant metrics include tool-selection accuracy, task-oriented grasp quality, tool-frame estimation error, contact establishment success, force regulation, trajectory accuracy, task completion, recovery performance, and damage frequency. Testing with unseen tools and altered target geometry is particularly important for distinguishing general functional understanding from memorized demonstrations.

Tool-use benchmarks should span different physical principles, including leverage, extension, rotation, scraping, wiping, insertion, probing, pouring, and fastening. Variation in geometry, material, scale, orientation, and target configuration reveals whether a system has learned reusable interaction concepts. Long-horizon evaluations can additionally test whether the robot can locate a tool, acquire it, use it, verify the result, and return it appropriately.

Ultimately, tool and instrument manipulation transforms the robot from a system limited by its native end effector into an agent capable of extending and reshaping its physical capabilities through external objects. Reliable tool use requires semantic understanding, affordance reasoning, task-oriented grasping, accurate geometry, contact-aware control, physical feedback, prediction, learning, and safety to operate as an integrated system.

For general-purpose manipulation, this capability is fundamental because human environments are designed around tools. A robot that can recognize functional opportunities, select or improvise an appropriate instrument, acquire it correctly, control the resulting robot-tool system, and verify the intended physical effect can address a far wider range of tasks. Tool use therefore represents a critical transition from object handling toward adaptable physical problem solving.

09.07. Drawer Door Button Articulated Object Manipulation [w/Code]

Articulated object manipulation concerns physical interaction with objects whose components move according to constrained mechanical relationships. Drawers translate along rails, doors rotate around hinges, buttons move through limited strokes, knobs rotate about fixed axes, and handles may actuate latches before another joint can move. Unlike free-object manipulation, success requires discovering and respecting the mechanism that governs allowable motion.

The central challenge is that visual appearance alone does not completely specify articulation. Two visually similar cabinet doors may have different hinge locations, opening directions, latch mechanisms, or motion limits. A robot must therefore infer not only object geometry but also latent kinematic structure. Manipulation becomes a process of estimating how components are connected, which degrees of freedom are available, and how applied actions change the mechanism state.

A useful representation models an articulated object as links connected by joints. Each movable component can be represented as a rigid link, while revolute, prismatic, or more complex joints describe relative motion. Joint axes, origins, limits, and current states define the configuration of the mechanism. This representation allows robot planning to reason about a drawer position or door angle rather than treating every observed geometry as an unrelated scene.

Perception begins by identifying actionable components such as handles, knobs, buttons, panels, drawer fronts, and door edges. RGB images provide semantic and appearance information, while depth sensing supplies three-dimensional geometry. Segmentation separates movable parts from surrounding structures, and pose estimation determines their spatial relationships. Functional perception must additionally identify which regions provide useful contacts for initiating the intended mechanism motion.

Handle detection is particularly important because handles often provide mechanical interfaces designed for human interaction. The robot must estimate graspable geometry, orientation, clearance, and the relationship between the handle and the movable link. Some mechanisms lack explicit handles, requiring the robot to pull an edge, push a panel, or press a surface. General manipulation therefore requires affordance detection beyond predefined handle categories.

Articulation models can be obtained from prior knowledge, learned from datasets, inferred visually, or estimated through physical interaction. A robot may recognize that an observed structure resembles a drawer and initialize a prismatic-joint hypothesis. Alternatively, it can test several motion hypotheses through gentle probing. Combining semantic priors with physical evidence provides greater robustness than assuming that visual classification alone determines the correct mechanism.

Interactive perception becomes valuable when joint parameters are uncertain. After establishing contact, the robot can apply a small controlled displacement or force and observe the resulting motion. If the object translates approximately along one axis, evidence supports a prismatic model. If the contact point follows an arc, a revolute mechanism is more likely. Physical interaction thus converts hidden mechanical constraints into observable motion.

System identification can refine the joint axis, pivot location, motion limits, friction, and required actuation force during manipulation. Initial estimates may be inaccurate because of partial observations or sensor noise. As the mechanism moves, the robot receives additional geometric and force information that can update its internal model. Planning and identification can therefore operate jointly rather than requiring a perfect mechanism model before interaction begins.

Drawer manipulation illustrates constrained translational interaction. The robot first approaches and grasps the handle or another suitable contact region, then applies force approximately along the drawer\'s permitted translation axis. Maintaining alignment is important because lateral forces can increase friction, cause binding, or destabilize the grasp. The controller should adapt as rail resistance, drawer loading, and extension distance change.

Opening a drawer also changes the surrounding collision geometry. The drawer front and contents move outward into the robot workspace, while previously hidden objects become visible. The robot must predict this swept volume and avoid collisions with its arm, body, furniture, or nearby people. After opening, perception should update the scene rather than continuing to rely on the geometry observed when the drawer was closed.

Door manipulation introduces rotational constraints. Once a handle is acquired and any latch is released, the robot must move the contact point along an arc determined by the hinge axis. A Cartesian straight-line trajectory would conflict with this constraint and may generate large contact forces. Constraint-aware planning instead follows the estimated kinematic manifold while allowing compliance to absorb modeling errors.

The hinge axis may not be directly visible, particularly for cabinets or appliances with concealed mechanisms. It can be estimated from object geometry, semantic knowledge, or observed motion during a small opening attempt. As the door rotates, successive observations provide information about the pivot. Online estimation can continuously refine the trajectory so that the robot does not depend on an exact hinge model established before contact.

Handles and latches often create sequential constraints. A lever handle may need to rotate downward before the door can move, while a refrigerator door may require sufficient pulling force to overcome a seal. Some drawers include push-to-open mechanisms, and safety cabinets may require multiple actions. The robot must represent these state-dependent transitions because applying the correct force to the wrong mechanism state can still produce failure.

Button manipulation appears simpler but requires high spatial precision and reliable contact-state reasoning. The robot must identify the active surface, approach approximately along an appropriate direction, establish contact, and produce sufficient displacement or force without damaging the device. Small buttons may require fingertip-scale accuracy, while compliant or touch-sensitive interfaces can require different interaction strategies from mechanical switches.

Successful button pressing also requires verification of the resulting state change. Mechanical displacement alone does not guarantee that the command was registered. The robot may observe illumination, display changes, sounds, machine motion, or other environmental responses. This illustrates a broader principle of articulated manipulation: success should be defined by the intended functional state transition rather than by completion of a commanded robot trajectory.

Force and torque sensing are especially useful because mechanical constraints become observable through interaction forces. Unexpected lateral forces can indicate poor drawer alignment, increasing torque can indicate a door approaching its limit, and a characteristic force drop may indicate latch release or button activation. These signals allow the controller to distinguish normal resistance from obstruction, jamming, incorrect contact, or mechanical failure.

Impedance and admittance control provide practical methods for interacting with uncertain mechanisms. Instead of forcing the end effector to follow a perfectly specified path, the robot can regulate its response to contact forces and allow the mechanism to guide part of the motion. Compliance reduces sensitivity to calibration errors and joint-model inaccuracies while preventing excessive forces from building when the observed mechanism differs from the assumed model.

Hybrid position-force control can assign different objectives along different directions. During drawer opening, the robot may regulate pulling motion along the rail direction while limiting lateral forces. During door opening, motion can follow the tangent of the estimated arc while maintaining stable handle contact. Such decomposition is useful because articulated objects naturally constrain some degrees of freedom while leaving others available for controlled motion.

Grasp maintenance becomes challenging as the mechanism moves. Changes in wrist orientation, arm configuration, friction, or handle geometry can cause slipping or approach joint limits. The controller should monitor grasp quality and manipulability throughout the trajectory rather than only at initial contact. Regrasping or repositioning may be necessary for mechanisms with large ranges of motion or unfavorable geometry.

Mobile manipulators can use base motion to extend the reachable articulation range. When opening a large door or long drawer, the arm alone may approach its kinematic limits. Coordinated base and arm movement allows the robot to maintain a favorable configuration while preserving contact. Whole-body planning can distribute motion among the mobile base, manipulator, wrist, and gripper while respecting the object\'s mechanical constraint.

Bimanual manipulation can support mechanisms requiring simultaneous stabilization and actuation. One arm may hold a cabinet steady while the other opens a stiff drawer, or one hand may operate a latch while the other moves a door. Two-arm interaction can also improve control of large panels. However, it introduces coupled constraints that require coordination of relative poses, contact forces, and collision avoidance.

Learning from demonstration can capture manipulation strategies that are difficult to specify analytically. Human demonstrations reveal useful approach directions, grasp locations, force profiles, timing, and recovery behaviors. By observing many mechanisms, a robot can learn representations that relate visual geometry and interaction history to likely articulation parameters. Such models can provide initial hypotheses that are subsequently refined through physical feedback.

Simulation can generate diverse articulated environments for training and testing. Cabinets, drawers, doors, switches, appliances, and industrial controls can be procedurally varied in dimensions, joint positions, friction, damping, limits, and handle geometry. Domain randomization exposes policies to mechanical variation, but accurate contact behavior remains difficult to simulate. Real-world interaction data is therefore necessary for reliable transfer.

World models can predict how articulated states evolve under candidate actions. A robot may compare whether pulling, pushing, rotating, or pressing is likely to produce the desired state transition. Predictive reasoning becomes valuable when the mechanism is unfamiliar or when multiple actions must occur in sequence. The world model can also estimate whether opening a component will create collisions or expose new task-relevant regions.

Vision-language-action models can connect natural-language goals such as "open the second drawer" or "press the green button" with visual scene interpretation and physical execution. Semantic knowledge helps identify likely components and expected interactions, while embodied policy data provides action grounding. Explicit constraint estimation and force monitoring remain important because linguistic plausibility cannot determine the exact mechanics of an unfamiliar device.

Failure modes include missed grasps, slipping handles, incorrect articulation hypotheses, blocked motion, excessive friction, hidden latches, joint-limit encounters, and unsuccessful state transitions. Recovery should diagnose the observed evidence before selecting another action. The robot may reposition its grasp, change the force direction, probe the mechanism, search for a latch, move its base, update the articulation model, or request assistance.

Safety supervision is essential because doors and drawers create moving collision surfaces and potential pinch points. The robot must consider not only its own links but also the swept geometry of the manipulated object. Force limits, velocity restrictions, human detection, pinch-zone monitoring, collision prediction, and emergency stopping should remain active throughout interaction. Unknown resistance should not automatically be overcome by increasing actuator force.

Evaluation should measure articulation understanding as well as task completion. Useful metrics include joint-type classification, axis estimation error, handle localization, contact success, trajectory consistency with the mechanism, interaction force, opening or closing range, state-transition success, recovery rate, and damage frequency. Testing across unseen mechanisms is necessary to distinguish genuine mechanical generalization from memorization of familiar furniture or devices.

A strong benchmark should include prismatic, revolute, push-button, rotary, latched, compliant, and compound mechanisms with variations in geometry and physical properties. Tasks should cover opening, closing, pressing, turning, partially positioning, and restoring mechanisms to desired states. Long-horizon scenarios can require opening a drawer, retrieving an object, closing the drawer, and verifying that the environment has returned to a safe configuration.

Ultimately, articulated object manipulation requires the robot to understand objects as mechanisms rather than static geometry. Perception proposes functional parts and kinematic hypotheses, interaction reveals hidden constraints, force feedback refines physical understanding, and compliant control follows the resulting motion structure. Planning, learning, prediction, and recovery must continuously update this representation as the mechanism changes state.

For general-purpose manipulation, drawers, doors, buttons, switches, handles, and related mechanisms are fundamental because human environments contain countless articulated interfaces. A robot capable of discovering how these structures move, establishing appropriate contact, respecting their constraints, verifying functional outcomes, and recovering from incorrect assumptions can interact with environments designed for people rather than specially engineered for robots.

09.08. General Manipulation in Unstructured Environments [w/Code]

General manipulation in unstructured environments requires robots to operate without assuming that objects, surfaces, obstacles, or task conditions have been precisely arranged in advance. Unlike structured industrial cells, real environments contain clutter, partial visibility, uncertain geometry, changing illumination, movable obstacles, and unexpected interactions. The robot must continuously interpret the scene and adapt its manipulation strategy as conditions evolve.

An unstructured environment should not be understood simply as a disorganized workspace. It is an environment in which the robot cannot rely on a complete and accurate model before execution begins. Homes, warehouses, construction sites, hospitals, laboratories, agricultural fields, and maintenance facilities all contain varying degrees of uncertainty. Successful manipulation therefore depends on perception, prediction, action, and feedback operating as a continuous closed loop.

Perception provides the initial representation of this uncertain world. RGB cameras can identify semantic categories, textures, and visual relationships, while depth cameras and LiDAR provide three-dimensional geometry. Tactile, force, proprioceptive, and acoustic sensing can reveal information unavailable from vision. General manipulation benefits from combining these modalities because any individual sensor may become unreliable under occlusion, poor lighting, reflective surfaces, or physical contact.

Scene understanding must distinguish manipulable objects from supporting surfaces, obstacles, containers, tools, articulated mechanisms, humans, and other robots. Instance segmentation and object detection provide semantic organization, while pose estimation and reconstruction provide geometric information for manipulation. However, the system should also represent uncertainty because estimated boundaries, poses, and object identities may be incomplete or incorrect.

Open-world perception becomes essential when the robot encounters objects that were not explicitly represented during training. A general-purpose robot cannot depend on a fixed list of known object classes. Foundation vision models and vision-language models can provide semantic priors for unfamiliar objects, while geometric reasoning can estimate graspable regions and physical relationships. Unknown identity does not necessarily prevent useful manipulation if relevant affordances can still be inferred.

Affordance reasoning allows the robot to focus on what an object permits rather than only what the object is called. A rigid edge may afford pulling, a flat region may support pushing, a handle may afford grasping, and an open container may receive another object. In unstructured environments, this functional interpretation is particularly valuable because objects may differ substantially from examples observed during training while still supporting familiar interaction patterns.

Clutter introduces significant manipulation difficulty because target objects may be partially occluded or physically constrained by neighboring objects. A direct grasp may be impossible even when the target is visible. The robot may need to push another object aside, change its viewpoint, rotate the target, remove an obstruction, or perform a sequence of intermediate actions. Manipulation planning therefore becomes coupled with active scene reorganization.

Active perception addresses uncertainty by deliberately selecting actions that improve future observations. A robot can move its camera, reposition its mobile base, rotate an object, open a drawer, or remove an occluder to reveal hidden geometry. The objective is not immediately to complete the task but to reduce uncertainty sufficiently for safer and more reliable manipulation. Perception thus becomes an action-dependent process rather than a passive sensing stage.

Three-dimensional scene representations support reasoning about reachability, free space, collision risk, and contact geometry. Point clouds, voxel maps, signed distance fields, meshes, and neural scene representations can describe different aspects of the environment. No single representation is universally optimal. Practical systems may maintain several complementary representations, using coarse maps for global planning and higher-resolution geometry near manipulation targets.

Object pose estimation remains useful but cannot always be treated as a precise deterministic value. Occlusion, deformable objects, reflective materials, and novel geometry may produce ambiguous estimates. Probabilistic pose distributions, confidence measures, or multiple hypotheses can represent this ambiguity. Motion planning can then prefer actions that remain safe across plausible states rather than optimizing exclusively for one uncertain estimate.

Grasp planning in unstructured scenes must consider more than local object geometry. A geometrically valid grasp may be inaccessible because of nearby clutter, may cause collision during approach, or may leave insufficient room for the subsequent task. Task-oriented grasp planning therefore evaluates reachability, collision clearance, stability, manipulability, and downstream actions together rather than selecting the highest-scoring isolated grasp.

Manipulation often changes the scene in ways that invalidate the original world model. Moving one object can expose another, shift surrounding clutter, alter support relationships, or create new collision risks. The robot should update its scene representation after meaningful actions and contacts. Long sequences cannot safely depend on a static snapshot because the environment evolves partly as a consequence of the robot\'s own behavior.

Closed-loop control provides robustness against these changes. Instead of planning a complete trajectory and executing it blindly, the robot repeatedly observes the environment, estimates state, executes a limited action, and replans when necessary. Visual servoing can correct geometric errors, while tactile and force feedback can correct contact errors. Receding-horizon execution limits the consequences of inaccurate predictions and allows rapid adaptation.

Contact-rich manipulation becomes particularly challenging when environmental geometry is uncertain. Inserting an object, opening an unfamiliar door, sliding an item across a surface, or retrieving something from clutter can generate contacts that were not predicted by the planner. Compliance allows the robot to tolerate small errors while force sensing reveals whether contact is expected, useful, obstructive, or potentially dangerous.

Impedance and admittance control provide mechanisms for regulating physical interaction without requiring perfect geometric knowledge. The robot can maintain compliant behavior along uncertain directions while controlling task-critical motion more strongly. Such strategies are useful when manipulating against surfaces, following constraints, or interacting with mechanisms whose exact parameters are unknown. Compliance converts some modeling errors into manageable physical deviations.

Manipulation under uncertainty also requires reasoning about object physical properties. Mass, friction, stiffness, fragility, center of gravity, and surface compliance influence whether a planned action will succeed. These properties may not be visible directly. The robot can use prior knowledge from object appearance and category, but physical interaction can refine estimates through lifting, pushing, probing, or controlled contact.

Exploratory manipulation turns action into a sensing mechanism. A small push can reveal whether an object is fixed or movable, lifting can estimate weight, and contact deformation can indicate compliance. This information can update the robot\'s internal model before committing to a larger action. Exploration is especially important for novel objects where semantic knowledge provides only weak information about actual physical behavior.

Dynamic environments introduce another level of uncertainty because humans, robots, vehicles, and objects may move during manipulation. The system must distinguish static geometry from dynamic agents and predict short-term motion when planning. A collision-free path at planning time may become unsafe seconds later. Continuous monitoring and trajectory adaptation are therefore required when manipulation occurs in shared or changing spaces.

Human-aware manipulation should account for both physical safety and behavioral predictability. The robot should avoid unexpectedly moving tools or objects through occupied regions and should maintain conservative speed and force limits near people. Human pose estimation and motion prediction can define dynamic safety zones. Clear approach trajectories and understandable interaction patterns also make robot behavior easier for nearby people to anticipate.

Mobile manipulation is valuable in unstructured environments because fixed-base reachability is inherently limited. A mobile manipulator can change its viewpoint, approach an object from another direction, reposition around obstacles, or transport objects between distant locations. However, navigation and manipulation can no longer be treated as independent problems. Base placement affects arm reachability, perception quality, collision clearance, and task success.

Whole-body planning coordinates the mobile base, torso, manipulator, wrist, and gripper to exploit available redundancy. Rather than positioning the base first and planning the arm afterward, integrated planning can select configurations that maximize manipulability while preserving visibility and safety. This becomes especially useful in confined spaces where small changes in base position can determine whether a manipulation task is feasible.

Long-horizon tasks require hierarchical reasoning because individual grasps and motions must contribute to a broader goal. Cleaning a workspace may involve locating objects, moving obstacles, opening containers, sorting items, using tools, and verifying completion. Task planning selects meaningful subgoals, while motion and manipulation policies execute them. Failures at lower levels should trigger local recovery or task-level replanning rather than complete mission termination.

Vision-language-action models provide a promising interface for general manipulation because language can express goals without specifying every geometric detail. A command such as "put the cleaning supplies back in the cabinet" can be decomposed using semantic knowledge while visual observations ground the relevant objects and locations. Learned action policies can provide reusable skills, but explicit geometric and safety mechanisms remain necessary for reliable physical execution.

World models can further support decision making by predicting how candidate actions may alter future observations and physical states. The robot can evaluate whether moving one object will expose the target, whether opening a container will create sufficient access, or whether a push could destabilize nearby items. Predictive simulation allows actions to be evaluated not only for immediate progress but also for their downstream consequences.

Learning from diverse robot datasets is important because unstructured environments contain far more variation than any single deployment site. Training data should include different objects, viewpoints, backgrounds, layouts, embodiments, interaction outcomes, and failure cases. Cross-embodiment datasets can expose models to broader physical experience, while site-specific adaptation can refine behavior for local equipment, geometry, and operational constraints.

Simulation can expand this diversity through procedural generation and domain randomization. Object layouts, textures, lighting, camera parameters, friction, mass, clutter density, and obstacle configurations can be varied automatically. Simulation is especially useful for rare or dangerous failures, but contact dynamics and perception artifacts may differ from reality. Sim-to-real transfer therefore benefits from real-world fine-tuning and continuous deployment data.

Failure detection is a fundamental capability rather than an exceptional condition. The robot must recognize missed grasps, object slips, unexpected collisions, blocked motion, perception errors, unreachable targets, and unsuccessful state transitions. Failure classification helps select appropriate recovery behavior. Repeating the same command without understanding the cause can amplify errors and potentially damage objects or the robot.

Recovery may involve reobserving the scene, changing viewpoint, selecting another grasp, moving an obstacle, adjusting force, repositioning the base, invoking another manipulation skill, or requesting human assistance. Effective recovery requires maintaining task context so that the robot understands what has already been completed. A robust system should degrade gracefully from autonomous execution toward safer or more supervised behavior as uncertainty increases.

Uncertainty should influence both planning and control. When confidence is high, the robot can execute efficiently, but uncertain states may require slower motion, additional sensing, larger collision margins, or conservative contact forces. Risk-aware planning explicitly trades efficiency against the probability and consequence of failure. This principle is critical because not all errors have equal cost in real environments.

Safety must remain independent from the assumptions of learned policies. Collision checking, joint and velocity limits, force thresholds, human proximity monitoring, workspace restrictions, and emergency stopping should supervise execution continuously. The robot should not interpret unexpected resistance simply as a command to apply more force. Safe behavior requires detecting when observations differ substantially from predicted conditions and reducing or terminating motion.

Evaluation of general manipulation should therefore emphasize robustness across distributions rather than performance on a single carefully arranged benchmark. Metrics can include task success, completion time, intervention rate, recovery success, collision frequency, object damage, perception uncertainty, grasp robustness, and performance degradation under clutter or environmental change. Testing should deliberately include novel objects, altered layouts, occlusion, sensor noise, and disturbances.

A mature benchmark should combine object manipulation, articulated mechanisms, tool use, clutter removal, navigation, and long-horizon task execution within continuously changing scenes. The robot should not know every object or mechanism in advance. Success should depend on discovering relevant structure, adapting behavior, and verifying outcomes. Such evaluation better reflects the demands of robots operating outside engineered automation cells.

Ultimately, general manipulation in unstructured environments is not defined by a single universal grasping algorithm or foundation model. It emerges from the integration of open-world perception, uncertainty representation, affordance reasoning, active sensing, task and motion planning, compliant control, multimodal feedback, learning, prediction, recovery, and safety. Each component compensates for limitations in the others as the environment changes.

The long-term objective is a robot that can enter an unfamiliar environment, interpret what is relevant, determine what can be manipulated, acquire missing physical information through interaction, select appropriate actions, execute them safely, and recover when assumptions fail. Achieving this capability transforms manipulation from predefined automation into adaptive physical intelligence capable of operating reliably in the complex and continuously changing human world.

09.09. Manipulation Foundation Model Architecture

A manipulation foundation model is a large pretrained model designed to provide reusable intelligence across many robotic manipulation tasks, environments, objects, and embodiments. Instead of training an independent policy for every task, the architecture learns shared representations from heterogeneous robot experience and multimodal data. These representations can later be adapted to new manipulation problems with substantially less task-specific data.

The central architectural objective is to connect semantic understanding with physically grounded action generation. A manipulation system must understand what objects are present, what the instruction means, which interactions are physically possible, and how robot motion should evolve over time. Foundation models attempt to place these capabilities within a common representation space so that knowledge acquired from one task can improve performance on related tasks.

The input layer is inherently multimodal. RGB images provide appearance and semantic information, depth or point clouds describe three-dimensional geometry, proprioception represents robot configuration, and force or tactile sensors reveal contact state. Natural-language instructions specify task intent, while previous actions and temporal observations provide context about how the environment has changed during execution.

Each modality can initially be processed by a specialized encoder. Vision transformers can transform images into visual tokens, point-based or voxel encoders can represent geometry, language transformers encode instructions, and state encoders convert joint positions, velocities, gripper states, or force measurements into embeddings. These modality-specific representations are then projected into a shared latent space where cross-modal relationships can be learned.

Tokenization provides a common interface between heterogeneous information sources. Image patches, object regions, language words, robot states, and actions can all be represented as structured token sequences. Continuous signals may be discretized into tokens or encoded directly as continuous embeddings. The representation choice influences computational efficiency, temporal resolution, action precision, and the type of generative policy that can be used.

A multimodal transformer can serve as the central backbone of the architecture. Self-attention captures relationships within each information stream, while cross-attention connects language, vision, state, and action context. The model can therefore associate phrases such as "pick up the red cup" with relevant visual regions, robot reachability, and appropriate manipulation behavior rather than processing each modality independently.

Pretraining is fundamental because the model must acquire broad representations before being specialized for individual tasks. Vision-language datasets provide semantic knowledge about objects and relationships, while robot demonstrations provide physical grounding. Video can contribute temporal understanding, and simulation can expand interaction diversity. Combining these sources allows the model to learn from data at scales unavailable through physical robot collection alone.

Robot trajectory data supplies the connection between perception and action. A trajectory may contain synchronized observations, language labels, robot states, actions, and outcomes across time. Training on large collections of such trajectories allows the model to learn recurring manipulation structures such as reaching, grasping, lifting, placing, pushing, opening, inserting, wiping, and tool use without defining every behavior as a separately engineered controller.

Cross-embodiment training expands the available experience by combining data from different robot platforms. Robots may have different kinematics, grippers, cameras, control rates, and action spaces, yet many manipulation tasks share semantic and geometric structure. A foundation architecture therefore separates reusable task knowledge from embodiment-specific execution sufficiently to permit transfer across heterogeneous robotic systems.

Embodiment information can be represented explicitly through robot tokens, kinematic descriptors, calibration parameters, or learned embodiment embeddings. These representations tell the model which physical system is executing the task. Robot-specific adapters or policy heads can translate shared latent representations into commands appropriate for a particular arm, gripper, humanoid, or mobile manipulator while preserving common higher-level knowledge.

Action representation is one of the most important architectural decisions. Discrete action tokenization allows actions to be generated autoregressively in a language-model-like framework. Continuous policies can instead predict end-effector displacement, joint commands, or action trajectories directly. Modern architectures may use diffusion, flow matching, or related generative mechanisms to model complex multimodal distributions over continuous robot behavior.

Action chunking allows the policy to generate a short sequence of coordinated future actions rather than predicting only the next control command. This reduces inference frequency and captures temporal structure within manipulation skills. A chunk may describe several coordinated end-effector poses and gripper commands, after which the robot observes the updated environment and generates another chunk through closed-loop execution.

Hierarchical architectures can separate semantic reasoning from high-frequency physical control. A slower foundation model may interpret instructions, identify relevant objects, decompose tasks, and select skills, while a faster policy generates trajectories. Low-level servo controllers then execute those trajectories at much higher frequencies. Such separation allows large models to contribute reasoning without requiring them to operate directly at motor-control rates.

Object-centric representations can improve manipulation reasoning by organizing observations around entities and relationships. Instead of treating the entire scene only as undifferentiated image tokens, the model can represent objects, parts, handles, containers, surfaces, and tools explicitly. Relational representations can then encode spatial, functional, and contact relationships that remain meaningful even when scene appearance changes.

Three-dimensional spatial representations are particularly important because manipulation ultimately occurs in physical space. Depth, point clouds, voxel features, neural fields, or reconstructed geometry can supplement two-dimensional visual tokens. Spatial grounding allows the model to distinguish semantic understanding from actionable geometry, such as where a handle is located, whether a grasp is reachable, and whether a trajectory will collide with the environment.

Temporal modeling provides another essential capability. A single observation cannot reveal whether an object has moved, whether a grasp succeeded, or whether a drawer is currently opening. Memory mechanisms, recurrent state, temporal transformers, or cached tokens can preserve interaction history. The model can then reason about progress, detect state transitions, and distinguish current conditions from earlier observations.

A world-model component can extend the architecture from reactive policy generation toward predictive manipulation. Given the current latent state and candidate actions, the model predicts future observations, object states, contacts, or rewards. Planning can compare alternative actions before execution. This becomes valuable for long-horizon tasks where an immediately successful action may create unfavorable conditions for subsequent manipulation.

The world model and action policy need not be implemented as completely separate networks. Shared latent representations can support both prediction and control, allowing learned physical structure to influence action generation. Predictive auxiliary objectives can improve representation learning even when explicit model-based planning is not used. The architecture can therefore combine reactive efficiency with varying degrees of prospective reasoning.

Language provides a flexible task-conditioning interface. Natural-language commands can describe goals, constraints, object attributes, spatial relationships, and multi-step activities. Large language representations also provide semantic priors that help interpret unfamiliar objects or tasks. However, language must be grounded in current sensory observations because semantic plausibility alone cannot determine the exact geometry or physical state of the environment.

Skill representations provide an intermediate level between language and raw control. Reusable latent skills can correspond to behaviors such as grasping, inserting, opening, wiping, or handing over without requiring rigid symbolic definitions. A high-level planner can select or compose these skills, while conditioned low-level policies adapt them to the current geometry. Such hierarchical reuse can improve data efficiency and long-horizon consistency.

Training objectives are usually multi-task rather than limited to action prediction. The model may learn action generation, masked observation reconstruction, contrastive vision-language alignment, future-state prediction, success estimation, value prediction, or inverse dynamics simultaneously. Auxiliary objectives encourage latent representations to retain semantic, spatial, temporal, and physical information useful across many downstream manipulation tasks.

Data normalization is critical when combining heterogeneous datasets. Different robots may encode positions in joint space or Cartesian coordinates, use different camera frames, record at different frequencies, and apply different gripper conventions. A shared training pipeline must transform these signals into consistent representations or provide enough metadata for the model to interpret their differences. Poor normalization can cause scale to increase without producing useful transfer.

Dataset composition also determines what the foundation model learns. Large quantities of repetitive demonstrations may provide less generalization than a smaller but diverse collection containing different objects, environments, viewpoints, tasks, failures, and embodiments. Training mixtures should balance common reusable skills with difficult edge cases. Failure trajectories and corrective behavior are especially valuable for teaching recovery rather than only nominal execution.

Simulation and synthetic data can broaden the training distribution at comparatively low cost. Procedural environments can vary object geometry, clutter, lighting, camera placement, physical parameters, and task configurations. Synthetic language instructions and automatically generated labels can further expand supervision. Nevertheless, real robot data remains necessary because contact dynamics, sensing artifacts, actuator behavior, and unexpected environmental effects are difficult to reproduce perfectly.

Post-training adapts the pretrained architecture to specific robots or deployment domains. Fine-tuning can specialize the model using demonstrations from a factory, laboratory, household, or mobile platform. Parameter-efficient adapters can reduce computational cost and preserve general representations. Preference optimization, reinforcement learning, or autonomous data collection can further improve task success after initial supervised training.

Closed-loop deployment is essential because even powerful foundation models cannot predict physical execution perfectly. The robot observes the scene, generates an action or action chunk, executes part of it, measures the result, and conditions the next inference on the updated state. This continuous perception-action-feedback cycle prevents small errors from accumulating indefinitely and provides opportunities for correction after unexpected events.

Uncertainty estimation should accompany action generation. The model may encounter unfamiliar objects, ambiguous instructions, severe occlusion, or states outside the training distribution. Confidence measures, ensemble disagreement, generative variance, or consistency tests can indicate when predictions are unreliable. Uncertainty can trigger additional observation, slower execution, alternative planning, or escalation to human supervision.

Safety architecture should remain separate from the learned foundation model. Generated actions must pass through deterministic checks for collision, joint limits, velocity, acceleration, workspace boundaries, force thresholds, and human proximity. A safety supervisor can reject or modify unsafe commands regardless of model confidence. This separation prevents learned semantic reasoning from becoming the sole authority over physically hazardous behavior.

Inference efficiency is a major deployment requirement because foundation models can contain billions of parameters while robotic interaction demands low and predictable latency. Model distillation, quantization, efficient attention, caching, action chunking, and hardware acceleration can reduce computational load. Architectures may also divide computation between cloud, on-premise servers, and edge processors according to latency, privacy, connectivity, and safety requirements.

Memory can extend manipulation beyond individual episodes. Short-term memory maintains the current task state and recent interactions, while longer-term memory can store environmental layouts, object properties, previous failures, or successful strategies. Retrieval mechanisms can provide relevant prior experience when similar situations occur. Persistent memory must nevertheless distinguish reusable knowledge from outdated observations in changing physical environments.

Failure detection and recovery should be treated as native architectural capabilities. The model can predict expected outcomes and compare them with actual observations after execution. Deviations may indicate grasp failure, slippage, collision, obstruction, or incorrect task interpretation. Recovery can then involve re-observation, replanning, regrasping, selecting another skill, or requesting assistance rather than repeatedly executing an invalid action.

Evaluation should measure transfer and generalization rather than only performance on training-like tasks. Important dimensions include unseen objects, new language instructions, altered environments, new task compositions, different embodiments, and physical disturbances. Long-horizon evaluation reveals whether semantic reasoning, spatial grounding, action generation, memory, feedback, and recovery remain coherent across extended interaction sequences.

A mature manipulation foundation model architecture therefore resembles an integrated intelligence stack rather than a single monolithic policy. Multimodal encoders provide perception, shared representations connect semantics and geometry, temporal memory preserves context, predictive models estimate consequences, planners organize goals, generative policies produce actions, feedback closes the physical loop, and safety systems supervise execution.

The long-term objective is to create a reusable physical intelligence substrate that can be adapted across robots and applications rather than rebuilding manipulation intelligence for every deployment. Such a foundation model would combine broad semantic knowledge with embodied experience, spatial reasoning, prediction, action generation, adaptation, and recovery. Its value lies not in replacing every robotics module, but in providing a common learned architecture through which previously isolated capabilities can increasingly cooperate.

09.10. General Manipulation Service Robot Case

A general manipulation service robot combines mobile autonomy, perception, manipulation, language understanding, and safe human interaction to perform diverse tasks in environments originally designed for people. Unlike a dedicated industrial robot that repeatedly executes one predefined operation, a service robot must move between locations, identify changing task conditions, manipulate unfamiliar objects, and adapt its behavior to requests expressed at a functional level.

A representative deployment can be considered in a mixed indoor facility containing offices, laboratories, storage rooms, kitchens, corridors, and shared service areas. The robot may be requested to deliver objects, organize supplies, open cabinets, retrieve items, operate buttons, clear tables, or transport tools. The environment is only partially structured, so successful operation depends on integrating navigation and manipulation rather than optimizing either capability independently.

The platform can consist of an autonomous mobile base, one or two robotic arms, dexterous or parallel grippers, and a multimodal sensor suite. RGB and depth cameras provide semantic and geometric perception, LiDAR supports localization and navigation, joint encoders provide proprioception, and wrist force-torque or tactile sensors measure physical interaction. Microphones can additionally provide speech input for natural human-robot communication.

A service request may begin with a command such as "bring the blue bottle from the kitchen to the meeting room." The robot must convert this language instruction into a task representation containing the target object, source location, destination, and required actions. The request does not specify navigation paths, grasp poses, collision-free trajectories, or recovery behavior. These details must be generated from the robot\'s current perception and learned knowledge.

Task decomposition transforms the request into executable subgoals. The robot may first localize itself, navigate to the kitchen, search for the bottle, approach a reachable position, grasp it, verify grasp success, navigate to the meeting room, identify the delivery location, place the bottle, and confirm completion. This hierarchical organization separates high-level intent from the detailed physical actions needed to achieve it.

Navigation and manipulation interact throughout the mission. The mobile base must stop where the target is visible and where the arm has sufficient reach and manipulability. A location that is optimal for navigation may be poor for grasping, while an ideal arm configuration may require a different base pose. Integrated mobile manipulation therefore evaluates visibility, reachability, collision clearance, and base accessibility together.

Open-world perception is necessary because service environments contain many objects that may never have appeared in the robot\'s training data. Vision-language models can provide semantic knowledge for recognizing or describing unfamiliar items, while geometric perception identifies shape, pose, support surfaces, and graspable regions. The robot does not always need an exact object category if it can infer the functional properties required by the task.

Scene memory can help the robot search efficiently. Previous observations may indicate that cups are commonly stored in a cabinet, cleaning supplies are usually located near a utility area, or a requested tool was recently observed on a workbench. Such memory should be treated as a prior rather than absolute truth because people continuously move objects. Current perception must verify stored information before manipulation.

When the target is partially hidden by clutter, direct grasping may not be possible. The robot can change its viewpoint, reposition its base, move an obstructing object, or perform active perception to reveal additional geometry. This turns object retrieval into a sequence of information-gathering and manipulation actions. The robot should choose intermediate actions that improve access without creating unstable or unsafe arrangements.

Grasp planning considers the intended downstream task rather than merely finding a stable contact. A bottle intended for delivery should be grasped securely while leaving sufficient clearance for transport and placement. A tool may require a functional grasp that preserves access to its working end. Candidate grasps can be evaluated according to stability, reachability, collision risk, object properties, and compatibility with subsequent actions.

After the grasp is executed, the robot must verify that the object was actually acquired. Gripper width, motor current, tactile measurements, force signals, and visual observations can provide complementary evidence. If the object slips or remains on the table, the system should classify the event as a grasp failure rather than continuing the mission under the false assumption that the object is being carried.

Transport introduces additional constraints because the robot\'s collision geometry changes after acquiring an object. The carried item may extend beyond the gripper or robot body and can collide with furniture, doors, or people. Navigation planning should therefore include the payload geometry and select speeds appropriate for object mass, fragility, and stability. The arm may adopt a transport posture that reduces collision risk and mechanical load.

Human-aware navigation is essential in shared environments. People can cross the robot\'s path, approach unexpectedly, or interact with objects that the robot intends to manipulate. The robot should continuously update dynamic obstacle estimates and maintain appropriate separation distances. Motion near people should use conservative velocity and acceleration limits while preserving trajectories that are understandable and predictable to nearby users.

The robot may need to manipulate articulated objects before reaching a target. Retrieving an item from a cabinet requires identifying the handle, estimating the door or drawer mechanism, establishing contact, opening it, and updating perception of the newly exposed interior. The robot must coordinate the mobile base and arm while respecting hinge or rail constraints and monitoring forces for obstruction or unexpected resistance.

Operating buttons, switches, elevators, automatic doors, and other human interfaces extends the service robot\'s autonomy across buildings. These actions require accurate localization of the interface, controlled contact, and verification of the resulting state transition. Pressing a button is not complete merely because the end effector moved forward; the robot should confirm illumination, door motion, display change, or another functional response.

Tool use further expands the service task repertoire. The robot may use a brush for cleaning, a handle for pulling, or a simple instrument for inspection. Tool manipulation requires selecting an appropriate object, acquiring it with a task-oriented grasp, estimating the tool frame, and controlling the functional tip. Force and tactile feedback become especially important when the tool establishes sustained contact with the environment.

Long-horizon service tasks often combine several manipulation categories. Clearing a table may require identifying multiple objects, sorting them by destination, opening a cabinet or waste container, transporting objects individually, and verifying that the workspace has been restored. The system must maintain task state across these actions so that temporary failures do not cause completed steps to be forgotten or unnecessarily repeated.

A manipulation foundation model can provide reusable representations across these diverse behaviors. Multimodal observations, language instructions, robot states, and previous actions can be processed by a shared model that supports semantic grounding and skill selection. Pretraining on heterogeneous robot demonstrations can supply broad manipulation priors, while deployment-specific adaptation improves performance with local furniture, objects, and operational routines.

Vision-language-action policies can generate manipulation behavior directly from sensory context and task instructions. They are particularly useful when a service request varies in wording or involves object categories not represented by conventional symbolic programming. However, learned policies should operate within explicit geometric, control, and safety constraints because semantic reasoning alone cannot guarantee collision-free or physically stable execution.

World models can help predict the consequences of candidate actions before execution. When retrieving an object from clutter, the robot may estimate whether moving one item will expose the target or destabilize neighboring objects. Before opening a cabinet, it can predict the swept volume of the door. Such prediction allows the system to consider downstream effects rather than selecting actions based only on immediate progress.

Closed-loop execution is necessary throughout the mission. The robot repeatedly observes the environment, updates its state estimate, executes a limited action, measures the result, and replans when required. Visual servoing can refine approach trajectories, while force and tactile feedback regulate contact. This continuous perception-action-feedback loop allows the robot to compensate for localization errors, object movement, grasp variation, and unexpected interactions.

Failure recovery is a routine operational requirement rather than an exceptional capability. Common failures include inability to locate the target, unreachable grasps, object slippage, blocked navigation, unexpected cabinet resistance, incorrect object identification, and failed placement. Each failure should produce a diagnostic response such as changing viewpoint, repositioning, regrasping, selecting another skill, or requesting human assistance.

Uncertainty determines how aggressively the robot should act. A clearly visible rigid object in an open workspace can be manipulated efficiently, while an uncertain object near a person may require additional sensing and slower movement. Confidence in perception, grasp quality, localization, and predicted outcomes can therefore influence velocity, collision margins, force limits, and whether autonomous execution should continue.

Safety supervision remains independent from high-level learned intelligence. Joint limits, collision detection, velocity and acceleration restrictions, force thresholds, human proximity monitoring, payload limits, and emergency stopping should continuously constrain robot behavior. Unexpected resistance should trigger reassessment rather than automatic force escalation. The safety layer must retain authority to stop actions generated by planning or learned policies.

Service robots also require operational awareness beyond individual manipulation actions. Battery state, charging availability, communication status, sensor health, actuator temperature, payload capacity, and mission priority can influence task scheduling. A robot should avoid beginning a long delivery when insufficient energy remains and should recognize when degraded sensing makes a manipulation task unsafe or unreliable.

Fleet operation can extend these principles when several service robots share a facility. Robots may divide delivery, cleaning, inspection, and material-handling tasks according to location, capability, battery state, and workload. Shared maps and task information can improve efficiency, while manipulation experience collected by one robot can potentially contribute to model improvement for others through controlled data aggregation and learning.

Deployment data provides an important mechanism for continual improvement. Failed grasps, unusual objects, difficult mechanisms, and human interventions identify situations where the current model is weak. Selected episodes can be reviewed, labeled, or used for additional training. The resulting updated policy can then be validated in simulation and controlled tests before being returned to operational robots.

Privacy and data governance become important when service robots operate in human environments. Cameras and microphones may observe people, documents, screens, or private spaces while performing legitimate tasks. Perception pipelines should minimize unnecessary retention, restrict access to recorded data, and separate operational sensing from training-data collection. Local or on-premise processing can reduce exposure when sensitive information is present.

Performance evaluation should reflect complete service outcomes rather than isolated manipulation benchmarks. Relevant measures include mission success rate, autonomous completion rate, average intervention frequency, object retrieval success, navigation efficiency, manipulation time, recovery success, collision events, object damage, and user waiting time. Reliability over repeated daily operation is more meaningful than a single successful demonstration.

Scenario-based validation should deliberately include environmental variation. Objects can be moved to unfamiliar locations, lighting can change, corridors can become crowded, cabinets can contain clutter, and requested items can differ from training examples. Tests should include sensor uncertainty and recoverable failures. The objective is to determine whether the robot can preserve useful behavior when assumptions made during development no longer hold.

A representative end-to-end service case may require the robot to receive a spoken request, identify the intended object, navigate through a shared corridor, open a cabinet, search inside, retrieve the item, close the cabinet, transport the payload, avoid moving people, place the object at the requested destination, and verbally or digitally report completion. Every stage depends on the successful state established by the preceding stages.

The architecture supporting such behavior is therefore best viewed as an integrated physical intelligence system. Language understanding establishes intent, perception grounds that intent in the environment, navigation creates access, manipulation changes physical state, world models predict consequences, feedback verifies outcomes, memory maintains context, and recovery manages deviations. Safety supervision surrounds the complete execution chain.

The practical value of general manipulation service robots lies in their ability to reuse the same platform across many tasks instead of requiring dedicated automation for every operation. As foundation models, multimodal perception, mobile manipulation, and adaptive control improve, the cost of introducing new service behaviors can progressively shift from hardware redesign and explicit programming toward data-driven adaptation and task instruction.

Ultimately, the service robot case demonstrates what general-purpose manipulation means in operational terms. Intelligence is not measured only by whether the robot can grasp an unfamiliar object, but by whether it can understand a useful request, navigate to the relevant place, interact with human-designed environments, manipulate objects and mechanisms safely, verify results, recover from failures, and complete an extended physical mission with minimal human intervention.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)