I'm LongbridgeAI, I can summarize articles.Apple (AAPL-US) research team recently unveiled the latest findings on HALO, an autonomous control system for humanoid robots. By leveraging vision-language models (VLM) and motion generation technology, the system enables robots not only to understand human commands but also to adjust their movements based on unfamiliar environments, autonomously completing tasks such as pushing boxes, sitting on chairs, and navigating obstacles.
Apple Unveils Humanoid Robot "Brain" HALO! Tested on Unitree G1 (Image: REUTERS)
The research team recently published a paper titled "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, introducing the design and experimental results of the HALO system. The system was deployed on Unitree Technology's (688836-CN) G1 humanoid robot for validation.
The research focuses on enabling robots to apply movements learned in simulated environments to real-world spaces that have not been previously scanned or mapped, thereby reducing the robot's dependence on specific environments.
According to demonstration videos released by the research team, the G1 robot equipped with HALO can push boxes, follow humans, and sit on chairs as instructed. It can also sidle through half-open doors or bend down and crawl under tables. These actions are not merely pre-set paths; they involve environmental recognition, spatial judgment, and adjusting body posture in response to obstacles.
HALO Adopts a Three-Layer Architecture to Control Robots, From Understanding Commands to Executing Actions
The core design of HALO splits the robot's control process into three levels: task orchestration, motion generation, and motion tracking, which handle command understanding, full-body motion planning, and actual joint control, respectively.
The top-level task orchestration module uses a vision-language model, defaulting to Alibaba's (09988-HK) Qwen3.6-35B-A3B, running locally on a Mac mini mounted on the robot, allowing it to process tasks without an internet connection. The system also supports calling cloud-based models via API to obtain stronger reasoning and planning capabilities.
This module is responsible for understanding human commands and combining them with images from the robot's perspective to determine how to complete the task. For example, when receiving the command "sit down," the robot must first recognize the surrounding environment to decide on a suitable location to sit. If it receives the command "sit on the chair" but the path to the chair is blocked by a table, it must break down the task into a sequence of continuous actions, such as bending down to pass the obstacle, approaching the chair, and sitting down.
During task execution, the module continuously confirms via video from the robot's perspective whether the action was successful, then decides whether to proceed to the next step or retry, issuing only one clear sub-task to the lower layer at a time.
The middle-layer motion generation module uses another model, Qwen3.5-2B, paired with a DiT decoder responsible for generating motions. It plans full-body movements for the next approximately 5 seconds based on the environmental view and current task. The system replans about 3 times per second, allowing the robot to continuously correct its movements as the environment changes.
The lowest-level motion tracker, trained through processes such as reinforcement learning and policy distillation, operates at a frequency of 50 times per second. It combines depth images with the robot's own state to convert the motions planned by the upper layers into actual joint control commands.
91% Success Rate in Tests Across 6 Unfamiliar Environments; Room for Improvement in Composite Tasks
In practical testing, the research team selected 6 real-world environments that had not been previously scanned or mapped, conducting 360 single-task tests with an average success rate of 91%.
However, the system's performance showed a gap when tasks involved multiple continuous actions. The research team conducted an additional 10 tests for each of 3 composite tasks in a single real-world environment, resulting in an average success rate of 70%.
Among these, the success rate for the continuous task of "bending down to pass an obstacle, walking to the target, and pushing a box" was only 50%, indicating that the robot still faces challenges when handling multi-step actions.
To train the robot to adapt to different environments, the research team established a large-scale synthetic data generation pipeline. They created approximately 300 hours of primary training data across 1,000 procedurally generated simulated home environments, pairing the visuals seen by the robot, the commands received, and the corresponding actions.
The data generation process consists of several stages: first, generating action sequences based on text commands; then placing these actions into virtual rooms; and finally, rendering the views seen by the robot while executing the tasks using simulation tools.
Taking "sit in an armchair" as an example, the system first breaks down the task into action segments such as walking forward, turning around, and sitting down. The motion generation model then produces an action sequence based on text prompts, waypoints, and key poses. The results are filtered through geometric checks to reduce unreasonable actions such as foot sliding or the body clipping through the ground.
Next, the research team arranged the actions in different virtual rooms, checked the positional relationships between the robot and furniture to exclude configurations that might cause collisions, and then rendered the robot's perspective views. The training process also adjusted lighting, color tones, and image blur levels to increase the diversity of visual data, allowing the same pipeline to generate complete training data comprising commands, images, and full-body motions.
Regarding computational resources, the research team used 250 NVIDIA (NVDA-US) H100 GPUs, taking approximately 48 hours to complete the synthesis of the primary training data. The motion generator was trained using imitation learning on 64 B200 GPUs, while the motion tracker was trained using reinforcement learning, policy distillation, and other methods on 4 H100 GPUs.
Still Relies on Manual Design; Haptic Perception is the Next Direction for Improvement
However, HALO has not yet completely eliminated human involvement. The research team pointed out that when adding new physical tasks, personnel still need to write prompts and design keyframes for actions, with some actions requiring the use of human video or motion capture materials.
Furthermore, the current system mainly relies on visual perception and struggles to directly grasp contact states. When objects leave the camera's field of view or are occluded, performance declines in tasks involving significant physical contact.
The research team has listed haptic perception as a direction for future improvement, hoping to further enhance the robot's motion control capabilities in complex environments.