<div id="readability-page-1"><p>Apple’s AI research team has developed a new model that could significantly advance how machines perceive depth, potentially transforming industries ranging from augmented reality to autonomous vehicles.</p><p>The system, calledDepth Pro, is able to generate detailed 3D depth maps from single 2D images in a fraction of a second—without relying on the camera data traditionally needed to make such predictions.</p><p>The technology, detailed in a research paper titled<em>“Depth Pro: Sharp Monocular Metric Depth in Less Than a Second,”</em>is a major leap forward in the field of monocular depth estimation, a process that uses just one image to infer depth.</p><p>This could have far-reaching applications across sectors where real-time spatial awareness is key. The model’s creators, led by Aleksei Bochkovskii and Vladlen Koltun, describeDepth Proas one of the fastest and most accurate systems of its kind.</p><figure><img width="830" height="853" src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.26.12%E2%80%AFAM.png/query-dz01ODQ?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" original-src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.26.12%E2%80%AFAM.png/query-dz01ODQ"/><figcaption>A comparison of depth maps from Apple’s Depth Pro, Marigold, Depth Anything v2, and Metric3D v2. Depth Pro excels in capturing fine details like fur and birdcage wires, producing sharp, high-resolution depth maps in just 0.3 seconds, outperforming other models in accuracy and detail. (credit: arxiv.org)</figcaption></figure><p>Monocular depth estimation has long been a challenging task, requiring either multiple images or metadata like focal lengths to accurately gauge depth.</p><p>ButDepth Probypasses these requirements, producing high-resolution depth maps in just 0.3 seconds on a standard GPU. The model can create 2.25-megapixel maps with exceptional sharpness, capturing even minute details like hair and vegetation that are often overlooked by other methods.</p><p>“These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction,” the researchers explain in their paper. This architecture allows the model to process both the overall context of an image and its finer details simultaneously—an enormous leap from slower, less precise models that came before it.</p><figure><img width="846" height="953" src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.34.18%E2%80%AFAM.png/query-dz01MzM?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" original-src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.34.18%E2%80%AFAM.png/query-dz01MzM"/><figcaption>A comparison of depth maps from Apple’s Depth Pro, Depth Anything v2, Marigold, and Metric3D v2. Depth Pro excels in capturing fine details like the deer’s fur, windmill blades, and zebra’s stripes, delivering sharp, high-resolution depth maps in 0.3 seconds. (credit: arxiv.org)</figcaption></figure><h2 id="h-metric-depth-zero-shot-learning">Metric depth, zero-shot learning</h2><p>What truly setsDepth Proapart is its ability to estimate both relative and absolute depth, a capability called “metric depth.”</p><p>This means that the model can provide real-world measurements, which is essential for applications like augmented reality (AR), where virtual objects need to be placed in precise locations within physical spaces.</p><p>AndDepth Prodoesn’t require extensive training on domain-specific datasets to make accurate predictions—a feature known as “zero-shot learning.” This makes the model highly versatile. It can be applied to a wide range of images, without the need for the camera-specific data usually required in depth estimation models.</p><p>“Depth Pro produces metric depth maps with absolute scale on arbitrary images ‘in the wild’ without requiring metadata such as camera intrinsics,” the authors explain. This flexibility opens up a world of possibilities, from enhancing AR experiences to improving autonomous vehicles’ ability to detect and navigate obstacles.</p><p>For those curious to experience Depth Pro firsthand, a live demo is available on the Hugging Face platform.</p><figure><img width="1387" height="375" src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.35.50%E2%80%AFAM.png/query-dz04MDA?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" original-src="https://imageproxy.pbkrs.com/https://venturebeat.com/wp-content/uploads/2024/10/Screenshot-2024-10-04-at-11.35.50%E2%80%AFAM.png/query-dz04MDA"/><figcaption>A comparison of depth estimation models across multiple datasets. Apple’s Depth Pro ranks highest overall with an average rank of 2.5, outperforming models like Depth Anything v2 and Metric3D in accuracy across diverse scenarios. (credit: arxiv.org)</figcaption></figure><h2 id="h-real-world-applications-from-e-commerce-to-autonomous-vehicles">Real-world applications: From e-commerce to autonomous vehicles</h2><p>This versatility has significant implications for various industries. In e-commerce, for example,Depth Procould allow consumers to see how furniture fits in their home by simply pointing their phone’s camera at the room. In the automotive industry, the ability to generate real-time, high-resolution depth maps from a single camera could improve how self-driving cars perceive their environment, boosting navigation and safety.</p><p>“The method should ideally produce metric depth maps in this zero-shot regime to accurately reproduce object shapes, scene layouts, and absolute scales,” the researchers write, emphasizing the model’s potential to reduce the time and cost associated with training more conventional AI models.</p><h2 id="h-tackling-the-challenges-of-depth-estimation">Tackling the challenges of depth estimation</h2><p>One of the toughest challenges in depth estimation is handling what are known as “flying pixels”—pixels that appear to float in mid-air due to errors in depth mapping.Depth Protackles this issue head-on, making it particularly effective for applications like 3D reconstruction and virtual environments, where accuracy is paramount.</p><p>Additionally,Depth Proexcels in boundary tracing, outperforming previous models in sharply delineating objects and their edges. The researchers claim it surpasses other systems “by a multiplicative factor in boundary accuracy,” which is key for applications that require precise object segmentation, such as image matting and medical imaging.</p><h2 id="h-open-source-and-ready-to-scale">Open-source and ready to scale</h2><p>In a move that could accelerate its adoption, Apple has madeDepth Proopen-source. The code, along with pre-trained model weights, is available on GitHub, allowing developers and researchers to experiment with and further refine the technology. The repository includes everything from the model’s architecture to pretrained checkpoints, making it easy for others to build on Apple’s work.</p><p>The research team is also encouraging further exploration ofDepth Pro’s potential in fields like robotics, manufacturing, and healthcare. “We release code and weights athttps://github.com/apple/ml-depth-pro,” the authors write, signaling this as just the beginning for the model.</p><h2 id="h-what-s-next-for-ai-depth-perception">What’s next for AI depth perception</h2><p>As artificial intelligence continues to push the boundaries of what’s possible,<em>Depth Pro</em>sets a new standard in speed and accuracy for monocular depth estimation. Its ability to generate high-quality, real-time depth maps from a single image could have wide-ranging effects across industries that rely on spatial awareness.</p><p>In a world where AI is increasingly central to decision-making and product development,<em>Depth Pro</em>exemplifies how cutting-edge research can translate into practical, real-world solutions. Whether it’s improving how machines perceive their surroundings or enhancing consumer experiences, the potential uses for<em>Depth Pro</em>are broad and varied.</p><p>As the researchers conclude, “Depth Pro dramatically outperforms all prior work in sharp delineation of object boundaries, including fine structures such as hair, fur, and vegetation.” With its open-source release,<em>Depth Pro</em>could soon become integral to industries ranging from autonomous driving to augmented reality—transforming how machines and people interact with 3D environments.</p></div>

AAPL

<p>Apple has launched Depth Pro, an AI model that revolutionizes 3D depth estimation by generating high-resolution depth maps from single images in just 0.3 seconds, without needing camera metadata. This technology enhances applications in augmented reality and autonomous vehicles, providing real-world measurements and exceptional detail. Depth Pro's zero-shot learning capability allows it to function across various images without extensive training. The model addresses challenges in depth estimation, such as flying pixels, and is open-source, promoting widespread adoption across industries.</p>

- Apple’s AI research team has introduced Depth Pro, a model that generates detailed 3D depth maps from single 2D images in just 0.3 seconds, enhancing monocular depth estimation.  
- This technology has significant implications for industries such as augmented reality and autonomous vehicles, allowing for real-time spatial awareness without the need for extensive training on specific datasets.  
- Depth Pro is open-source, enabling developers to refine the model further, potentially transforming applications in robotics, e-commerce, and healthcare.

Apple releases Depth Pro, an AI model that rewrites the rules of 3D vision