NVIDIA has unveiled Neuralangelo, a ground-breaking AI model developed by NVIDIA Research that converts 2D video clips into intricate 3D structures. The model has the potential to revolutionize numerous fields, from art and video game development to robotics and industrial digital twins.
Neuralangelo is able to construct lifelike virtual replicas of buildings, sculptures, and various other real-world objects. This innovative model outperforms previous methods by translating the textures of complex materials, such as roof shingles, glass panes, and smooth marble, from 2D videos into 3D assets. The high fidelity of its 3D reconstructions simplifies the process for developers and creative professionals to quickly create usable virtual objects for their projects using footage captured by smartphones.
“The 3D reconstruction capabilities Neuralangelo offers will be a huge benefit to creators, helping them recreate the real world in the digital world,” said Ming-Yu Liu, Senior Director of Research and co-author of the paper. “This tool will eventually enable developers to import detailed objects whether small statues or massive buildings into virtual environments for video games or industrial digital twins”.
In a demonstration, NVIDIA researchers showcased the model’s ability to recreate objects as iconic as Michelangelo’s David and as commonplace as a flatbed truck. Neuralangelo can also reconstruct building interiors and exteriors, as exemplified by a detailed 3D model of the park at NVIDIA’s Bay Area campus.
Neuralangelo employs instant neural graphics primitives, the technology behind NVIDIA Instant NeRF, to capture finer details that earlier AI models to reconstruct 3D scenes have struggled with, including accurately capturing repetitive texture patterns, homogenous colors, and strong color variations. The model selects several frames from a 2D video of an object or scene filmed from various angles to capture different viewpoints. After determining the camera position of each frame, Neuralangelo’s AI creates a rough 3D representation of the scene and then optimizes the render to sharpen the details.
The final result is a 3D object or large-scale scene that can be utilized in virtual reality applications, digital twins, or robotics development.
Neuralangelo is one of nearly 30 projects by NVIDIA Research set to be presented at the Conference on Computer Vision and Pattern Recognition (CVPR) taking place June 18-22 in Vancouver.