Shap-E as a conditional generative model for 3D assets

 

OpenAI researchers Heewoo Jun and Alex Nichol have unveiled a new artificial intelligence model that can create 3D models assets ready for machine printing using text and descriptions. Dubbed ‘Shap-E’, the text-to-3D-model assets software can be used with 3D printing to speed up producing objects without heavily relying on computer renders. The recently developed open-source software is from OpenAI, the masterminds behind ChatGPT and DALL-E (see more here and here), bringing the text-to-product idea to life in just a few words or descriptions.

 

Shap-E is being developed to be a conditional generative model for 3D assets. OpenAI has already built and trained a neural network called DALL·E that creates images from text captions, with the results more attuned for paper printing and web-resource uses. The AI company has also tried its hands with Shap-E’s predecessor Point-E, an explicit generative model that accumulates millions of point clouds to churn out an image, but claims that ‘Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space.’

 

While there is a possibility for the model created by Shap-E to be used for 3D printing, the researchers give a heads-up that without adequate validation or safety testing, producing tools and parts from text to real objects may pose potential risks. ‘This could potentially be harmful when the produced samples do not adequately meet the desired prompt,’ the researchers add.

from avocado chair to banana plane, OpenAI’s new tool generates text-to-3D print objects
images courtesy of OpenAI researchers Heewoo Jun and Alex Nichol | header image by Inés Álvarez Fdez on Unsplash

 

 

3D printing-ready models using text-to-image generator

 

In a series of comparative tests between Shap-E and Point-E that Jun and Nichol run, they found that Shap-E could match or outperform a similar explicit generative model given the same dataset, model architecture, and training compute. The outcomes that the recent software could create showcase softer edges, more shadowed outlines, and more plump figures compared to the pixelated – while still visible – images by Point-E. ‘For example, we find that Point-E sometimes ignores the small slits in the bench, whereas Shap-E attempts to model them,’ the researchers state in their published study.

 

When it comes to 3D printing, users have to feed the rendered model to the machine before it can produce a tangible product. With the entrance of Shap-E, OpenAI is also exploring the possibility of already generating textured images just by typing descriptions which can then be fed to the machine without hustling away with rendering software. So far, Shap-E can spawn diverse, interesting objects without relying on images as an intermediate representation. ‘These results highlight the potential of generating implicit representations, especially in domains like 3D where they can offer more flexibility than explicit representations,’ the researchers write.

from avocado chair to banana plane, OpenAI’s new tool generates text-to-3D print objects
the recent generative model can produce sharper and more detailed images

 

 

Improving the limitations of Shap-e

 

Amidst some visible improvements from the last text-to-image generative software, Jun and Nichol acknowledge the limitations the research team still has to look into. They mention that Shap-E has a limited ability to compose concepts, struggle to bind multiple attributes to different objects, and can fail to produce the correct amount of objects when requested (they asked for four cupcakes but Shap-E generated three muddled and converged ones).

 

The samples Shap-E can also produce still lack fine details as they can come across as rough or resemble graphics found in video games from the previous eras. ‘For the best results, Shap·E could potentially be combined with optimization-based 3D generative techniques,’ say Jun and Nichol. For those who want to try the software, the research team has uploaded the files and instructions on their open-source page, but knowing that they’re in the middle of their tests, figuring out how to navigate through the downloadable app might take time.

 

 

project info:

 

name: Shap-E

company: OpenAI

research team: Heewoo Jun, Alex Nichol