Researchers from the Australian National University, the University of Oxford, and the Beijing Academy of Artificial Intelligence have introduced a groundbreaking AI system called “3D-GPT” that has the ability to generate 3D models based on text descriptions provided by users. This system, as detailed in a paper published on arXiv, offers a more efficient and intuitive way to create 3D assets compared to traditional 3D modeling workflows.
In their research, the team behind 3D-GPT explained that the system dissects procedural 3D modeling tasks into accessible segments and assigns the appropriate AI agent for each task. Multiple AI agents are used to understand the text prompt and execute modeling functions.
3D-GPT has been designed to position large language models (LLMs) as proficient problem solvers, breaking down the modeling process and appointing specialized AI agents. These agents include a task dispatch agent that parses the text instructions, a conceptualization agent that adds missing details to the initial description, and a modeling agent that determines parameters and generates code for 3D software.
By breaking down the modeling process and utilizing specialized AI agents, 3D-GPT is able to interpret text prompts, enrich descriptions with additional details, and ultimately generate 3D assets that accurately match the user’s vision. The system enhances initial scene descriptions, transforming them into detailed forms while adapting the text based on subsequent instructions.
Testing was conducted using prompts such as “a misty spring morning, where dew-kissed flowers dot a lush meadow surrounded by budding trees.” 3D-GPT successfully generated complete 3D scenes with realistic graphics that faithfully reflected the elements described in the text.
Although the graphics produced by 3D-GPT are not yet photorealistic, this agent-based approach shows promise in simplifying 3D content creation. The modular architecture of the system allows for independent improvements to each agent component.
The researchers highlight that 3D-GPT not only interprets and executes instructions, delivering reliable results, but also effectively collaborates with human designers.
By generating code to control existing 3D software rather than building models from scratch, 3D-GPT offers a flexible foundation for future advancements in modeling techniques.
The potential of large language models in 3D modeling is showcased by 3D-GPT, providing a basic framework for future advancements in scene generation and animation. This research has the potential to revolutionize the 3D modeling industry, making the process more efficient and accessible. With the rise of the metaverse era and the increasing importance of 3D content creation in various industries, tools like 3D-GPT could prove invaluable to creators and decision-makers in gaming, virtual reality, cinema, and multimedia experiences.
Although the 3D-GPT framework is still in its early stages and has limitations, its development represents a significant step forward in AI-driven 3D modeling and opens up exciting possibilities for future advancements.
VentureBeat’s mission is to serve as a digital town square for technical decision-makers to acquire knowledge about transformative enterprise technology and engage in transactions. [Discover our Briefings.](https://venturebeat.com/newsletters/?utm_source=VBsite&utm_medium=bottomBoilerplate)
