Real-Time 3D Model Generation

Recently ByteDance has published Seed3D 2.0. One super interesting idea is it generates 3D content in two stages, creating global structure first and then refining geometric details. (Although this idea seems to first appeared in VLIPP). This pipeline design somewhat resembles 3D graphics pipeline where model goes through vertex shader and fragment shader.

As GPUs become faster, soon we might be able to generate 3D model in real time. A new 3D model every frame. Characters that can evolve throughout the game story. We are already in the era of real-time frame generation with DLSS-FG now. With real-time 3D model generation, it would be a lot easier for everybody to create their own games and animations.

Since we’ll be able to create 3D models on the fly, 3D graphics APIs can be designed around unstructured text and media, and all 3D model/texture/lighting information are produced by the model. GPU can be the black box that turns imagination into full virtual reality by itself.