We introduce Shape Tokens, a 3D representation that is continuous, compact, and easy to integrate into machine learning models. Shape Tokens serve as conditioning vectors, representing shape information within a 3D flow-matching model. This flow-matching model is trained to approximate probability density functions corresponding to delta functions concentrated on the surfaces of 3D shapes. By incorporating Shape Tokens into various machine learning models, we can generate new shapes, convert images to 3D, align 3D shapes with text and images, and render shapes directly at variable, user-specified resolutions. Additionally, Shape Tokens enable a systematic analysis of geometric properties, including normals, density, and deformation fields. Across tasks and experiments, the use of Shape Tokens demonstrates strong performance compared to existing baselines.
From left to right:
- Input image
- Spatter-image (CVPR 2024): trained on Objaverse
- Point-e (2022): trained on several million proprietary 3D meshes.
- Make-a-shape (ICML 2024): trained on 18 datasets, including Objaverse
- Ours: trained on Objaverse
Note that this video does not intend to compare individual methods — these models differ in their training data (e.g., Point-e was trained on proprietary 3D meshes) and mechanisms (e.g., Splatter-image is not a generative model, our method assumes a known camera model). We provide the results for the viewer’s reference. [Credits]
From left to right:
- Ground-truth surface normal
- Pointersect (CVPR 2023)
- Ours
