Noam Aigerman

Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects

Amir Barda

Matheus Gadelha

Vladimir Kim

Amit H. Bermano

Thibault Groueix

We propose a generative technique to edit 3D shapes, represented as meshes, NeRFs, or Gaussian Splats, in approximately 3 seconds, without t… (voir plus)he need for running an SDS type of optimization. Our key insight is to cast 3D editing as a multiview image inpainting problem, as this representation is generic and can be mapped back to any 3D representation using the bank of available Large Reconstruction Models. We explore different fine-tuning strategies to obtain both multiview generation and inpainting capabilities within the same diffusion model. In particular, the design of the inpainting mask is an important factor of training an inpainting model, and we propose several masking strategies to mimic the types of edits a user would perform on a 3D shape. Our approach takes 3D generative editing from hours to seconds and produces higher-quality results compared to previous works.

2024-11-30

ArXiv (prépublication)

Sketch-guided Cage-based 3D Gaussian Splatting Deformation

Tianhao Xie

Eugene Belilovsky

Tiberiu Popa

3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and c… (voir plus)omputer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.

2024-11-19

ArXiv (prépublication)

Differentiation Through Black-Box Quadratic Programming Solvers

Connor W. Magoon

Fengyu Yang

Shahar Kovalsky

2024-10-08

ArXiv (prépublication)

DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement

Qimin Chen

Zhiqin Chen

Vladimir Kim

Hao (Richard) Zhang

Hao Zhang 0002

Siddhartha Chaudhuri

2024-10-03

Lecture Notes in Computer Science (publié)

MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation

Hyunwoo Kim

Itai Lang

Thibault Groueix

Vladimir Kim

Rana Hanocka

We propose MeshUp, a technique that deforms a 3D mesh towards multiple target concepts, and intuitively controls the region where each conce… (voir plus)pt is expressed. Conveniently, the concepts can be defined as either text queries, e.g.,"a dog"and"a turtle,"or inspirational images, and the local regions can be selected as any number of vertices on the mesh. We can effectively control the influence of the concepts and mix them together using a novel score distillation approach, referred to as the Blended Score Distillation (BSD). BSD operates on each attention layer of the denoising U-Net of a diffusion model as it extracts and injects the per-objective activations into a unified denoising pipeline from which the deformation gradients are calculated. To localize the expression of these activations, we create a probabilistic Region of Interest (ROI) map on the surface of the mesh, and turn it into 3D-consistent masks that we use to control the expression of these activations. We demonstrate the effectiveness of BSD empirically and show that it can deform various meshes towards multiple objectives. Our project page is at https://threedle.github.io/MeshUp.

2024-08-27

ArXiv (prépublication)

Temporal Residual Jacobians For Rig-free Motion Transfer

Sanjeev Muralikrishnan

Niladri Shekhar Dutt

Siddhartha Chaudhuri

Vladimir Kim

Matthew Fisher

Niloy J. Mitra

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume acces… (voir plus)s to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two coupled neural networks that individually predict local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its superiority in generating realistic and natural-looking animations on unseen body shapes against SoTA alternatives. Supplemental video and code are available at https://temporaljacobians.github.io/ .

2024-07-20

ArXiv (prépublication)

Temporal Residual Jacobians For Rig-free Motion Transfer

Sanjeev Muralikrishnan

Niladri Shekhar Dutt

Siddhartha Chaudhuri

Vladimir Kim

Matthew Fisher

Niloy J. Mitra

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume acces… (voir plus)s to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two coupled neural networks that individually predict local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its superiority in generating realistic and natural-looking animations on unseen body shapes against SoTA alternatives. Supplemental video and code are available at https://temporaljacobians.github.io/ .

2024-07-20

ArXiv (prépublication)