GSDeformer: Direct, Real-time and Extensible Cage-based Deformation for 3D Gaussian Splatting

Jiajun Huang, Shuolin Xu, Hongchuan Yu, Jian Jun Zhang, Hammadi Nait Charif
National Centre for Computer Animation (NCCA), Bournemouth University
arXiv Video Code (Coming Soon)

TL;DR: Real-time cage-based deformation on 3DGS or it's variants, without architectual changes

Abstract

We present GSDeformer, a method that achieves cage-based deformation on 3D Gaussian Splatting (3DGS).

Our method bridges cage-based deformation and 3DGS using a proxy point cloud representation. The point cloud is created from 3DGS, and deformations on the point cloud translate to transformations on the 3D Gaussians that comprise 3DGS. To handle potential bending from deformation, we employ a splitting process to approximate it. Our method does not extend or modify the core architecture of 3DGS; thus, it can work with any existing trained vanilla 3DGS as well as its variants. We also automated cage construction from 3DGS for convenience.

Experiments show that GSDeformer produces superior deformation results than current methods, is robust under extreme deformations, does not require retraining for editing, runs in real-time(60FPS), and can extend to other 3DGS variants.

Method Diagram

Our method deforms gaussians making up the 3DGS representation by converting it to a proxy point cloud representation, deforms it and inferrring the deformation to apply on the gaussians. Splitting are performed to approximate bending gaussians.

Quality and Roubustness

Our method can perform complex deformation, and remains robust even under extreme deformations

Real-time

Our method achieves real-time deformation (on A5000).

Extensibly

Our method can be extended to work with variants of 3DGS, such as FLoD(adding Level-of-Detail), or 2DGS (adding normal reconstruction). Our method can also be combined with other editing methods, such as GaussianEditor.

BibTeX


      @misc{huang2024gsdeformerdirectrealtimeextensible,
        title={GSDeformer: Direct, Real-time and Extensible Cage-based Deformation for 3D Gaussian Splatting}, 
        author={Jiajun Huang and Shuolin Xu and Hongchuan Yu and Jian Jun Zhang and Hammadi Nait Charif},
        year={2024},
        eprint={2405.15491},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2405.15491}, 
      }