The proposed architecture, which is based off the pix2vox framework. We provide two additions: 1) incorporating domain adaptation in the style of DANN through a domain classifier and a gradient reversal layer, and 2) since we have classification labels for both the source and target domain, we also classify the produced voxel.
In this project, several unsupervised domain adaptation methods were applied to the task of single view 3D reconstruction, including MMD, Deep CORAL, and DANN. The source domain is ShapeNet which is synthetic, while the target domain is OOWL which is in the real domain (I helped design OOWL, see here for more details). Overall, we found that our proposed DANN with a multitask learning voxel classification branch led to the best results. For details, please refer to the paper.
An example object (lamp) from datasets used for our project. OWILD, OOWL, and OOWLSeg are part of the ODDS dataset, and are real. Meanwhile, ShapeNet is synthetic.Reconstruction results when domain adaptation is used with OOWL as the target domain. We compare domain adaptation using Deep CORAL, DANN, and our proposed architecture (DANN + Voxel Classify).Learned feature map embeddings visualized with t-SNE. Purple denotes the source domain (ShapeNet), yellow denotes the target domain (OOWL). On the left we show the result when domain adaptation is not used. On the center and right, we show results achieved by applying domain adaptation (DANN or CORAL) to the vanilla pix2vox model.
Citation Info
Leung, B., Singh, S., & Horodniceanu, A. (2020). Domain adaptation for real-world single view 3d reconstruction. ArXiv:2108.10972 [Cs]. http://arxiv.org/abs/2108.10972
Github
