BILATERAL COARSE-TO-FINE NETWORK FOR POINT CLOUD COMPLETION Tran Thanh Phong Nguyen, Son Lam Phung, Vinod Gopaldasani, Jane Whitelaw University of Wollongong, NSW, Australia ABSTRACT Point cloud completion aims to accurately estimate complete point clouds from partial observations. Existing methods of- ten directly infer the missing points from the partial shape, but they suffer from limited structural information. To address this, we propose the Bilateral Coarse-to-Fine Network (BCF- Net), which leverages 2D images as guidance to compen- sate for structural information loss. Our method introduces a multi-level codeword skip-connection to estimate structural details. Experimental results show that BCF-Net outperforms state-of-the-art point cloud completion networks on synthetic and real-world datasets. Index Terms— Point cloud completion, 3D object mod- eling, bilateral filtering, image guidance. 1. INTRODUCTION Point cloud completion is a vital technique for improving the quality of 3D data, particularly when capturing occluded or constrained views. As 3D applications become more widespread, the need for accurate and complete 3D data is crucial. This paper focuses on point cloud completion, which predicts the complete 3D shape of an object from partial ob- servations. Despite its significance, point cloud completion still has many challenges in improving the accuracy and effi- ciency, and coping with complex and diverse environments. There are still two critical research gaps that need to be addressed. First, current approaches [1–9] predict the com- plete point cloud using only a partial point cloud as input. This leads to the loss of crucial information from the miss- ing parts, making the prediction of missing points uncertain. Most methods [1–5] leverage an encoder-decoder architec- ture to address this issue, where an encoder maps the input point clouds into a codeword, and a decoder reconstructs a complete point cloud by decoding the codeword back to Eu- clidean space. However, the lack of information about the missing points in the partial point cloud remains a signifi- cant challenge. Second, the convolution operation cannot be directly applied to point clouds due to their irregularity and This research is supported by a joint scholarship from Safety Equipment Australia Pty. Ltd. and the University of Wollongong. The first author (T. T. P. Nguyen) is also supported by the Centre for Occupational, Public and Environmental Research in Safety and Health (COPERSH) and the Centre for Signal and Information Processing (CSIP) at the University of Wollongong. unorderedness, leading some methods [10–12] to convert the point cloud into voxels for 3D convolutional neural networks. However, this voxelization operation has two issues. First, it results in an irreversible loss of geometric information, and second, it can be computationally expensive. This paper proposes a novel approach to address the gaps in existing point cloud completion methods. To com- pensate for the information loss, we combine 2D and 3D modules in a non-trivial design. Specifically, we leverage guided features extracted from 2D images to complete the point cloud in a coarse-to-fine manner, thereby providing a more accurate prediction of missing points. To address the structural information problem, we propose the concatena- tion of multi-scale codewords/latent spaces, which performs as a skip-connection operation. This concatenation brings information from multiple levels to complete the point cloud. To enhance the 2D codeword, we leverage a variational auto- encoder to regularize the latent space distribution, improving the overall performance of the method. Additionally, our ap- proach avoids the computational expense of voxelization by proposing a lightweight yet practical network consisting of both 2D and 3D modules. This network can handle irregular and unordered point clouds, providing more flexibility and efficiency in point cloud completion tasks. The main contributions of this paper are threefold: • We propose a novel auto-encoder architecture that combines 2D and 3D modules to address the structural loss of incomplete point clouds. Code is available at https://github.com/phongnguyenai/BCF-Net. • We introduce a multi-level codeword combination that functions as a multi-scale skip-connection operation to predict and maintain structural details. • We present experimental results that demonstrate im- proved completion outcomes compared to existing ap- proaches on both synthetic and real-world data. 2. RELATED WORK This section presents a brief review of algorithms for point cloud completion and view-point guidance on point clouds. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 978-1-7281-6327-7/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICASSP49357.2023.10095048