3D room geometry reconstruction using audio-visual sensors
3D room geometry reconstruction using audio-visual sensors
In this paper we propose a cuboid-based air-Tight indoor room geometry estimation method using combination of audio-visual sensors. Existing vision-based 3D reconstruction methods are not applicable for scenes with transparent or reflective objects such as windows and mirrors. In this work we fuse multi-modal sensory information to overcome the limitations of purely visual reconstruction for reconstruction of complex scenes including transparent and mirror surfaces. A full scene is captured by 360°C cameras and acoustic room impulse responses (RIRs) recorded by a loudspeaker and compact microphone array. Depth information of the scene is recovered by stereo matching from the captured images and estimation of major acoustic reflector locations from the sound. The coordinate systems for audio-visual sensors are aligned into a unified reference frame and plane elements are reconstructed from audio-visual data. Finally cuboid proxies are fitted to the planes to generate a complete room model. Experimental results show that the proposed system generates complete representations of the room structures regardless of transparent windows, featureless walls and shiny surfaces.
Audio systems, Geometry, Loudspeakers, Mirrors, Stereo image processing, 3D reconstruction, Acoustic reflectors, Acoustic room impulse response, Audio-visual, Audio-visual sensors, Room geometry estimations, Sensory information, Visual reconstruction, Image reconstruction
621-629
Kim, H.
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, L.
c74406cb-15d2-4575-b086-97b55421649e
Jackson, P.J.
19738914-62dc-4779-92b7-186b66302d5f
Fazi, F.M.
e5aefc08-ab45-47c1-ad69-c3f12d07d807
Hilton, A.
12782a55-4c4d-4dfb-a690-62505f6665db
7 June 2018
Kim, H.
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, L.
c74406cb-15d2-4575-b086-97b55421649e
Jackson, P.J.
19738914-62dc-4779-92b7-186b66302d5f
Fazi, F.M.
e5aefc08-ab45-47c1-ad69-c3f12d07d807
Hilton, A.
12782a55-4c4d-4dfb-a690-62505f6665db
Kim, H., Remaggi, L., Jackson, P.J., Fazi, F.M. and Hilton, A.
(2018)
3D room geometry reconstruction using audio-visual sensors.
In 2017 International Conference on 3D Vision (3DV).
IEEE.
.
(doi:10.1109/3DV.2017.00076).
Record type:
Conference or Workshop Item
(Paper)
Abstract
In this paper we propose a cuboid-based air-Tight indoor room geometry estimation method using combination of audio-visual sensors. Existing vision-based 3D reconstruction methods are not applicable for scenes with transparent or reflective objects such as windows and mirrors. In this work we fuse multi-modal sensory information to overcome the limitations of purely visual reconstruction for reconstruction of complex scenes including transparent and mirror surfaces. A full scene is captured by 360°C cameras and acoustic room impulse responses (RIRs) recorded by a loudspeaker and compact microphone array. Depth information of the scene is recovered by stereo matching from the captured images and estimation of major acoustic reflector locations from the sound. The coordinate systems for audio-visual sensors are aligned into a unified reference frame and plane elements are reconstructed from audio-visual data. Finally cuboid proxies are fitted to the planes to generate a complete room model. Experimental results show that the proposed system generates complete representations of the room structures regardless of transparent windows, featureless walls and shiny surfaces.
This record has no associated files available for download.
More information
e-pub ahead of print date: 10 October 2017
Published date: 7 June 2018
Additional Information:
cited By 4
Venue - Dates:
International Conference on 3D Vision, 2017-09-15
Keywords:
Audio systems, Geometry, Loudspeakers, Mirrors, Stereo image processing, 3D reconstruction, Acoustic reflectors, Acoustic room impulse response, Audio-visual, Audio-visual sensors, Room geometry estimations, Sensory information, Visual reconstruction, Image reconstruction
Identifiers
Local EPrints ID: 440604
URI: http://eprints.soton.ac.uk/id/eprint/440604
ISSN: 2475-7888
PURE UUID: 0d628677-16b9-40d9-b167-143505abbd2c
Catalogue record
Date deposited: 12 May 2020 16:31
Last modified: 17 Mar 2024 04:01
Export record
Altmetrics
Contributors
Author:
H. Kim
Author:
L. Remaggi
Author:
P.J. Jackson
Author:
A. Hilton
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics