The University of Southampton
University of Southampton Institutional Repository

3D room geometry reconstruction using audio-visual sensors

3D room geometry reconstruction using audio-visual sensors
3D room geometry reconstruction using audio-visual sensors
In this paper we propose a cuboid-based air-Tight indoor room geometry estimation method using combination of audio-visual sensors. Existing vision-based 3D reconstruction methods are not applicable for scenes with transparent or reflective objects such as windows and mirrors. In this work we fuse multi-modal sensory information to overcome the limitations of purely visual reconstruction for reconstruction of complex scenes including transparent and mirror surfaces. A full scene is captured by 360°C cameras and acoustic room impulse responses (RIRs) recorded by a loudspeaker and compact microphone array. Depth information of the scene is recovered by stereo matching from the captured images and estimation of major acoustic reflector locations from the sound. The coordinate systems for audio-visual sensors are aligned into a unified reference frame and plane elements are reconstructed from audio-visual data. Finally cuboid proxies are fitted to the planes to generate a complete room model. Experimental results show that the proposed system generates complete representations of the room structures regardless of transparent windows, featureless walls and shiny surfaces.
Audio systems, Geometry, Loudspeakers, Mirrors, Stereo image processing, 3D reconstruction, Acoustic reflectors, Acoustic room impulse response, Audio-visual, Audio-visual sensors, Room geometry estimations, Sensory information, Visual reconstruction, Image reconstruction
2475-7888
621-629
IEEE
Kim, H.
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, L.
c74406cb-15d2-4575-b086-97b55421649e
Jackson, P.J.
19738914-62dc-4779-92b7-186b66302d5f
Fazi, F.M.
e5aefc08-ab45-47c1-ad69-c3f12d07d807
Hilton, A.
12782a55-4c4d-4dfb-a690-62505f6665db
Kim, H.
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, L.
c74406cb-15d2-4575-b086-97b55421649e
Jackson, P.J.
19738914-62dc-4779-92b7-186b66302d5f
Fazi, F.M.
e5aefc08-ab45-47c1-ad69-c3f12d07d807
Hilton, A.
12782a55-4c4d-4dfb-a690-62505f6665db

Kim, H., Remaggi, L., Jackson, P.J., Fazi, F.M. and Hilton, A. (2018) 3D room geometry reconstruction using audio-visual sensors. In 2017 International Conference on 3D Vision (3DV). IEEE. pp. 621-629 . (doi:10.1109/3DV.2017.00076).

Record type: Conference or Workshop Item (Paper)

Abstract

In this paper we propose a cuboid-based air-Tight indoor room geometry estimation method using combination of audio-visual sensors. Existing vision-based 3D reconstruction methods are not applicable for scenes with transparent or reflective objects such as windows and mirrors. In this work we fuse multi-modal sensory information to overcome the limitations of purely visual reconstruction for reconstruction of complex scenes including transparent and mirror surfaces. A full scene is captured by 360°C cameras and acoustic room impulse responses (RIRs) recorded by a loudspeaker and compact microphone array. Depth information of the scene is recovered by stereo matching from the captured images and estimation of major acoustic reflector locations from the sound. The coordinate systems for audio-visual sensors are aligned into a unified reference frame and plane elements are reconstructed from audio-visual data. Finally cuboid proxies are fitted to the planes to generate a complete room model. Experimental results show that the proposed system generates complete representations of the room structures regardless of transparent windows, featureless walls and shiny surfaces.

Full text not available from this repository.

More information

e-pub ahead of print date: 10 October 2017
Published date: 7 June 2018
Additional Information: cited By 4
Venue - Dates: International Conference on 3D Vision, ,, 2017-09-15
Keywords: Audio systems, Geometry, Loudspeakers, Mirrors, Stereo image processing, 3D reconstruction, Acoustic reflectors, Acoustic room impulse response, Audio-visual, Audio-visual sensors, Room geometry estimations, Sensory information, Visual reconstruction, Image reconstruction

Identifiers

Local EPrints ID: 440604
URI: http://eprints.soton.ac.uk/id/eprint/440604
ISSN: 2475-7888
PURE UUID: 0d628677-16b9-40d9-b167-143505abbd2c
ORCID for H. Kim: ORCID iD orcid.org/0000-0003-4907-0491

Catalogue record

Date deposited: 12 May 2020 16:31
Last modified: 18 Feb 2021 17:41

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×