The University of Southampton
University of Southampton Institutional Repository

Immersive spatial audio reproduction for VR/AR using room acoustic modelling from 360 images

Immersive spatial audio reproduction for VR/AR using room acoustic modelling from 360 images
Immersive spatial audio reproduction for VR/AR using room acoustic modelling from 360 images
Recent progresses in Virtual Reality (VR) and Augmented Reality (AR) allow us to experience various VR/AR applications in our daily life. In order to maximise the immersiveness of user in VR/AR environments, a plausible spatial audio reproduction synchronised with visual information is essential. In this paper, we propose a simple and efficient system to estimate room acoustic for plausible reproducton of spatial audio using 360° cameras for VR/AR applications. A pair of 360° images is used for room geometry and acoustic property estimation. A simplified 3D geometric model of the scene is estimated by depth estimation from captured images and semantic labelling using a convolutional neural network (CNN). The real environment acoustics are characterised by frequency-dependent acoustic predictions of the scene. Spatially synchronised audio is reproduced based on the estimated geometric and acoustic properties in the scene. The reconstructed scenes are rendered with synthesised spatial audio as VR/AR content. The results of estimated room geometry and simulated spatial audio are evaluated against the actual measurements and audio calculated from ground-truth Room Impulse Responses (RIRs) recorded in the rooms.
120-126
IEEE
Kim, Hansung
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, Luca
c74406cb-15d2-4575-b086-97b55421649e
Jackson, Philip J.B.
c658b148-ce3e-418d-ba12-9e970c9563dc
Hilton, Adrian
12782a55-4c4d-4dfb-a690-62505f6665db
Kim, Hansung
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Remaggi, Luca
c74406cb-15d2-4575-b086-97b55421649e
Jackson, Philip J.B.
c658b148-ce3e-418d-ba12-9e970c9563dc
Hilton, Adrian
12782a55-4c4d-4dfb-a690-62505f6665db

Kim, Hansung, Remaggi, Luca, Jackson, Philip J.B. and Hilton, Adrian (2019) Immersive spatial audio reproduction for VR/AR using room acoustic modelling from 360 images. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE. pp. 120-126 . (doi:10.1109/VR.2019.8798247).

Record type: Conference or Workshop Item (Paper)

Abstract

Recent progresses in Virtual Reality (VR) and Augmented Reality (AR) allow us to experience various VR/AR applications in our daily life. In order to maximise the immersiveness of user in VR/AR environments, a plausible spatial audio reproduction synchronised with visual information is essential. In this paper, we propose a simple and efficient system to estimate room acoustic for plausible reproducton of spatial audio using 360° cameras for VR/AR applications. A pair of 360° images is used for room geometry and acoustic property estimation. A simplified 3D geometric model of the scene is estimated by depth estimation from captured images and semantic labelling using a convolutional neural network (CNN). The real environment acoustics are characterised by frequency-dependent acoustic predictions of the scene. Spatially synchronised audio is reproduced based on the estimated geometric and acoustic properties in the scene. The reconstructed scenes are rendered with synthesised spatial audio as VR/AR content. The results of estimated room geometry and simulated spatial audio are evaluated against the actual measurements and audio calculated from ground-truth Room Impulse Responses (RIRs) recorded in the rooms.

Full text not available from this repository.

More information

Accepted/In Press date: 15 February 2019
e-pub ahead of print date: 15 August 2019
Published date: 2019
Venue - Dates: IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Japan, 2019-03-23 - 2019-03-27

Identifiers

Local EPrints ID: 438834
URI: http://eprints.soton.ac.uk/id/eprint/438834
PURE UUID: 1c00dde0-26e0-4b0e-b28f-7df461041ea7
ORCID for Hansung Kim: ORCID iD orcid.org/0000-0003-4907-0491

Catalogue record

Date deposited: 25 Mar 2020 17:31
Last modified: 07 Oct 2020 02:27

Export record

Altmetrics

Contributors

Author: Hansung Kim ORCID iD
Author: Luca Remaggi
Author: Philip J.B. Jackson
Author: Adrian Hilton

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×