The University of Southampton
University of Southampton Institutional Repository

Recovering surface normal and arbitrary images: a dual regression network for photometric stereo

Recovering surface normal and arbitrary images: a dual regression network for photometric stereo
Recovering surface normal and arbitrary images: a dual regression network for photometric stereo
Photometric stereo recovers three-dimensional (3D) object surface normal from multiple images under different illumination directions. Traditional photometric stereo methods suffer from the problem of non-Lambertian surfaces with general reflectance. By leveraging deep neural networks, learning-based methods are capable of improving the surface normal estimation under general non-Lambertian surfaces. These state-of-the-art learning-based methods however do not associate surface normal with reconstructed images and, therefore, they cannot explore the beneficial effect of such association on the estimation of the surface normal. In this paper, we specifically exploit the positive impact of this association and propose a novel dual regression network for both fine surface normals and arbitrary reconstructed images in calibrated photometric stereo. Our work unifies the 3D reconstruction and rendering tasks in a deep learning framework, with the explorations including: 1.~generating specified reconstructed images under arbitrary illumination directions, which provides more intuitive perception of the reflectance and is extremely useful for visual applications, such as virtual reality, and 2.~our dual regression scheme introduces an additional constraint on observed images and reconstructed images, which forms a closed-loop to provide additional supervision. Experiments show that our proposed method achieves accurate reconstructed images under arbitrarily specified illumination directions and it significantly outperforms the state-of-the-art learning-based single regression methods in calibrated photometric stereo.
1057-7149
3676-3690
Ju, Yakun
26e6954c-7ceb-49e1-bd23-3a9cd73bc673
Dong, Junyu
ef350fb2-8682-4a0a-b60e-ebcb7f55085f
Chen, Sheng
9310a111-f79a-48b8-98c7-383ca93cbb80
Ju, Yakun
26e6954c-7ceb-49e1-bd23-3a9cd73bc673
Dong, Junyu
ef350fb2-8682-4a0a-b60e-ebcb7f55085f
Chen, Sheng
9310a111-f79a-48b8-98c7-383ca93cbb80

Ju, Yakun, Dong, Junyu and Chen, Sheng (2021) Recovering surface normal and arbitrary images: a dual regression network for photometric stereo. IEEE Transactions on Image Processing, 30, 3676-3690. (doi:10.1109/TIP.2021.3064230).

Record type: Article

Abstract

Photometric stereo recovers three-dimensional (3D) object surface normal from multiple images under different illumination directions. Traditional photometric stereo methods suffer from the problem of non-Lambertian surfaces with general reflectance. By leveraging deep neural networks, learning-based methods are capable of improving the surface normal estimation under general non-Lambertian surfaces. These state-of-the-art learning-based methods however do not associate surface normal with reconstructed images and, therefore, they cannot explore the beneficial effect of such association on the estimation of the surface normal. In this paper, we specifically exploit the positive impact of this association and propose a novel dual regression network for both fine surface normals and arbitrary reconstructed images in calibrated photometric stereo. Our work unifies the 3D reconstruction and rendering tasks in a deep learning framework, with the explorations including: 1.~generating specified reconstructed images under arbitrary illumination directions, which provides more intuitive perception of the reflectance and is extremely useful for visual applications, such as virtual reality, and 2.~our dual regression scheme introduces an additional constraint on observed images and reconstructed images, which forms a closed-loop to provide additional supervision. Experiments show that our proposed method achieves accurate reconstructed images under arbitrarily specified illumination directions and it significantly outperforms the state-of-the-art learning-based single regression methods in calibrated photometric stereo.

Text
TIP-22815-2020 - Accepted Manuscript
Download (3MB)
Text
TIP2021-March - Version of Record
Restricted to Repository staff only
Request a copy

More information

Accepted/In Press date: 25 February 2021
e-pub ahead of print date: 11 March 2021
Published date: 18 March 2021

Identifiers

Local EPrints ID: 447330
URI: http://eprints.soton.ac.uk/id/eprint/447330
ISSN: 1057-7149
PURE UUID: 514327cc-a5e3-4967-b4e1-8923017b563c

Catalogue record

Date deposited: 09 Mar 2021 17:32
Last modified: 19 Mar 2021 17:37

Export record

Altmetrics

Contributors

Author: Yakun Ju
Author: Junyu Dong
Author: Sheng Chen

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×