Handling Sub-Image Queries In Content-Based Retrieval of High Resolution Art Images

Stephen Chan¹, Kirk Martinez¹, Paul Lewis¹, C. Lahanier², J. Stevenson³

¹Intelligence, Agents and Multimedia Research Group, University of Southampton, UK

²Centre de Recherche et de Restauration des Musées de France

³Victoria and Albert Museum, London, UK

Abstract

The paper describes work which is part of the Artiste project to develop a distributed database of Art Images from four major European art galleries. One aspect of the project involves the development of content-based retrieval and navigation facilities and a particular objective is to provide a facility for retrieving an image from the collection or navigating to related information in the database, given a query image which may be only a part of a particular image in the collection. The query sub-image may be a poor quality reproduction of part of the original and may be digitised under significantly different conditions. The paper outlines one of the recent approaches we have developed to facilitate such modes of retrieval and navigation. It is based on the use of colour coherence vectors extracted from image patches for the query and target images at a range of scales with multiple vector matching in order to find the best sub-image matches. Some results of the application of the technique are described and its success at sub-image location from a collection of images, including many at very high resolution, is demonstrated.

Introduction

ARTISTE is an EC funded project under the IST work programme and aims to develop an Integrated Art Analysis and Navigation Environment hosted on a distributed database and accessed via the World Wide Web. The environment will include, amongst other things, facilities for searching and analysing digital images and will include tools, not only for retrieval using metadata but also for a wide range of content based retrieval activities. Partners in the collaboration include NCR Systems Engineering Copenhagen, Giunti Interactive Labs, Centre de Reserche et de Restoration des Musées de France at the Louvre, the National Gallery London, the Uffizi in Florence, the University of Southampton and the Victoria and Albert Museum London.

Content-based image retrieval is a challenging and active research area with the potential to provide powerful facilities for image searching. But although many techniques have been described in the research literature, it is probably not an exaggeration to maintain that the capabilities of content matching alone are still relatively basic as general purpose approaches although some powerful applications specific methods can be developed. General techniques based on such features as colour distribution, texture, outline shape and spatial colour distribution have been popular in the research literature and in content based retrieval systems but many techniques only work on the complete images, will not allow the query image to be a sub-image of the image to be retrieved and require similarity in image resolution between query and target.

In this paper we describe in more detail some of the problems of content based retrieval using colour matching and present an approach to sub-image retrieval for collections which include large numbers of very high resolution art images.

Content-Based Retrieval

Several extensive general reviews of content-based image retrieval techniques have appeared in recent years (Smeulders 2000). CBR has been applied to Art images since the 1980s with the Morelli project (Vaughan 91) and IBM's QBIC for example has been applied to such images in a collaboration with UC-Davis (Holt 97). Previous approaches usually applied a generic CBR system to a group of Art images to see how useful it would be. In Artiste we aim to solve specific CBR problems in this way rather than producing a generic solution for all queries.

Use of the colour histogram (Swain-91) for comparing images has been popular primarily because it is easy to compute, is fairly insensitive to small variations between images and is relatively efficient in terms of computation speed and storage requirements. It has the disadvantage that it does not capture any information about the distribution of the colours spatially within the image and various techniques to capture such information have been proposed including the colour coherence vector approach (Pass-96).

Sub-Image Matching and The M-CCV Method

In this part of the project, the aim is to develop a robust technique to retrieve database images of an artwork given a query image, which represents all or part of the target artwork. The query image may have been captured at a different resolution from the database image, may only represent a fragment of that image and may be distorted or degraded in some way. For example, the query image may be a fragment from an image captured prior to restoration of an artwork and the target image may represent the painting after restoration.

In general, global techniques such as the colour histogram are not effective for sub-image matching in their basic form. Some attempts have been made to retrieve sub-images using colour histograms by dividing the target images into pyramids of patches and recording the colour histogram for each. However, in our approach called the Multi-scale Colour Coherence Vector (M-CCV) method, we use the colour coherence vector (CCV) rather than the basic colour histogram as the representation of the individual image patches as it carries some useful local spatial information. The colour coherence vector records the numbers of coherent and incoherent pixels of each colour in the patch where a coherent pixel is one which belongs to a patch of similar coloured pixels whose relative size is above a certain threshold. We pre-compute the colour coherence vectors CCVs for overlapping patches for each database image, repeating the process at reduced resolution until the reduced image corresponds to a single patch. The query sub-image is also sub-divided into patches and the best match is obtained by combining the CCV match scores. The CCVs are coded for rapid matching and can be compared at an average rate of 265,000 CCVs per second.

Results

In figure 1 we show a query image which is a fragment of the Moise présenté à Pharaon by Orsel Victor captured before restoration work at the Louvre gallery. The best match is shown in the same figure and it can be seen that the best match is with the correct parent image but after restoration. (The before restoration image was not in the database). Note that the position of the match is highlighted, a necessary facility when dealing with ultra high resolution images. This particular parent image is 6328 by 4712 pixels. It should be stressed that the parent image was scanned at 1.47 pixels per mm of the original painting whereas the sub-image query was captured at a quite different resolution, 0.92 pixels per mm. There were over 1000 images in the database varying in size from 440,000 pixels to 30,000,000 pixels and the retrieval process took about 45.8 seconds on a Pentium III 600Mhz PC. No multidimensional indexing has been used so far although it is intended that the process will be accelerated by its introduction.

In figure 2 we show a further example of the multi scale CCV retrieval in action. The query image is a selection from a fabric from the V&A collection and the best retrievals show the parent tapestry and others containing a similar motif to the sub-image query pattern. The retrieval process took 13.74 seconds to complete for the same database.

Figure 2. The query image and the top three matches are shown with their respective ranking values.

Conclusions and Future Work

In this short paper we have shown that the multi-scale colour coherence vector (M-CCV) technique can provide effective sub-image retrieval and is also applicable when the sub-mage and target are captured at different resolutions. The technique will form part of a battery of content-based retrieval and navigation tools which, together with integrated metadata based retrieval and navigation, will constitute the Artiste art image retrieval and analysis system.

Acknowledgements

The authors are grateful to the European Commission for their support through grant IST-1999-11978 and to their collaborators (NCR (Dk), Giunti Interactive Labs (I), Uffizi Gallery (I), IT Innovation Centre (UK), The National Gallery (UK)) on the ARTISTE project for image data and useful conversations.

References

[Holt 97] Bonnie Holt and Ken Weiss. The QBIC Project in the Department of Art and Art History at

UC Davis. In The 60^th ASIS Annual Meeting 1997, Digital Collections: Implications for users,

Funders, Developers and Maintainers, volume 34, pages 189-195. ASIS, 1997.

[Matas 2000] J. Matas, D. Koubaroulis, and J. Kittler. Colour Image Retrieval and Object Recognition Using

Multimodal Neighbourhood Signature. In D. Vernon, editor, Proceedings of the European

Conference on Computer Vision, LNCS volume 1842, pages 48-64, Berlin, German, June

2000. Springer.

[Pass 96] Greg Pass, Ramin Zabih, and Justin Miller. Comparing Images Using Color

Coherence Vectors. In MultiMedia, pages 65-73. ACM, 1996.

[Smeulders 2000] Arnold W. M. Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh

Jain. Content-Based Image Retrieval at the End of the Early Years. In Transactions On Pattern

Analysis And Machine Intelligence, volume 22 of 12, pages 1349-1380. IEEE, December

2000.

[Swain 91] M. J. Swain and D. H. Ballard. Color Indexing. International Journal of Computer Vision,

7(1):11-32, 1991.

[Vaughan 91] W. Vaughan. “Automated Picture referencing; Another look at Morelli” in Computers

and the History of Art, Vol. 2 (1991), no. 2, pp.7 -18. [Harwood Academic Publishers ISSN 1048 - 6798]