|
|
Search Options | Help | Site Map | Cultivate Web Site |
|
||
| Home | Current Issue | Index of Back Issues |
| Issue 7 Home | Editorial | Features | Regular Columns | News & Events | Misc. | ||
By Matthew Addis, Paul Lewis and Kirk Martinez - June 2002
Matthew Addis, Paul Lewis, Kirk Martinez and other members of the ARTISTE consortium review its achievements in developing an image retrieval system based on metadata and content that explores and analyses thousands of images from major art galleries across Europe.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ARTISTE [1] is a European Commission-funded collaboration, investigating the use of integrated content and metadata-based image retrieval across disparate databases in several major art galleries across Europe. Collaborating galleries include the Louvre in Paris, the Victoria and Albert Museum in London, the Uffizi Gallery in Florence and the National Gallery in London.
Museums and galleries often have several digital collections ranging from public access images to specialised scientific images used for conservation purposes. Direct access from one gallery to another is currently uncommon for textual data and almost unheard of in terms of image-based search and retrieval. Cross-collection access is recognised as important, however, for example to compare the treatments and conditions of Europe's paintings, which form a core part of our cultural heritage.
A key aim of ARTISTE is to provide an image retrieval system that can provide integrated cross-collection searching. Whilst ARTISTE is primarily designed for inter-museum searching and as a building block for public access systems, it could equally be applied to museum intranets.
An article on ARTISTE in the first issue of Cultivate [2] presented the project objectives and technical approach. Now that ARTISTE is nearing completion, this article looks at how those objectives have been fulfilled and discusses future work to continue and build upon the achievements of the project.
The ARTISTE system currently holds over 60,000 images from four separate collections belonging to the Uffizi, C2RMF (restoration centre for French museums including the Louvre), National Gallery and Victoria and Albert Museum. Although these collections are stored in separate databases and all have their own unique schema for the metadata that describes their contents, ARTISTE makes it possible to search quickly and transparently as if they were a single entity.
ARTISTE has been well received by the user members within the consortium. Further feedback has been obtained after a scaled down version [3] was made available to the 70 members of the ARTISTE Interest User Group (AIUG) [4] as a publicly accessible dissemination system. The most notable features of ARTISTE include:
Users of the ARTISTE system search for images using a query wizard. The wizard prompts the user with self-explanatory and non-technical questions. The wizard permits forward and backward movements through the process of search-generation. In this way, users of ARTISTE can quickly and simply build up sophisticated queries without needing to understand the technical details about the algorithms being used and why.
Since ARTISTE is a distributed system of servers that are able to communicate with each other, a user starts the query process by selecting one or more image collections.
 |
| Figure 1: Selection of Image Collections |
The user then progresses to define the detailed aspects of the query, in which they might combine content-based retrieval with metadata searching. (More about these methods later). When users are happy with the query they submit it to the system. They can request to be notified by e-mail when the query is complete.
 |
| Figure 2: Final Step: Search Summary |
ARTISTE then distributes the query to the chosen collections and collates the results as they come back. While the query is executing, users are given constant updates on the progress of their query whilst waiting for the results to be returned.
 |
| Figure 3: Query Executing |
The results of a search are shown on successive pages of 'thumbnails' with either 9, 15 or 21 images per page. The use of thumbnails allows search results to be navigated quickly over the Internet.
 |
| Figure 4: Thumbnail results |
The images are ordered according to how closely they match the query. Clicking on 'more info' for a particular thumbnail retrieves the full size image.
 |
| Distance | 0.0 |
| Collection | VAM |
| ARTISTE Image ID | 12642 |
| Local Image Id | pcd8839390910005-008 |
| Figure 5: Full size image |
The results of all queries are stored in the system so users can go back to work they have done in the past.
 |
| Figure 6: Search history |
The results of previous queries can also be used as input to new queries to allow more refined searches to be made.
ARTISTE allows users to specify the type of image search they wish to perform. Some examples are shown below. Technical descriptions of some of the algorithms are presented later in this article.
This category of search uses the appropriate algorithm to find images that have a similar distribution of colour to the image submitted. The algorithm is automatically selected by ARTISTE, depending on whether the user is looking for colour or black and white images and also on whether a colour or black and white image is submitted.
If the user is looking for colour images and a colour image is submitted, then the algorithm selected will be 'Colour Coherence Vector' - CCV. Alternatively, if the user is either searching for colour images but submits a black and white image, or is searching for black and white images, regardless of whether a colour or black and white image is submitted, the algorithm used will be the 'Mono-Histogram'.
An example of a search for an image of similar colour is shown below. In this example, the query image is also contained within the database. It is therefore not surprisingly found in first place. The rest of the retrieved results contain areas of contiguous colour similar to that of the query image. This includes the background since the algorithm, unlike a human, has no way of determining what is the subject and what is a backdrop.
 |
| Figure 7: Similar colour search |
Typically a similar colour search would be used in combination with a metadata-based search. For example, if a metadata search for the word 'vase' was used in conjunction with this similarity search, only images of vases and containing a similar colour distribution would be retrieved.
This type of query uses an appropriate algorithm to find images that have a similar pattern to the image submitted. The algorithm used is the 'Pyramid Wavelet Transform' - PWT and matching is based only on the texture, i.e. repeating patterns, in the whole image. The example below shows the results obtained when searching for similar textures. In this case, the dataset contains a set of fabrics, and the similarity between the repeating pattern of the query fabric shows up clearly in the results.
 |
| Figure 8: Similar pattern search |
This type of search is also appropriate in a painting restoration context. Below is an example of a search for paintings with a similar layout of 'stretchers' (wooden planks) on the back of the painting.
 |
| Figure 9: Query image, stretchers |
 |
| Figure 10: Stretchers Results page |
The results from using this algorithm could be used as a measure of how many wooden planks are affecting the presence of damage such as cracks on the painting surface.
A query image may be a sub-image of an image within the database. The requirement is not only to identify from which parent image the query is derived but also to locate its position in the parent image. Some of the images in the collection are very large (up to 800 Mbytes) and also very high-resolution (20 pels/mm), demanding special purpose algorithms for effective handling. Since the query may have been recorded at a significantly different resolution from its parent and in a different state of restoration, or simply under different lighting conditions, robust algorithms are required. A multi-scale search technique based on colour coherence vectors has been developed and has given useful results.
If the user is looking for colour images and a colour image is submitted, then the algorithm selected will be 'Multi-Scalar Colour Coherence Vector' - MCCV. Alternatively, if the user is either searching for colour images but submits a black and white image, or is searching for black and white images, regardless of whether a colour or black and white image is submitted, the algorithm used will be the 'Multi-Scalar Mono-Histogram'.
The example query below shows the quality of results obtained. The matching is based only upon the general colour layout, where all the retrieved results contain areas of contiguous skin-like colour and areas of contiguous dark brown, similar to the colours in the query. To help the reader, we have indicated with a green rectangle the area in each result image that matches the query image. In this example it is actually the second result that contains the query image.
 |
| Figure 11: Sub-image results |
ARTISTE provides a colour-picking tool so that users can define one or more colours that they want to use as the basis of a search. The colours need not correspond to sizeable regions of an image since the algorithm used is based on how similar the colours in the image are compared to the colour selection.
The example below shows the results of a query where a particular shade of red has been defined using the colour picker tool (the interface to the tool is shown as the query).
 |
| Figure 12: Colour search results |
ARTISTE can attempt to retrieve images in a collection that match low quality monochrome query images, for example a facsimile of a painting that might be in the database. The retrieved images have a similar layout of dark and light pixels to the query image. An example of query by fax is shown below. An explanation of how the query image was analysed using PWT is given in the section Query by Fax.
 |
| Figure 13: Fax image results |
ARTISTE supports a wide variety of image analysis algorithms, some of which are described below. These form the basis of content-based retrieval. In some cases, the algorithms can be combined into composite queries and a normalised distance measure for each algorithm is used to determine the overall match of a result image to the query.
We have already looked at PWT, an algorithm that can be used to locate wooden planks, particularly at the back of paintings.
Another algorithm aims to detect the presence of cracks on a painting. X-ray images are used instead of conventional surface images, as they expose more clearly the structure of the cracks. These techniques are typically combined with a metadata-based search to limit the content-based search to a sub-class of the total image collection, for example images taken using x-rays or images of the backs of paintings.
In our system, each algorithm is applied to the images in the collection to generate a set of image content descriptors called feature vectors. A feature vector can be considered as a way of indexing an image to describe an aspect such as colour distribution or texture. The feature vectors are then integrated and stored with the text metadata for each image in the collection database. When a search needs to be made, the required algorithm (e.g. CCV) is run on the query image to create a query feature vector. This query feature vector is then compared with all the corresponding feature vectors for the images in the collection. The comparison of feature vectors results in a measure of distance between the query image and each image in the collection. The images in the collection are then returned to the user as a series of thumbnails in order of increasing distance.
Some of the individual algorithms are explained in more detail below.
The colour histogram-matching algorithm simply uses the frequency of occurrence of each colour of the histogram within the image. The more of a particular colour an image contains, the higher its frequency will be within the histogram.
The histogram is made with 64 bins, a compromise between speed (it takes longer to match more bins), and accuracy (the less bins the less discriminating the results would be - i.e. images which are less similar would have lower distances).
Before a histogram is used for colour matching it is normalised by the number of pixels in the image. This means that colour matching is not influenced by the size of the images that are being compared.
 |
| Figure 14: Histogram comparison |
Two examples of colour histograms are shown above. Note, in the second example, how the background is dominant in the image. Because the colours of the pots are spread fairly evenly over the bins, and the background is predominantly one colour and therefore one bin of the histogram, once the histogram is normalised the background dominates the vector. This is the main drawback of histogram matching: the background information is all included within the feature and it cannot be ignored.
In the first example, the background is more evenly distributed over bins (due to the shading) and the object groups into only a few bins (due to its flat colour distribution). Therefore the background does not dominate the histogram so much.
 |
 |
| Figure 15: | Figure 16: |
| Monochrome-man | Monochrome histogram |
The monochrome histogram-matching algorithm simply uses the frequency of occurrence of each level of brightness of the histogram within the image. The more of a particular brightness an image contains, the higher its frequency will be within the histogram. Colour images in the database are converted to monochrome for matching with this algorithm, by converting RGB values to monochrome. The reason a monochromatic histogram is required, is that the colour histogram is not discriminating enough for monochromatic images. A 64 bin colour histogram has only 4 bins dedicated to grey-scale values. This means that most grey-scale images would look similar in a colour histogram.
One problem with simple histogram matching is that no consideration is given to whether colour occurs in contiguous regions, i.e. large blocks, or is fragmented into many small areas. A CCV (Colour Coherence Vector) algorithm is used to address this problem. A coherent region of colours in an image is a region of colour that is larger than some threshold. The algorithm retrieves images which have similar distributions of coherent colours.
A histogram of 64 bins is generated for both coherent and incoherent colours and these are matched separately. Coherence and incoherence are arbitrarily defined as greater and less then 5% of the total image area, respectively. This means if a pixel is part of a region that is less than 5% of the total image area it is added to the incoherent histogram within the CCV. For example, the chessboard image below is 50% black and 50% white, arranged into 64 squares, where 32 are white and 32 are black. Each square constitutes a region, each of which is 1/64th of the total image area. This is approximately 1.5% of the image and hence is considered incoherent. If a pixel is part of an area that is greater than 5% of the total image area it is added to the coherent histogram within the CCV. For example, the second image contains the same amount of black and white (50% of each) as the first example, but this time the black and white areas are contiguous.
 |  |
| Figure 17: | Figure 18: |
| Incoherent regions | Coherent regions |
Finally, because the colour histogram, monochrome histogram and CCV algorithms only consider the characteristics of an image as a whole, they are not suitable for searches that look for a sub-image (query image) within larger images (images in the collection). To address this problem a Multi-Scalar version of CCV has been developed. As shown in the 'pyramid' graphic below, the algorithm divides the image into a number of tiles (e.g. regions divided by the white lines in the bottom level of the pyramid) for each of a number of resolutions (the three levels of the pyramid). Both the query image and the collection images are converted into such a pyramid structure, and then each of the tiles in the query image are compared against each of the features for the tiles in the database image using the CCV matching algorithm.
A similar method has been applied to the monochrome histogram-matching algorithm.
 |
| Figure 19: Pyramid graphic |
PWT allows retrieval of similar images based on the general texture distribution of the image. In this context, image texture refers to repeating patterns throughout the whole image.
The PWT decomposes an image based on a wavelet transform, which can be thought of as similar to a Fourier transform, which transforms the image domain into a frequency domain. The frequency components of the image are analysed and a number of descriptors generated which represent the amounts of a discrete number of frequencies in the image.
 |
| Figure 20: |
| Decomposition |
| in image domain |
 |
| Figure 21: |
| Decomposition in |
| frequency domain |
Images are resized to 512x512 to perform this decomposition, which yields 22 frequency descriptors for an image. This makes the matching very fast. The comparison is achieved using a standard Euclidean distance measure.
The query by fax is based upon a set of PWT measures of the image at various threshold levels of a monochrome instance of the image. A Query by Fax feature vector consists of 99 PWT features at various levels of threshold (between 1% black and 99% black) of the image.
 |
| Figure 22: Query by Fax: Database Images converted to 99 PWT levels on left, query image on right |
Matching is performed by a simple step process:
More detail on the algorithms can be found in the ARTISTE Interest User Group (AIUG) Newsletters [6]; in help pages accompanying the public demonstrator of ARTISTE [7] ; and in papers prepared by the University of Southampton [8].
Linking is a familiar concept on the WWW. The traditional approach is to embed hard-coded links in an HTML page, which point to another Web resource that is associated to the original page in some way. However, this has several disadvantages: links have to be specifically authored for each document, the linking is inflexible, and links are difficult to maintain. These negative aspects are circumvented with dynamic linking.
In ARTISTE, instead of hard-coding the links, a separate link database is maintained and the links are applied dynamically at presentation time. Links are applied on a keyword basis. If the keyword 'teapot' has a link to further information about teapots, then this link will be applied every time that the word 'teapot' is displayed.
As well as being dynamic, linking can also be distributed because each ARTISTE site can maintain its own link database with links relevant to its own users. Thus if users from the National Gallery accesses the ARTISTE system via their web site they will be presented with different links than those for users from the Uffizi.
Dynamic linking has several advantages over conventional hard-coded static links:
An example of dynamic linking in ARTISTE is shown below. This shows the result of an ARTISTE query (background) with links that have been dynamically added on the word 'teapot' so that the user can navigate quickly and easily to an on-line shop (foreground) that sells similar items.
 |
| Figure 23: Dynamic linking |
The facilitation of cross-collection access to digital image information has been identified above as an objective of ARTISTE. This means not only allowing seamless searching across the collections of the institutions participating in ARTISTE but also achieving interoperability between those collections and other digital library resources. To that end ARTISTE makes use of existing open metadata standards such as Dublin Core and RDF Schema, while also supporting the Open Archive Initiative (OAI) information retrieval standard for distributed access.
The goal of the OAI harvesting protocol is to supply and promote an application-independent interoperability framework that can be used by a variety of communities engaged in publishing content on the Web. ARTISTE is an OAI data provider and has implemented support for the Open Archives Initiative Protocol for Metadata Harvesting, thus providing open access to metadata stored with each museum and gallery collection. OAI service providers can use metadata harvested via the OAI protocol as a basis for building value-added services.
ARTISTE is also participating in an initiative to redesign the primary open standard for interoperability between digital libraries, z39.50, using web technologies such as XML and SOAP. The z39.50 into the Next Generation (ZING) initiative [5] has proposed a Search and Retrieve Web Service (SRW) based on the z39.50 protocol for searching databases that contain metadata and objects. ARTISTE is one of the early implementers of SRW and has devised a service which enables distributed image content and metadata-based searches over the ARTISTE collections. Having emerged from the digital library community z39.50 has been traditionally concerned with text based searching and ARTISTE has been working with ZING to incorporate into the SRW protocol the ability to deal with content-based searching and thus expand international standards of information retrieval.
The museum community requires more sophisticated 3D models and other multimedia objects to represent fully the artefacts in their collections. Out of the ARTISTE project, which has developed a 2D image retrieval system, a new project consortium has been convened to develop both the technology and the expertise to help create, manage and present cultural archives of 3D models and associated multimedia objects.
SCULPTEUR [9], again supported by the European Commission, will exploit semantic web technology.
The project objectives are to:
ARTISTE has developed a successful image retrieval system based on metadata and content capable of exploring and analysing thousands of images from major art galleries across Europe. The project has seamlessly translated local metadata schemas to common standards so that the individual collections are searched as if they were a single entity. Content analysis algorithms are now in place that can handle many different types of query, appropriate to the diverse needs of the museum community. ARTISTE is contributing to the development of open standards to enable interoperability between museum and gallery collections worldwide.
By enhancing facilities for multimedia information organisation, storage and retrieval, ARTISTE has gone a long way towards meeting the increasing need for intelligent information extraction and presentation from distributed resources.
A current constraint on the uptake of multimedia digital libraries is the limited amount of structured metadata available in such systems. However, there exists a large amount of relevant information on the Web, and with the emerging semantic web approach to information structuring there are many new and exciting possibilities for enriching multimedia information collections through information exchange with other repositories.
Matthew Addis
IT Innovation Centre
2 Venture Road
Chilworth Science Park
Southampton SO16 7NP
United Kingdom
URL: <http://www.it-innovation.soton.ac.uk/
Email: mja@it-innovation.soton.ac.uk
Matthew Addis is a leading researcher at the IT Innovation Centre, Southampton, UK, which is a partner organisation in the ARTISTE project.
Paul Lewis
Department of Electronics and Computer Science
University of Southampton
Southampton SO17 1BJ
United Kingdom
URL: <http://www.ecs.soton.ac.uk/~phl/
Email: phl@ecs.soton.ac.uk
Kirk Martinez
Department of Electronics and Computer Science
University of Southampton
Southampton SO17 1BJ
United Kingdom
URL: <http://www.ecs.soton.ac.uk/~km/
Email: km@ecs.soton.ac.uk
Paul Lewis and Kirk Martinez lead the team at the University of Southampton where the algorithms have been developed for the project.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
For citation purposes:
Addis, M., Lewis, P., Martinez, K. "ARTISTE image retrieval system puts European galleries in the picture", Cultivate Interactive, issue
7, 11 July 2002
URL: <http://www.cultivate-int.org/issue7/artiste/>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Related articles:
If you would like to view similar articles to this one click on a key word below:
< - ARTISTE - images - metadata - retrieval - digital library - Dublin Core - rdf - >
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Translation:
If you wish to translate this page choose the preferred language using the
WorldLingo icon below.
Cultivate Interactive is not responsible for the outcome of this translation software. For further information see a previous article on Machine Translation.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feedback:
If you wish to read or submit comments on this article we now provide
access to the Crit annotation service.
Before using this service you should read the guidelines.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
|
|
Use the Crit service