Enhanced image retrieval using spatial information and ontologies

Abstract

New approaches are essential to improve the inference of semantic relationships from low-level features for image annotation and retrieval. Current research on image annotation sometimes represents images in terms of regions and objects, but pays little attention to the spatial relationships between those regions or objects. Annotations are most frequently assigned at the global level, and even when assigned locally the extraction of relational descriptors is often neglected. To enrich the semantic description of the visual information, the use of spatial relationships offers one way to describe objects in an image more richly and often captures a relevant part of information in the image. In this thesis, new approaches for enhancing image annotation and retrieval by capturing spatial relationships between labelled objects in images are developed. Starting with an assumption of the availability of labelled objects, algorithms are developed for automatically extracting absolute object positional terms and relative terms describing the relative positions of objects in the image. Then, by using order of magnitude height information for objects in the domain of interest, relative distance of objects from the camera position in the real world are extracted, together with statements about nearness of objects to each other in the image and nearness in the real world. A knowledge-based representation is constructed using spatial and domain specific ontologies, and the system stores the asserted spatial statements about the images, which may then be used for image retrieval. The resulting Spatial Semantic Image System is evaluated using precision, recall and F-scores to test retrieval performance, and a small user trial is employed to compare the system’s spatial assertions with those made by users. The approach is shown to be capable of handling effectively a wide range of queries requiring spatial information and the assertions made by the system are shown to be broadly in line with human perceptions.

More information

Identifiers

ORCID for Mark J. Weal: orcid.org/0000-0001-6251-8786

Catalogue record

Export record