Semiotics and Agents for Integrating and Navigating Through Media Representations of Concepts

Abstract

The purpose of this paper is two-fold. We begin by exploring the emerging trend to view multimedia information in terms of low-level and high-level components; the former being feature-based and the latter the "semantics" intrinsic to what is portrayed by the media object. Traditionally, this has been viewed by employing analogies with generative linguistics (e.g. compositional semantics). Recently, a new perspective based on the semiotic tradition has been alluded to in several papers. We believe this to be a more appropriate approach. From this, we propose an approach for tackling this problem which uses an associative data structure expressing authored information together with intelligent agents acting autonomously over this structure. We then show how neural networks can be used to implement such agents. The agents act as "vehicles" for bridging the gap between multimedia semantics and concrete expressions of high-level knowledge, but we suggest that traditional neural network techniques for classification are not architecturally adequate.

More information

Identifiers

ORCID for Wendy Hall: orcid.org/0000-0003-4327-7811

Catalogue record

Export record