Its About Time: Link Streams as Continuous Metadata

• Hypertext systems have grown to encompass a rich multimedia experience
  In addition to hyperlinked text and still images we now use many different media types
• We increasingly include streamed media, such as audio and video, in hypermedia applications
  This media usually has a temporal dimension - it exists and changes over a period of time
• We have developed the idea of continuous metadata to be used alongside and enrich these media streams
  We'll consider why we might need continuous metadata, where we might use it, how we might deliver it to the user, and the metadata which might be transferred
• We examined the use of continuous metadata as a linkservice delivery infrastructure
  And introduce some proof of concept implementations which demonstrate this

• Streaming media will usually take the form of:
  • pre-stored media
  Its already been recorded and archived somewhere, when the user wants to access it we serve it as a stream
  • live broadcast media
  The content is streamed as its produced (there may be a small editing / safety delay, as with traditional broadcasting) straight to the user
  • live interactive media
  Such as videoconferencing - each user might produce their own stream, and there isn't control by one broadcaster (this is the hardest to deal with!)
• The transmission of the stream can be:
  • one-to-one
  or point to point - in networking terms, unicast. The stream travels directly from the source (server) to the user (client).
  • one-to-many
  or multicast. One stream will be (literally) broadcast to many users. Of course, if there are many sources, we can have groups communicating with other groups. What's the benefit? This can be supported at the network level.

1. Explicit support for temporal (streamed) media
   We need to successfully deliver streamed media - this puts extra requirements on the protocol stack, from the network up to the application
2. The ability to specify and reserve a required quality of service (QoS)
3. Synchronisation within and between multimedia elements
   The presentation has to now co-ordinate elements in a temporal manner
4. Presentation and communication between groups of collaborating users
   i.e. multicast

• Real-time networking protocols enable timely delivery and control of streamed media (RTP, RTCP, RTSP)
  The Real-time Transport Protocol (RTP) and associated Real-time Control Protocol (RTCP) enable us to send and receive packets of data over the network in a temporally relevant manner. The Real-time Streaming Protocol (RTSP) (not really a protocol) allows us to co-ordinate these data flows (start, stop etc.)

• Real-time networking protocols enable timely delivery and control of streamed media (RTP, RTCP, RTSP)
• QoS architectures such as Integrated Services and Differentiated Services
  IntServ attempts to set up a guaranteed bandwidth route across the network - hard to roll out across a network. DiffServ has comparative levels of service - it groups usage into different levels of priority to provide better-than-best-effort. Both can be incorporated easily into IPv6 (Internet Protocol Version 6) network, so eventually this will be provided throughout the Internet.
• Multicast routing
  Currently only available as a bolt-on to the Internet via the mbone, multicast routing, allowing group to group communication, is a standard feature of IPv6

• Real-time networking protocols enable timely delivery and control of streamed media (RTP, RTCP, RTSP)
• QoS architectures such as Integrated Services and Differentiated Services
• Multicast routing
• Application presentation of interacting temporal multimedia elements (SMIL etc.)
  From Hypermedia Systems through the Amsterdam Hypermedia model to SMIL and SMIL2, not to mention the research into the technicalities of synchronising communications in the multimedia community.

• Hypertext systems augment documents with metadata such as context and linking information
• These systems can include temporal media elements within presentations
• We can (or will be able to) deliver and control streamed temporal media over networks to such systems which are distributed
  As we see from the previous slide

• Hypertext systems augment documents with metadata such as context and linking information
• These systems can include temporal media elements within presentations
• We can (or will be able to) deliver and control streamed temporal media over networks to such systems which are distributed
• But what about metadata that augments the streamed temporal media?
  Is the hyperstructure also temporal? Should it also be streamed? How do we deliver and present it?
  Quick answer: Yes

• Could be viewing a recording of the presentation, which has been augmented with metadata
• Could be viewing the presentation live, from another location
• Cameras throughout the room record video and sound
  Might also record metadata about camera positions and suchlike
• The current slide might be linked to at the relevant point
  The user might or might not decide to view them, but obviously you'll get a better view from the originals than compressed video!

• Pervasive "smart" devices could link to information about audience members as they interact with the speaker
  So when someone asks a question you'll see details about them, a link to their homepage, whether they know what they're talking about!
• Automated analysis and querying of linkbases
  The user might choose which linkbase or context based on personal interest; the presenter might suggest one
• Annotation, by local and remote audience members, at the time and in retrospect, available to all users

A dream come true?!

• Stored Mediadata: Little justification for streaming except due to sheer volume
  Stored multimedia is a persistent entity. It can be described by associated metadata in the same way as any document, but with a temporal element. Metadata might be used to find, deliver and navigate the material, but is likely to be small in comparison to the original material.
• Live Mediadata: Metadata is not available for pre-fetching
  It will become available as the multimedia stream is generated. Metadata might be about camera angles, or from real-time processing of the stream. A delay might be acceptably introduced to allow a pipeline of processing.
  Similarly, if a first user starts to view a long presentation, their annotations might not be available to a second user who starts to view the same presentation before the first user has finished.
  In some cases it might be necessary to stream pre-existing metadata because a receive-only device might join a live broadcast at an arbitrary point in time.
• Live Mediadata with multiple users: content metadata is created on-the-fly with little time for pre-processing
  Video-conferencing imposes tight time constraints on this style of synchronous interaction. Although session and party metadata might be available in advance, the large quantities of possible content metadata will not be.

• Metadata is data about data; we will think of it in this generic sense
  There are lots of useful standards to encode metadata: MPEG7 for multimedia data, XLink links, RDF vocabularies... we'll use the one which is suitable
• Metadata could be multiplexed with the mediadata and transported as a single stream, but we will carry it as a separate, independent flow
  If you're only going to carry limited descriptive metadata about a particular piece of media, you might want to combine them (MPEG/MPEG7).
  We believe there is good reason to keep the metadata as a distinct flow, following the open hypermedia convention of separation of links from content. Links are a form of metadata, and as with open hypermedia we can create a much more flexible framework of distributed delivery, processing and presentation. Metadata can also originate and be used in different places to the mediadata - e.g. video via satellite, metadata over a IP/modem connection.
• Metadata may be unicast or multicast
  As might the mediadata. But it might be that not every party needs to receive identical metadata flows; some nodes might not need the mediadata at all; so it's again useful to have separate metadata flows
• The metadata is continuous, not streamed
  We don't call it streamed metadata because this incorrectly implies that the metadata flow will be a non-stop stream of bits - relatively high bandwidth data. Continuous metadata need not be high volume, and there may be significant lulls between bursts of data (even if the transporting connection is kept open). What is important is that...
• The continuous metadata has temporal significance; it matters when it is transmitted and received; it is handled in a continuous manner
  The type and content of the metadata are of lesser importance to us. We're happy to carry any payload!

• The Temporal Linking Service (TLS) delivers continuous linking information (metadata) to hypermedia clients
• Builds on work applying Open Hypermedia to audio with RTSP, in turn from the Microcosm Sound Viewer. Demonstrates the concepts of continuous metadata
  In the initial system the client and server were developed using the Java Media Framework, streaming the mediadata using RTP.
• The server retrieves metadata from its link database (linkbase) and delivers it to the client using an XML markup

• The client resolves links relevant to the mediadata through an associated web browser
• On-time delivery of metadata from the server to the client is preferred over delayed, but guaranteed, delivery
  This highlights a problem of dealing with continuous metadata - what happens if a delay is introduced? (Stall the whole system?) We need a protocol to enable each component in the system to determine the local transmission deadline of each item of temporal metadata; late metadata should be dropped before entering the metadata stream to increase the chances for other metadata.
• Uses the HTTP/1.1 derivative Temporal Linking Transfer Protocol to explore synchronisation issues between client and server
  A metadata flow is maintained for the duration of the connection over TCP. A client selects a flow through a descriptor (a URI) to receive the 'never-ending' linkbase
• The link data is augmented with timing constraints so the client can display them with temporal relevance
  This is achieved through use of the TLTP BIAS command, which is specified from client to server.

Demoing tomorrow!

• Initial TLTP prototype stores metadata as simple unidirectional links
• Second prototype stores and communicates links using the Fundamental Open Hypermedia Model (FOHM)
• TLTP was mapped onto performatives of the Southampton Framework for Agent Research (SoFAR).
  which supports the authoring of multi-agent systems for distributed environments
• A service and client agent are defined
  to represent the TLTP server and client

• The service agent requests temporal link data from an agent that confirms to FOHM
  The SoFAR framework is matchmaker between able TLS service agents and FOHM agents.
• Temporal information is stored within the FOHM model by extending items in the knowledge base with time LocSpecs
  In a non-live scenario we can perform a single query. In a live scenario a more demanding conversation ensues, with the FOHM client periodically checking for link data within the specified time range (as the knowledge base is continually expanding).

• We can broaden the concepts of continuous metadata, and the ideas introduced through our prototypes, into a framework for the delivery and processing of continuous metadata in a distributed environment

• As with the prototypes, we deal with mediadata and metadata.
  We characterise media and metadata:
  • one persons metadata is anther's media
    
    For instance a flow of MIDI information would be metadata for a raw audio mediadata flow in one case, in another there may be no audio stream and the MIDI might form a mediadata flow augmented by other metadata. The framework should be flexible enough to support both these cases, but clarification is required.
    we define the mediadata flow to be the one against which the timing of metadata flows are made
    Normally this will be the flow that carries the high volume multimedia (since it will have the greatest amount of associated data
  • meta-meta-data?
    a metadata flow may develop a derivative metadata flow about it
    derivative metadata is another metadata flow based on the original metadata
    albeit one with a more complex relationship with other metadata. But it's still just metadata!

• the point at which we consider the mediadata within the framework is the mediadata source.
  For simplicity, we initially presume that each mediadata flow is derived from a single source; with a more complex implementation there is no reason why a mediadata flow cannot enter the framework in a distributed manner.
• the mediadata flow should use a suitable transport that
  e.g. RTP for audio or video
  • ensures timely delivery
  • provides identity and timing information to the framework
    for presentation and synchronisation purposes, discussed later
• the metadata source may or may not be located at the same node as the mediadata source
  for a live news feed a provider might construct a metadata flow of relevant links at the same broadcast point as the mediadata; while viewing a video of a pre-recorded lecture a user may wish to receive metadata annotations from a source other than that of the original lecture.
  The metadata source must always output information in a temporally relevant manner, and to do so it may require a flow of mediadata from the appropriate source.
• the metadata flow must be real-time and reliable
  It can't be lossy in the RTP sense of dropping an odd packet or two. We can't interpolate lost metadata

• an identifying code
  When multiple flows are combined at a presentation point an identifier is needed to deal with packets from a particular flow in a consistent manner; the identifier should also allow derivation of the mediadata flow with which it must be synchronised.
• validity timestamps
  to bound when the metadata is true so it can be synchronised to the mediadata
• display hints
  Should the metadata be displayed for a period outside its validity
• standardised content type identifier
  though the content could be of any type (RDF etc.) as long as the relevant nodes understand it

• Filter nodes are distributed between metadata sources, presentation points and other filter notes
  They expand upon the idea of filter chains in Microcosm
  • each is likely to perform a specific processing task - complex functionality comes from chaining a number of nodes together
  • utilises separate metadata flows
  • will have the effect of adding or subtracting metadata at a presentation point
    To add data, the filter output flow can be synchronised with the original metadata flow at the presentation point. To remove, or truly filter, the metadata, the filter output should be the only flow accepted at the presentation point: the original flow must be dropped. To accommodate this, metadata flow identities must incorporate the notion of derivatives, such that the history of a flow can be traced back through filters to the original metadata source identity.
• Control is complicated by the delays filters introduce
  but it's pretty complicated anyway. Setup and control might be through a separate channel, maybe using RTSP

• Sources send to groups of presentation points or filters using multicast flows
  Most advantageous in a live broadcast or group presentation environment
• Filters only have to join a multicast group to receive the output of another filter or source
• This allows:
  • allows a much more intricate web of flows
  • increases scalability of filter nodes
  For example, in a live video broadcast a hypermedia server may identify relevant sections of the picture using image processing techniques. It would be an inefficient use of both the hypermedia server and network resources for the many clients who receive the video stream to query the server individually; using the framework the hypermedia server only processes the video once, then the results are multicast through a metadata flow to as many presentation points as requested.

• Explored the idea of continuous metadata used to enrich streamed multimedia
• Introduced proof of concept prototypes to demonstrate these ideas
• Generalised the concepts introduced into a framework for the distribution and processing of continuous metadata
• Coming soon: demo tomorrow