The University of Southampton
University of Southampton Institutional Repository

Towards a scalable, open standards service for cross-protocol data transfers across multiple sources an sinks

Towards a scalable, open standards service for cross-protocol data transfers across multiple sources an sinks
Towards a scalable, open standards service for cross-protocol data transfers across multiple sources an sinks
Data Transfer Service (DTS) is an open-source project that is developing a document-centric message model for describing a bulk data transfer activity, with an accompanying set of loosely coupled and platform-independent components for brokering the transfer of data between a wide range of (potentially incompatible) storage resources as scheduled, fault-tolerant batch jobs. The architecture scales from small embedded deployments on a single computer to large distributed deployments through an expandable ‘worker-node pool’ controlled through message-orientated middleware. Data access and transfer efficiency are maximized through the strategic placement of worker nodes at or between particular data sources/sinks. The design is inherently asynchronous, and, when third-party transfer is not available, it side-steps the bandwidth, concurrency and scalability limitations associated with buffering bytes directly through intermediary client applications. It aims to address geographical–topological deployment concerns by allowing service hosting to be either centralized (as part of a shared service) or confined to a single institution or domain. Established design patterns and open-source components are coupled with a proposal for a document-centric and open-standards-based messaging protocol. As part of the development of the message protocol, a bulk data copy activity document is proposed for the first time.
1364-503X
Meredith, David
52056084-4fcf-4b93-8b8b-23ff1e307667
Crouch, Stephen
a136ad57-82ec-4664-8d8e-79a605808e6d
Galang, Gerson
dfdece6f-4e34-491b-953a-2a1a242b957c
Jiang, Ming
0f3c0781-ba1f-456a-88b1-b39ec938264f
Hung, Nguyen
6b0cabb2-c7a9-4f80-b0f1-85df881b3bce
Turner, Peter
af5f357d-f9f5-42fc-a739-761653b4cb87
Meredith, David
52056084-4fcf-4b93-8b8b-23ff1e307667
Crouch, Stephen
a136ad57-82ec-4664-8d8e-79a605808e6d
Galang, Gerson
dfdece6f-4e34-491b-953a-2a1a242b957c
Jiang, Ming
0f3c0781-ba1f-456a-88b1-b39ec938264f
Hung, Nguyen
6b0cabb2-c7a9-4f80-b0f1-85df881b3bce
Turner, Peter
af5f357d-f9f5-42fc-a739-761653b4cb87

Meredith, David, Crouch, Stephen, Galang, Gerson, Jiang, Ming, Hung, Nguyen and Turner, Peter (2010) Towards a scalable, open standards service for cross-protocol data transfers across multiple sources an sinks. [in special issue: e-Science: past, present and future II] Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 368 (1926). (doi:10.1098/rsta.2010.0148).

Record type: Article

Abstract

Data Transfer Service (DTS) is an open-source project that is developing a document-centric message model for describing a bulk data transfer activity, with an accompanying set of loosely coupled and platform-independent components for brokering the transfer of data between a wide range of (potentially incompatible) storage resources as scheduled, fault-tolerant batch jobs. The architecture scales from small embedded deployments on a single computer to large distributed deployments through an expandable ‘worker-node pool’ controlled through message-orientated middleware. Data access and transfer efficiency are maximized through the strategic placement of worker nodes at or between particular data sources/sinks. The design is inherently asynchronous, and, when third-party transfer is not available, it side-steps the bandwidth, concurrency and scalability limitations associated with buffering bytes directly through intermediary client applications. It aims to address geographical–topological deployment concerns by allowing service hosting to be either centralized (as part of a shared service) or confined to a single institution or domain. Established design patterns and open-source components are coupled with a proposal for a document-centric and open-standards-based messaging protocol. As part of the development of the message protocol, a bulk data copy activity document is proposed for the first time.

This record has no associated files available for download.

More information

Published date: 10 September 2010
Organisations: Civil Maritime & Env. Eng & Sci Unit

Identifiers

Local EPrints ID: 339157
URI: http://eprints.soton.ac.uk/id/eprint/339157
ISSN: 1364-503X
PURE UUID: cb9d778a-3382-4904-ad92-6cc17d6284db
ORCID for Stephen Crouch: ORCID iD orcid.org/0000-0001-8985-6814

Catalogue record

Date deposited: 24 May 2012 13:27
Last modified: 15 Mar 2024 03:09

Export record

Altmetrics

Contributors

Author: David Meredith
Author: Stephen Crouch ORCID iD
Author: Gerson Galang
Author: Ming Jiang
Author: Nguyen Hung
Author: Peter Turner

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×