The University of Southampton
University of Southampton Institutional Repository

Managing very-large distributed datasets

Managing very-large distributed datasets
Managing very-large distributed datasets
In this paper, we introduce a system for handling very large datasets, which need to be stored across multiple computing sites. Data distribution introduces complex management issues, particularly as computing sites may make use of different storage systems with different internal organizations. The motivation for our work is the ATLAS Experiment for the Large Hadron Collider (LHC) at CERN, where the authors are involved in developing the data management middleware. This middleware, called DQ2, is charged with shipping petabytes of data every month to research centers and universities worldwide and has achieved aggregate throughputs in excess of 1.5 Gbytes/sec over the wide-area network. We describe DQ2’s design and implementation, which builds upon previous work on distributed ?le systems, peer-to-peer systems and Data Grids. We discuss its fault tolerance and scalability properties and brie?y describe results from its daily usage for the ATLAS Experiment.
0302-9743
775-792
de Oliveira Branco, Miguel
4e506200-44aa-4ce1-a899-266aa029ab74
Zaluska, Ed
43f6a989-9542-497e-bc9d-fe20f03cad35
De Roure, David
02879140-3508-4db9-a7f4-d114421375da
de Oliveira Branco, Miguel
4e506200-44aa-4ce1-a899-266aa029ab74
Zaluska, Ed
43f6a989-9542-497e-bc9d-fe20f03cad35
De Roure, David
02879140-3508-4db9-a7f4-d114421375da

de Oliveira Branco, Miguel, Zaluska, Ed and De Roure, David (2008) Managing very-large distributed datasets. Lecture Notes in Computer Science, 5331, 775-792.

Record type: Article

Abstract

In this paper, we introduce a system for handling very large datasets, which need to be stored across multiple computing sites. Data distribution introduces complex management issues, particularly as computing sites may make use of different storage systems with different internal organizations. The motivation for our work is the ATLAS Experiment for the Large Hadron Collider (LHC) at CERN, where the authors are involved in developing the data management middleware. This middleware, called DQ2, is charged with shipping petabytes of data every month to research centers and universities worldwide and has achieved aggregate throughputs in excess of 1.5 Gbytes/sec over the wide-area network. We describe DQ2’s design and implementation, which builds upon previous work on distributed ?le systems, peer-to-peer systems and Data Grids. We discuss its fault tolerance and scalability properties and brie?y describe results from its daily usage for the ATLAS Experiment.

Text
gada08.pdf - Version of Record
Download (374kB)

More information

Published date: November 2008
Organisations: Web & Internet Science

Identifiers

Local EPrints ID: 267323
URI: http://eprints.soton.ac.uk/id/eprint/267323
ISSN: 0302-9743
PURE UUID: a14a1378-ba99-4682-8aec-3f21d4706b55
ORCID for David De Roure: ORCID iD orcid.org/0000-0001-9074-3016

Catalogue record

Date deposited: 03 May 2009 19:48
Last modified: 19 Jul 2019 22:18

Export record

Contributors

Author: Miguel de Oliveira Branco
Author: Ed Zaluska
Author: David De Roure ORCID iD

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×