Managing very large distributed datasets on a data grid

de Oliveira Branco, Miguel, Zaluska, Ed, De Roure, David, Lassnig, Mario and Garonne, Vincent (2009) Managing very large distributed datasets on a data grid Concurrency and Computation Practice and Experience, 22, (11), pp. 1338-1364. (doi:10.1002/cpe.1489).


[img] PDF cpepaper_revised.pdf - Version of Record
Restricted to Repository staff only

Download (1MB)


In this work we address the management of very large datasets, which need to be stored and processed across many computing sites. The motivation for our work is the ATLAS Experiment for the Large Hadron Collider, where the authors have been involved in the development of the data management middleware. This middleware, called DQ2, has been used for the last several years by the ATLAS Experiment for shipping petabytes of data to research centres and universities worldwide. We describe our experience in developing and deploying DQ2 on the Worldwide LHC Computing Grid, a production Grid infrastructure formed of hundreds of computing sites. From this operational experience, we have identied an important degree of uncertainty that underlies the behaviour of large Grid infrastructures. This uncertainty is subjected to a detailed analysis, leading us to present novel modeling and simulation techniques for Data Grids. In addition, we discuss what we perceive as practical limits to the development of data distribution algorithms for Data Grids given the underlying infrastructure uncertainty, and propose future research directions.

Item Type: Article
Digital Object Identifier (DOI): doi:10.1002/cpe.1489
ISSNs: 1532-0626 (print)
Keywords: distributed systems, data management, grid computing, modelling, simulation
Organisations: Web & Internet Science
ePrint ID: 267753
Date :
Date Event
21 August 2009Published
Date Deposited: 04 Aug 2009 13:16
Last Modified: 17 Apr 2017 18:43
Further Information:Google Scholar

Actions (login required)

View Item View Item