The University of Southampton
University of Southampton Institutional Repository

Fast test suite-driven model-based fault localisation with application to pinpointing defects in student programs

Fast test suite-driven model-based fault localisation with application to pinpointing defects in student programs
Fast test suite-driven model-based fault localisation with application to pinpointing defects in student programs

Fault localisation, i.e. the identification of program locations that cause errors, takes significant effort and cost. We describe a fast model-based fault localisation algorithm that, given a test suite, uses symbolic execution methods to fully automatically identify a small subset of program locations where genuine program repairs exist. Our algorithm iterates over failing test cases and collects locations where an assignment change can repair exhibited faulty behaviour. Our main contribution is an improved search through the test suite, reducing the effort for the symbolic execution of the models and leading to speed-ups of more than two orders of magnitude over the previously published implementation by Griesmayer et al. We implemented our algorithm for C programs, using the KLEE symbolic execution engine, and demonstrate its effectiveness on the Siemens TCAS variants. Its performance is in line with recent alternative model-based fault localisation techniques, but narrows the location set further without rejecting any genuine repair locations where faults can be fixed by changing a single assignment. We also show how our tool can be used in an educational context to improve self-guided learning and accelerate assessment. We apply our algorithm to a large selection of actual student coursework submissions, providing precise localisation within a sub-second response time. We show this using small test suites, already provided in the coursework management system, and on expanded test suites, demonstrating the scalability of our approach. We also show compliance with test suites does not reliably grade a class of “almost-correct” submissions, which our tool highlights, as being close to the correct answer. Finally, we show an extension to our tool that extends our fast localisation results to a selection of student submissions that contain two faults.

Automated assessment, Automated debugging, Model-based fault localisation, Symbolic execution
1619-1366
445-471
Birch, Geoff
4e118f9f-4a3a-4f28-893d-57423b360c16
Fischer, Bernd
0c9575e6-d099-47f1-b3a2-2dbc93c53d18
Poppleton, Michael
4c60e63f-188c-4636-98b9-de8a42789b1b
Birch, Geoff
4e118f9f-4a3a-4f28-893d-57423b360c16
Fischer, Bernd
0c9575e6-d099-47f1-b3a2-2dbc93c53d18
Poppleton, Michael
4c60e63f-188c-4636-98b9-de8a42789b1b

Birch, Geoff, Fischer, Bernd and Poppleton, Michael (2019) Fast test suite-driven model-based fault localisation with application to pinpointing defects in student programs. Software and Systems Modeling, 18 (1), 445-471. (doi:10.1007/s10270-017-0612-y).

Record type: Article

Abstract

Fault localisation, i.e. the identification of program locations that cause errors, takes significant effort and cost. We describe a fast model-based fault localisation algorithm that, given a test suite, uses symbolic execution methods to fully automatically identify a small subset of program locations where genuine program repairs exist. Our algorithm iterates over failing test cases and collects locations where an assignment change can repair exhibited faulty behaviour. Our main contribution is an improved search through the test suite, reducing the effort for the symbolic execution of the models and leading to speed-ups of more than two orders of magnitude over the previously published implementation by Griesmayer et al. We implemented our algorithm for C programs, using the KLEE symbolic execution engine, and demonstrate its effectiveness on the Siemens TCAS variants. Its performance is in line with recent alternative model-based fault localisation techniques, but narrows the location set further without rejecting any genuine repair locations where faults can be fixed by changing a single assignment. We also show how our tool can be used in an educational context to improve self-guided learning and accelerate assessment. We apply our algorithm to a large selection of actual student coursework submissions, providing precise localisation within a sub-second response time. We show this using small test suites, already provided in the coursework management system, and on expanded test suites, demonstrating the scalability of our approach. We also show compliance with test suites does not reliably grade a class of “almost-correct” submissions, which our tool highlights, as being close to the correct answer. Finally, we show an extension to our tool that extends our fast localisation results to a selection of student submissions that contain two faults.

Text
Birch2019_Article_FastTestSuite-drivenModel-base - Version of Record
Available under License Creative Commons Attribution.
Download (1MB)

More information

Accepted/In Press date: 13 July 2017
e-pub ahead of print date: 27 July 2017
Published date: 8 February 2019
Keywords: Automated assessment, Automated debugging, Model-based fault localisation, Symbolic execution

Identifiers

Local EPrints ID: 428700
URI: https://eprints.soton.ac.uk/id/eprint/428700
ISSN: 1619-1366
PURE UUID: a49592d2-9bf2-483a-9a13-7ce541e2a4bd

Catalogue record

Date deposited: 07 Mar 2019 17:30
Last modified: 13 Mar 2019 17:30

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×