Fast, fully-automated, model-based fault localisation and repair with test suites as specification

Abstract

Fault localisation, i.e. the identification of program locations that cause errors, takes significant effort and cost. We describe a fast model-based fault localisation algorithm which, given a test suite, uses symbolic execution methods to fully automatically identify a small subset of program locations where genuine program repairs exist. Our algorithm iterates over failing test cases and collects locations where an assignment change can repair exhibited faulty behaviour. Our main contribution is an improved search through the test suite, reducing the effort for the symbolic execution of the models and leading to speed-ups of more than two orders of magnitude over previously published work.

We implemented our algorithm for C programs, using the KLEE symbolic execution engine, and demonstrate its effectiveness on the Siemens TCAS variants. Its performance is in line with recent alternative model-based fault localisation techniques, but narrows the location set further without rejecting any genuine repair locations. We explore extending the low-quality repairs constructed by the localisation process to synthesise a high-quality repair.

We also show how our tool can be used in an educational context to improve selfguided learning and accelerate assessment. We apply our algorithm to a large selection of actual student coursework submissions, providing precise localisation within a subsecond response time. We show this using small test suites, already provided in the coursework management system, and on expanded test suites, demonstrating scaling. We also show that compliance with test suites does not predictably score a class of "almost correct" submissions, which our tool highlights. Finally, we show an extension to our tool which enables a selection of student submissions to be localised beyond programs that conform to a single-fault assumption.

More information

Identifiers

Catalogue record

Export record