The University of Southampton
University of Southampton Institutional Repository

Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities

Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities
Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities
Assessment has been identified as one of the major challenges faced by Higher Education Institutions (Whitelock, et al, 2007). As a response to the challenge, in a project funded by the Joint Information Systems Committee (JISC) Open Mentor (OM) was developed as a learning support tool for tutors to help them reflect on the quality of feedback given to their students on assignments submitted electronically. Its development was based on the fundamental theory that there was convincing evidence of systematic connections between different types of tutor comments and the level of attainment in an assignment (Whitelock, et al 2004). OM analyses, filters, and classifies tutor comments through an algorithm based on Bale’s Interaction Process. As a result, tutor’s feedback comments are classified into four categories namely: Positive reactions, Teaching points, Questions and Negative reactions. The feedback provided is analysed against an ideal number of feedback comments that an assignment given a mark of a specific band should have. Reports are provided in OM to support tutors in the task of reflecting on their feedback structure, content and style. The JISC-funded Open Mentor technology transfer (OMtetra) project is continuing the work initiated by the Open University implementing OM at the University of Southampton and King’s College London. OMtetra aims at taking up OM and extending its use by developing the system further and ultimately offering better support to tutors and students in the assessment process. A group of tutors from the University of Southampton and Kings’ College are at present using OM in their teaching and assessment. In this paper, we explore potential improvements to OM in three aspects: user interface, technology implementation and analysis algorithm design. For the user experience aspect suggested additions to OM include the creation of a simple entry form where tutors may validate the results of the analysis of the feedback comments. In addition, enhancements to OM will facilitate uploading of students and modules information into the system. Presently, OM utilises a built-in database of users that needs to be maintained separately from institutional systems. Improvements for this system feature include a more flexible authentication module which would simplify the deployment of the system in new environments and thus promote uptake by a larger number of institutions. In order to reach this goal, the system will be migrated to an open source framework which provides out-of-the-box integration with various authentication systems. The last to improve is the analysis algorithm. Currently, OM classifies tutors’ comments into four categories by applying an underlying text matching algorithm. This method could be improved if tutors are allowed to confirm comments’ classification through the OM interface and a free-text classification algorithm. As the number of users grow, so will the algorithm and analysis process, making it more comprehensive and intelligent as the keywords used during analysis are dynamically expanded. OMtetra is an on-going project with a lot of potential. We believe that the outcomes from the development and trial implementations of OM will contribute highly to the area of assessment in higher education.
Whitelock, Denise M.
6db8e4f5-d9c4-4d40-a473-a8ee524627d3
Gilbert, Lester
a593729a-9941-4b0a-bb10-1be61673b741
Hatzipanagos, Stylianos
aad919cf-73ac-4ade-b153-5a534610251d
Watt, Stuart
47247352-2d92-410a-b92b-5cf2fda5082a
Zhang, Pei
a881c1a8-503d-4429-be9b-6895a69be4f0
Gillary, Paul
f2840398-fcfe-4059-a453-62b62a4777ad
Recio, Alejandra
d05c4e43-3399-466d-99e0-01403a04b467
Whitelock, Denise M.
6db8e4f5-d9c4-4d40-a473-a8ee524627d3
Gilbert, Lester
a593729a-9941-4b0a-bb10-1be61673b741
Hatzipanagos, Stylianos
aad919cf-73ac-4ade-b153-5a534610251d
Watt, Stuart
47247352-2d92-410a-b92b-5cf2fda5082a
Zhang, Pei
a881c1a8-503d-4429-be9b-6895a69be4f0
Gillary, Paul
f2840398-fcfe-4059-a453-62b62a4777ad
Recio, Alejandra
d05c4e43-3399-466d-99e0-01403a04b467

Whitelock, Denise M., Gilbert, Lester, Hatzipanagos, Stylianos, Watt, Stuart, Zhang, Pei, Gillary, Paul and Recio, Alejandra (2012) Addressing the Challenges of Assessment and Feedback in Higher Education: A collaborative effort across three UK Universities. INTED 2012, Valencia, Spain. (In Press)

Record type: Conference or Workshop Item (Paper)

Abstract

Assessment has been identified as one of the major challenges faced by Higher Education Institutions (Whitelock, et al, 2007). As a response to the challenge, in a project funded by the Joint Information Systems Committee (JISC) Open Mentor (OM) was developed as a learning support tool for tutors to help them reflect on the quality of feedback given to their students on assignments submitted electronically. Its development was based on the fundamental theory that there was convincing evidence of systematic connections between different types of tutor comments and the level of attainment in an assignment (Whitelock, et al 2004). OM analyses, filters, and classifies tutor comments through an algorithm based on Bale’s Interaction Process. As a result, tutor’s feedback comments are classified into four categories namely: Positive reactions, Teaching points, Questions and Negative reactions. The feedback provided is analysed against an ideal number of feedback comments that an assignment given a mark of a specific band should have. Reports are provided in OM to support tutors in the task of reflecting on their feedback structure, content and style. The JISC-funded Open Mentor technology transfer (OMtetra) project is continuing the work initiated by the Open University implementing OM at the University of Southampton and King’s College London. OMtetra aims at taking up OM and extending its use by developing the system further and ultimately offering better support to tutors and students in the assessment process. A group of tutors from the University of Southampton and Kings’ College are at present using OM in their teaching and assessment. In this paper, we explore potential improvements to OM in three aspects: user interface, technology implementation and analysis algorithm design. For the user experience aspect suggested additions to OM include the creation of a simple entry form where tutors may validate the results of the analysis of the feedback comments. In addition, enhancements to OM will facilitate uploading of students and modules information into the system. Presently, OM utilises a built-in database of users that needs to be maintained separately from institutional systems. Improvements for this system feature include a more flexible authentication module which would simplify the deployment of the system in new environments and thus promote uptake by a larger number of institutions. In order to reach this goal, the system will be migrated to an open source framework which provides out-of-the-box integration with various authentication systems. The last to improve is the analysis algorithm. Currently, OM classifies tutors’ comments into four categories by applying an underlying text matching algorithm. This method could be improved if tutors are allowed to confirm comments’ classification through the OM interface and a free-text classification algorithm. As the number of users grow, so will the algorithm and analysis process, making it more comprehensive and intelligent as the keywords used during analysis are dynamically expanded. OMtetra is an on-going project with a lot of potential. We believe that the outcomes from the development and trial implementations of OM will contribute highly to the area of assessment in higher education.

Text
INTED2012_FinalPaperOU-KCL-UoS.pdf - Version of Record
Download (69kB)

More information

Accepted/In Press date: March 2012
Additional Information: Event Dates: March 2012
Venue - Dates: INTED 2012, Valencia, Spain, 2012-03-01
Organisations: Electronic & Software Systems

Identifiers

Local EPrints ID: 273207
URI: http://eprints.soton.ac.uk/id/eprint/273207
PURE UUID: 80a3b862-8832-4f07-b13b-0a72e22cde91
ORCID for Alejandra Recio: ORCID iD orcid.org/0000-0003-2823-4573

Catalogue record

Date deposited: 13 Feb 2012 10:47
Last modified: 11 Dec 2021 04:27

Export record

Contributors

Author: Denise M. Whitelock
Author: Lester Gilbert
Author: Stylianos Hatzipanagos
Author: Stuart Watt
Author: Pei Zhang
Author: Paul Gillary
Author: Alejandra Recio ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×