Developing Formative Assessments for Postgraduate Students in Engineering

The MSc Road Management and Engineering programme at the University of Birmingham, as in many taught engineering programmes, is facing a challenge to increase student numbers whilst at the same time reduce student-staff contact time.  To address these issues and at the same time maintain the quality of the programme, it was felt necessary to support lectures and tutorials delivered using traditional methods with on-line resources that would encourage self-learning.  To this end the research sought to develop computer based assessments (CBA) which would motivate the students to deeper and further learning.  The challenge was to create an appropriate mixture of questions that address different cognitive levels, are interesting and challenging to engender enthusiasm, and provide appropriate feedback to the answers given that may stimulate further learning.

To facilitate the development process, a formal taxonomy was used to aid question design and to ensure that the higher learning levels of problem-solving were addressed in the CBA developed.  The taxonomy used was the RECAP model devised by Imrie (1995) which is a modified version of Bloom’s taxonomy (Bloom et al., 1956).

In order to develop assessments that would be able to drive student learning in the way envisaged, three pieces of software were considered: WebCT, Question Mark Perception and TRIADS (Tripartite Interactive Assessment Delivery System) (Mackenzie, 1999),

To select the optimal tool an evaluation was made of the relative merits of the three types of software using a number of criteria.  As a result, the TRIADS software was chosen as the most appropriate as it offers the greatest number of question styles and it was felt that it would allow questions to be developed that were challenging and interesting and which would address all levels of learning.

The CBA development concerned the Pavement Engineering and Road Asset Management modules of the MSc programme which had a large resource of paper-based questions, mainly in the form of tutorials and past examination questions.  These materials provided the basis for the CBA questions and currently, seventeen different TRIADS question types have been used to author 37 questions.  To allow the students to access the resource wherever and whenever they wished, the assessments were made available from a Virtual Learning Environment (VLE) hosted on a web server and on CD-ROM.

Two purposes of the formative assessments were to encourage the students to gain as deep as possible understanding of the subject and to motivate them to self-learn.  To facilitate this it was desired to provide feedback to the students based on the answers given.  To achieve this, the existing TRIADS templates were modified to incorporate additional feedback components.  These modifications enable the tutor to configure a question so that the following types of feedback may be given to the student: the score achieved, an indication of the parts of the questions answered correctly or incorrectly, the correct answers to the questions, a model answer and a list of references which the student may study either to improve their understanding or broaden their knowledge of the subject area.  For multiple choice questions the content of the feedback depends on the answer given, whilst for all other question types it is specific to the score achieved.

In the current configuration of the assessment, after answering a question, the students are provided with a score and an indication of the parts of the question they answered correctly or incorrectly.  In addition they may be given a model answer if their score is above a threshold value set for the question.  This threshold depends on the level of learning being assessed.  Additionally, references are provided to other resources such as lecture notes or texts.  If after having studied the references the students are still unable to answer a question satisfactorily, they are encouraged to discuss the question with their peers or the lecturer.  For some of the more challenging questions a discussion forum is provided, via a VLE, to enable the students and staff to engage in a dialogue.

To date the CBAs have been used by four consecutive years of the MSc programme and in order to assess qualitatively whether the goals of introducing the assessments were being met and to enable the assessments to be improved for future years, students are asked to complete a questionnaire.  The questionnaire consisted of a number of open questions to enable the students to describe in detail their perceptions of the assessments in terms of added value to their learning experience.  To enable the authors to differentiate among the range of responses several categories of response (known as ordinal categories) were selected into which all the open-ended answers were sorted.  Following this analysis, a number of observations were made: nearly all of the students enjoyed doing the assessments and found them easy to use and visually appealing, all of the students considered it beneficial to be able to access the assessments in their own time.  In addition the assessments were considered to be useful for revision and helpful in consolidating learning.  Three-quarters of the students found the feedback components useful in furthering their understanding of the subject.

All in all the introduction of CBA as part of the syllabus appears to be promising. TRIADS, the software selected, fulfilled the expectations of both the staff and the students.  As far as the former are concerned TRIADS is a tool to support their teaching.  More importantly, as far as the latter are concerned, it is a tool that not only increased but also motivated further learning.  In addition, the research indicates that there is strong evidence to suggest the usefulness of the CBA approach.  However, if such an approach is to be introduced and used systematically in any programme of studies it is necessary to make an assessment as to whether the perceived benefits of reducing student-staff contact time and improving student learning outweigh the costs of providing a CBA in terms of staff resources, hardware and software.

Funding for the initial research was provided by the Learning Development Unit at the University of Birmingham.  A grant provided by the Engineering and Physical Sciences Research Council (EPSRC) facilitated further the implementation of the work under the Road Management Masters Training Programme.  The authors are grateful also to the MSc students who trialled the assessments.

BLOOM, B.S., ENGELHART, M.D., FURST, E.J., HILL, W.H. and KRATWOHL, D.R., 1956 Taxonomy Of Educational Objectives, Handbook I: Cognitive Domain.  (New York: David Mackay).

IMRIE, B. W., 1995, Assessment for Learning: quality and taxonomies.  Assessment and Evaluation in Higher Education. Vol. 20, No. 2. pp 175 - 189.

MACKENZIE, D., 1999, Recent developments in the tripartite interactive assessment delivery system (TRIADS).  Proceedings of the Third International Conference on CAA, Loughborough UK.

: Back to Fall 2005 Issue Vol. 1, No. 2

: Back to List of Issues

: Back to Table of Contents

Created on 27 Jan 2015, Last modified on 27 Jan 2015