On-line assessment

‘If lower-order learning is an unintended educational consequence of on-line assessment, then any perceived or real gains made in efficiency, staff workload reduction and/or cost savings are at a questionable price.’

Why consider on-line assessment?

A good deal of investigation and development is underway in Australian universities into the possibilities for effective and efficient on-line and computer-based assessment. The current commercial ‘virtual learning environments’, which integrate various curriculum elements at subject level into a single software portal, usually offer various built-in options for student assessment. As well, many on-line assessment initiatives are being locally developed to suit specific curriculum needs.

There are many reasons why on-line assessment is being adopted by Australian universities. Many academics are seeking to diversify assessment tasks, broaden the range of skills assessed and provide students with more timely and informative feedback on their progress. Others are wishing to meet student expectations for more flexible delivery and to generate efficiencies in assessment that can ease academic staff workloads. All staff involved in such initiatives are discovering they face a large number of technical and educational decisions.

The move to on-line and computer based assessment is a natural outcome of the increasing use of information and communication technologies to enhance learning. As more students seek flexibility in their courses, it seems inevitable there will be growing expectations for flexible assessment as well.

At the same time, in a climate of increasing academic workloads, the adoption of on-line assessment may help to manage large volumes of marking and assessment-related administration efficiently. The automation of routine on-line tasks, in particular, may have the potential in the long-term to provide time/cost-efficient student assessment, though the present evidence suggests that some on-line assessment, at least in the early stages, can add significantly both to staff workload and to overall expenses.

Is on-line assessment improving the assessment of student learning?

The major educational question to be answering is whether on-line assessment is having any influence on the quality of teaching and learning. Such a broad judgement is difficult to make, but there are growing educational concerns with current developments. While ICTs have certainly opened up significant possibilities for transforming the role and practice of assessment in higher education, the full possibilities are probably yet to be realised. There is some evidence that on-line assessment, unless carefully planned, can encourage students to focus on lower level cognitive skills. The educational effectiveness of on-line assessment that concentrates primarily or exclusively on true/false or multiple choice responses, for example, is highly questionable in a higher education environment . As is widely known, such approaches to assessment can have direct negative effects on student approaches to learning by encouraging narrow reproduction rather than the development of higher order cognition that involves, for example, critical evaluation.

If lower-order learning is an unintended educational consequence of on-line assessment, then any perceived or real gains made in efficiency, staff workload reduction and/or cost savings are at a questionable price.

Realising the educational benefits of on-line assessment

This section provides a framework for making sound educational decisions about the use of on-line assessment. As would be expected, this framework is congruent with the basic principles for effective assessment in any format.

Assessment plays a central role in student learning whatever the mode. On-line learning can challenge students to learn new skills and ways of studying and learning:

On-line assessment is but one of many modes that may be used to indicate to students which aspects of their learning are valued and will be rewarded. The use of on-line tools to assess learner progress toward subject objectives can take many forms including:

Reactions to on-line assessment

There is a range of perceptions among academic staff about the worth of on-line learning:

The broad choice of types of on-line assessment available generate a wide range of reactions from students:

One question asked frequently by students in relation to on-line assessment is, beyond flexibility, ‘What is the point?’ Where assessment tasks appear to add little or nothing to student learning, or worse, to hinder student learning, this is a reasonable question.

In the case of on-line examinations, the point, from a staff perspective might be to save time and effort. However, the experience of staff working in this area indicates that the design of on-line examinations can rapidly become far more complex and time-consuming than preparing conventional paper and pen examinations. This is in part because computers offer the potential to present students with more complex scenarios through the use of interactive resources (images, sound, simulation) than does print. If the potential for the use of interactive resources has not been realised and the verification protection is set up so that there is little difference between taking an exam in paper-and-pencil format and taking it on-line, it may be difficult for students to see the point in on-line examinations.

34 strategies for developing effective on-line assessment

Together these thirty-four strategies can be summarised into three checklists:

Access and usage checklist

Quality of teaching and learning checklist

For on-line examinations in particular:

For evaluation in particular:

Technical and administrative checklist

For on-line tests and exams in particular:

Where to start?

There are at least three useful guiding principles when starting to use on-line assessment. These are:

Start with clear educational objectives

Begin by considering how you would like to influence student learning then consider which technologies, if any, are appropriate to influence it in these ways. In particular, it might be useful to consider the following question:

When making the decision about whether or not, and in which way(s) to use on-line assessment, it is also essential to take into account:

Table 1 summarises some examples of how subject objectives, modes of on-line assessment and learner characteristics intersect. Academic staff may need to consider learner access to and competence with technology for each objective stated in the table as it is likely to be relevant to each objective. Where students from educational backgrounds that may not have included learning experiences involving email, listservs and web forums form part of the student cohort, it is necessary to equip these students with the skills necessary to undertake assessment tasks that require the use of these formats.

Table 1: Objectives, modes and learner characteristics of on-line learning

If the goal or purpose is to:

NB. Table adapted from Leask (1999) and the University of South Australia, 2001.

Start small

The basic advice here is, initially, aim for quality rather than quantity.

A complete overhaul of the entire curriculum may not be the best place to start with introducing on-line assessment. Many academic staff have successfully started with an on-line assessment task that is a relatively minor proportion of the assessment for a subject. That way, any technical, educational or other difficulties that might arise can be resolved without the risk of seriously disadvantaging students.

If might also be useful to start with formative rather than summative assessment on-line. Any efforts made towards this end will be useful in themselves in providing feedback to students and therefore assisting learning, as well as in providing a ‘trial run’ for the more ambitious objective of putting summative assessment on-line.

Start where success is most likely

Successful on-line assessment is most likely if that assessment is aligned with teaching and learning objectives. In other words, there should be a strong relationship between the purpose of on-line assessment task(s) and the intended outcomes of the subject. It is wise to design assessment tasks that require the students to integrate the material they have learned in the subject with their own interpretations of that material. The on-line assessment should also allow students to communicate their understandings and allow the provision of feedback to students on their efforts to these ends.

As well, academics involved in developing on-line assessment share numerous objectives, which are useful for guiding such development including:

These objectives illustrate how well on-line assessment suits formative assessment objectives. They also show how developments in on-line assessment are taking place in the context of efforts to build a richer overall teaching and learning environment — an example of the way in which information and communication technologies can be major catalysts for academics to revisit fundamental educative principles.

References

Leask, B. (1999). ‘Issues in on-line delivery: quizzes and discussion groups’, University of South Australia. www.unisanet.unisa.edu.au/flc/pdonline/Discussions_and_quizzes_materials.doc. Accessed 10/12/01

Morgan, C. and O’Reilly, M. (1999). Assessing Open and Distance Learners. Kogan Page, London.

University of South Australia, (2001). www.unisanet.unisa.edu.au/flc/pdonline/IssuesinOnlineAssessment.doc. Accessed 10/12/01