Skip to content.
Sections
Personal tools
You are here: Home » Features » Exploring e-Assessment: A practitioner’s perspective

Exploring e-Assessment: A practitioner’s perspective

Helen Ashton
Last modified 17 Nov, 2008
Published 28 Oct, 2008
e-Assessment promises lecturers and tutors the opportunity to both streamline the assessment process and to allow for greater variety and creativity in testing students. We asked Helen Ashton, an active member of the JISC CETIS Assessment Special Interest Group and a lecturer specialising in the design and development of e-assessment systems, to explore e-assessment from a practitioner's perspective. In this article she looks at each stage of the e-assessment life cycle from what to assess to how to measure student performance.

In its broadest sense, e-assessment is the use of computers in assessment and may be applied to any use of computers in the setting, delivery, marking and reporting of assessments (including, for instance, personal response systems and e-portfolios). This article discusses one facet of e-assessment, namely the computer based test (e-testing) and in this article, e-assessment and e-testing are used interchangeably.

E-assessment is becoming more prevalent in Higher Education. Many readers will know someone who has ventured into this "brave new world". With the increased focus on Virtual Learning Environments (VLE) many practitioners have found they have access to an e-assessment system as part of their VLE, within which they can create assessments, make these available to their students, and review the results afterwards. E-assessment systems are also becoming more user-friendly and sophisticated, making it possible to include e-testing as another part of the assessment toolkit.

As systems become easier to use, and the pressures to embrace e-assessment increase, it is important to realise that these tools do not provide an instant panacea to resolve all assessment problems. Skill and expertise are required to understand how and when these systems can be used in an educationally effective manner.

1. Why use e-assessment?

From a practitioner viewpoint, the question "should e-assessment be used?" comes after consideration of the assessment within a course as a whole.

Often we talk about assessment as being formative or summative – is the purpose for assisting or for measuring student learning? In practice, assessments are often a bit of both, with one being of primary importance. How the resulting information from the assessment is to be used affects the way in which the assessment will be carried out as well as the way in which it will be approached by both students and teachers. For instance, if students are aware that an assessment is entirely formative in nature, they might choose to sit the assessment earlier in the course of their learning with the explicit purpose of using it to highlight areas of weakness where they will then focus their attention. A teacher using formative assessment would be less interested in grades, but more interested in improvements in learning, or spotting areas where students have misconceptions. Articles such as those by Rowntree (1999) offer a more detailed overview of issues to be considered when planning assessment [1]. Once the purpose of the assessment has been identified e-assessment may be considered as a possible assessment vehicle.

There are a variety of different reasons practitioners look to e-assessment. These may be:

  • to reduce the cost and burden of marking;
  • to handle increased class sizes;
  • to allow students more freedom as to when and where they carry out their learning; to provide additional assessment opportunities such as repeated practice;
  • or to provide richer and/or more immediate feedback (JISC 2008[2]).

Personal and institutional drivers are likely to be a combination of these, and may not always be educationally motivated. It is important to be aware that moving to e-assessment does not always lead to a reduction in costs, for instance, e-assessment which is driven by saving time on marking may simply shift the effort on to question setting and specifying marking schemes.

2. What and How to Assess?

Whatever the motivation, it is frequently commented that assessment drives learning. This means it is vital that all assessments, including e-assessments, are educationally valid, and that the focus and level of assessments within a course send out the desired message about what is considered important.

Not all types of assessment are suitable candidates for e-assessment. For example, if what is being assessed is a practical skill, then e-assessment is less likely to be an appropriate "how" at this stage in its development (although not impossible if simulations could be used). Creativity is another area where e-assessment is less useful in the form of e-testing, but an essay or portfolio (computer based or not) may well fit the task. On the other hand, if what is being assessed is knowledge based, or results in a well defined answer, then the use of e-assessment becomes more promising.

The decision of how to assess will be strongly influenced by the functionality of the specific e-testing system available. Understanding the requirements of the assessment and the impact different question types will have on the appropriateness of the resulting assessment is vital. E-assessment systems are still evolving, and compromises are likely. Whilst it is possible to create e-assessments which cross a variety of subjects and levels, it is also true that the limitations of current systems will render the use of e-assessment inappropriate in other areas.

2.1 What do I want to assess?

Knowing what the question is to assess is vital. It may sound obvious, but all too often questions are written because they follow a historical, or subject driven pattern – "We have always done it this way". The assumption is made that if a particular form of questioning has been used to assess a learning objective before, then it must be educationally valid. Venturing into any new form of assessment provides an opportunity to take a closer look at the educational validity. Those who have been involved in creating e-assessment questions often say that it provided an invaluable insight as it has forced them to explicitly state (and discuss with colleagues) what was often implicit about what was valued and rewarded in students’ answers.

2.2 How can I pose the question?

Each discipline has its own traditions in terms of assessment, from the balance of continuous versus end of course assessment, to the style of questions used. These practices are influenced by experience and the nature of the discipline, as well as internal and external stakeholders (particularly accreditation bodies). In general, though, written questions can be classified into 2 main groups: selected and constructed response.

Selected response questions are those where the student chooses an answer from a predefined set. Examples include: multiple choice (a single answer from a list); true/false or yes/no (specific forms of multiple choice); multiple response (more than one answer from a list); and sequencing (i.e. place these items in order starting with the most important).

Constructed response questions are those where the student must create their own answer. Examples include: Essays; Free text answers (typically a sentence or two); Single word or phrase answers; Numerical responses; Mathematical expressions; Graphical responses (i.e. flow chart, graph etc.).

The nature of selected response questions means that the student needs to recognise which answer is correct. It may be that students use elimination strategies to reduce the number of possible correct answers. The probability of guessing the correct answer should be factored into the marking and interpretation of the performance data. Marking is quick and consistent as there should be no ambiguity over the correct answer. This form of question can be usefully used to sample efficiently knowledge over a wide area of the curriculum, and can be used to focus attention at various stages within a course.

Constructed response questions do not rely on recognition of the answer. They allow the student to express the answer they believe is correct – they must know and create the answer. Typically the questions are more open-ended and allow for a variety of possible responses. They can be used to allow students to demonstrate knowledge, explain reasoning, develop an argument or evaluate a situation. The questions are typically easier to write, but harder to mark than selected response questions. Marking is usually more subjective and there may be lower consistency between markers, and across a number of answers for a single marker.

2.3 How will I pose the question?

E-assessments come with their own set of limitations, one of which is the style of questions that can be set. Most e-assessment systems will primarily handle objective style questions - those which have a single, or small set of, definitive correct answer(s). The majority of these are selected response questions, but a limited set of constructed response question types also exist in some systems, with automatic marking typically limited to short answers.

A wide variety of selected response questions is available – reflecting the fact that it is much easier to create computer systems which can mark these. Within e-assessment these question types can take a number of forms including:

  • multiple choice (and variants such as true/false),
  • multiple response,
  • hotspot (select an area of an image),
  • sequencing,
  • drag and drop (i.e. classify a number of items by dragging and dropping into various columns).

At first glance, it may seem a relatively simple task to specify such questions. However, ensuring the creation of robust questions which are both valid and reliable requires considerable skill and experience. There are numerous guides to writing selected response questions, particularly multiple choice and multiple response. However, even armed with the right information, writing good questions can be difficult and time consuming, especially for practitioners with little or no experience with this form of questioning. Well thought out though, selected response question types can be used to assess knowledge or recall. Assessing higher order skills, such as application, analysis, synthesis or interpretation, is less common, although examples do exist in some areas utilising techniques such as assertion-reasoning [3]. Of course, if selected response questions are being used prior to using e-assessment, any educational issues should already have been addressed, and converting any existing questions into electronic format should pose few problems.

Automatically marked constructed response questions are less common. Delivery of the question to the student is not particularly problematic, but, they are much more difficult to mark automatically. Many e-assessment systems do contain some automatically marked constructed response questions, but the marking is limited by the complexity of describing the variation of acceptable correct answers and the difficulty of programming this knowledge into a computer. For example, a textual input would typically be limited to specifying a small number of correct answers, which must match exactly; a mathematical answer is likely to be restricted to numerical answers allowing specification of an accuracy or range.

More advanced question types exist in a number of specialist systems, but these are not widely available. Work is ongoing in improving the capabilities of marking a variety of constructed response question types: for essays and shorter multi-phrase or multi-sentence answers; in marking mathematical expressions and in creating, interpreting and marking diagrammatical answers.

In some cases, automatic marking is not attempted and the assessment system is used to assist the teacher with marking, providing feedback and analysing the data. For example, submission of short essay answers using an e-assessment system can shorten the time between the student answering and the teacher looking at the submission, allowing for more timely feedback. E-assessment systems can also be used to support double marking, or multiple markers. However, for many, the lack of automatic marking and feedback removes many of the benefits sought.

2.4 Considering different approaches

Given that the majority of e-assessment systems primarily allow for the authoring of selected response questions, how might this influence the educational validity of the questions that can be created? Some of the issues can be illustrated by an example.

Consider the following learning objective:

"Draw a straight line from an equation of the form x + y = a, where a is an integer."

[taken from a SCHOLAR assessment on Intermediate Mathematics, a Scottish qualification]

How might this be presented as a selected or constructed response question? The possibilities presented within this example were created in the SCHOLAR assessment system (PASS-IT) - similar options for the selected response versions do exist in other systems, however the availability of constructed response questions of this type are more limited.

Multiple Choice

One option is to use a multiple choice question (Figure 1).

Figure 1: A multiple choice question

Figure 1: A multiple choice question.

The nature of the question requires a graphical response which necessitates using images. In this case there are 3 options for the learner to choose from.

There are a number of common errors students make in tackling this sort of question. If the main purpose of the assessment is formative, teachers will be interested in diagnosing misconceptions and providing feedback. To do this, answers which result from learners exhibiting the misconception need to be available. When only a small number of possible wrong answers are available it is likely that the answer the student believes is correct is not one of the options. This is a common problem with selected response questions.

If the students’ answer is not available, they may do one of a number of things: guess at an answer (possibly by first eliminating some options) or perhaps review their solution. How desirable this is will depend on the purpose of the assessment: if they guess correctly the misconception will go unnoticed, if they guess incorrectly the fact that they are guessing reduces the chances of identifying the reason behind the problem. Unintentional prompts to review their answer may be desirable in a formative assessment, but in a summative assessment this lets the student know their original, intended answer is wrong.

Another issue with many selected response questions is the space taken up by presenting an appropriate set of options. Where the options are graphical, a small number of options can quickly take up a lot of space, as can be seen above. This can cause problems if scrolling is required to view the question and options together. The problem tends not to be so extreme for textual response questions, and it may be possible to use display options like drop down lists to overcome this problem. However, care needs to be taken as including too many options from which to select can also cause problems with excessive reading, or necessitating selection from a number of very similar options.

Finally it cannot be ignored that the intention of the question is to assess the ability to create the appropriate straight line, not recognise it. Typically this involves calculating and plotting two points on a graph and drawing a line through them. However, with these options presented in Figure 1, working out one point correctly may allow the student to choose the correct answer (i.e. the point (0, 8) only exists on one option). Of course, more options could be included, but then issues of space, scrolling etc. occur as mentioned above.

Hotspot

Another option could be to use a hotspot question type (Figure 2). Whilst technically a selected response question type, in this case it can be created to operate like a constructed response question by creating numerous clickable hotspots of which two could be chosen to illustrate the straight line. Figure 2 shows two points on the correct line which have been chosen to indicate an answer (indicated by red circles).

Figure 2: A hotspot question

Figure 2: A hotspot question

At this level students would work with whole numbers; however this still leaves 441 hotspots to be created (which is time consuming!). The advantage of this approach is that the student can create one of many possible answers. This opens up the possibility of being able to identify misconceptions. The question itself takes up less screen space as all possible answers are created on one image. The student is also carrying out a more appropriate task than with the multiple choice question – they must calculate and plot two points. The remaining disadvantage is that the student does not actually submit an answer of a straight line. The impact of this will depend on what is being assessed and the capability of the student to interpret this information. The impact may well be greater if a student is stuck and is allowed to reveal the answer, again, seeing only two points rather than a straight line. For the purpose of this question seeing the straight line was considered to be significant.

Constructed Response

Finally, consider how this might look as a constructed response question. Figure 3 shows the initial presentation of the question – the student sees the equation of the line they have to draw, and an appropriate grid, but no possible lines are shown. Figure 4 shows the result after the student has constructed the answer.

Figure 3: Initial presentation of the question

Figure 3: Initial presentation of the question

Figure 4: Question showing student answer

Figure 4: Question showing student answer

With this format of the question the student can create their own answer. They can also see the straight line associated with their points. The author can choose what it is they wish to record, mark and give feedback on depending on the learning objective (e.g. the location of the points or the gradient and intercept). In this case it was desirable to give feedback on the location of the points plotted. If one point is plotted incorrectly then appropriate feedback can be given (either immediately or at the end of the test). If immediate feedback is given, it is much quicker (and neater) to change the position of a point by clicking and dragging on the computer, than to erase the point and line on paper.

As mentioned earlier, the availability of constructed response questions which allow automatic feedback and marking is limited. However, as can be seen from this example, this is where a lot of rich information can be obtained, allowing diagnosis and feedback for learning to be provided. The importance of automatically marked constructed response questions should not be underestimated if e-assessment is to become an important part of the learning and assessment process. The current functionality which exists in specialist systems needs to be more widely available, and further research and development needs to be carried out to extend the possibilities even further.

3. Feedback

With growing numbers in higher education and an increasing emphasis on formative assessment and independent learning, it is not uncommon for the motivation to consider e-assessment to be the automatic marking and the associated feedback that can be provided.

Feedback can be provided during the assessment (immediately after the submission of each answer), or via a post-assessment report (to student and/or teacher). For either situation it is important to consider what feedback will be useful to the learner.

If the question is automatically marked, the student may see the mark achieved or a tick or cross. This in itself can be useful, indicating areas where more revision or support is needed. However, it is usually possible to provide further textual (and sometimes graphical or multimedia) feedback.

For selected response question types it is typically possible to specify individual feedback for all responses, or a "correct" and "incorrect" response. Feedback needs to provide more than "Well done" or "Try again" – after all the mark/tick/cross will provide this information. Feedback needs to help the student reflect on their answer and help them move forward in their understanding – see Nicol (2004) [4] or Chickering & Gamson (1987) [5]. Remember, with a selected response question, the student must choose from a list of predefined options. A choice of a correct answer may be a lucky or educated guess - because the student has not constructed this themselves, it is important not to assume that the student knows why this is the correct answer.

For automatically marked constructed response questions, again, it may be possible to provide a "correct" and "incorrect" response. It is also useful to provide feedback on more than just the final answer. In the earlier example the final answer is the straight line, however, feedback is provided to tell them which of their points have been correctly plotted. An incorrectly placed point could have originated from a mistake in the calculation of the co-ordinates of the point, or as a result of plotting the x and y values against the wrong axis. Feedback on smaller parts of the process means students can check their answer in stages, correcting mistakes and building confidence, or are guided to check the appropriate part of their working. As the students progress this level of feedback may be removed to help them build their own checking mechanisms.

Options beyond this which can detect and respond to typical misconceptions exist, but again, only in a small number of specialist systems. The practitioner will be well versed in the common errors that students make in their own discipline - if the most common of these can be included as possible answers with appropriate marks and feedback then the sophistication of the support given to a student can be dramatically increased. This kind of functionality could be extremely useful if it was more generally available.

Specifying feedback to be automatically delivered is not the only way in which the data can be used to assist student learning and provide feedback. E-assessment systems record a wealth of data as students sit the tests – this data can be used to great effect.

4. How to use the data from e-assessment

With traditional pen and paper assessments the marking and provision of feedback is usually done by the teacher. This is an important part of the teacher developing an awareness of the progress of the class as a whole and at an individual level. This knowledge can then be used to adjust and inform teaching, or to respond to a struggling student.

In e-assessment, much of the work by the teacher is done at the start. If the assessments are marked and feedback is provided automatically there is a danger that the teacher can become removed from understanding the progress and problems of their students. Only a few years ago it was unusual to see an e-assessment system that did much more than give the overall marks for an assessment. However things have been changing and there is now a drive for access to the rich underlying information which can be used to inform and guide students and teachers.

Typically this data will be accessed through a web based reporting system, and usually students and teachers will have their own system (students only seeing their own data, teachers seeing everyone in their classes). Obviously what can be accessed is defined by the design of the reporting system, but it is worth considering here some of the areas that could be useful to support the teacher in developing that all important awareness of the learning taking place – in this case "who has been sitting an assessment?", and "how are the students performing?".

4.1 Who has been sitting the assessments?

Whether an assessment has been set to be completed by a certain date, or free access has been permitted, it can be useful to see who has been sitting the assessments.

An individual assessment to be completed by a certain time: Perhaps most interesting here is who has NOT completed the assessment. Non-participating students can be very hard to identify – in higher education attendance at classes is typically not recorded, and may not even be compulsory. Therefore a student who is not participating may not be identified until they have failed to hand in a piece of summative work. However, if formative e-assessments are available, this may be one way of identifying vulnerable students. Having been alerted to non-participation in one assessment, it should be easy to see their overall assessment participation. It may be that they have simply missed one test, but low or non-participation overall may be a signal to see if this is a student who needs some intervention.

A range of assessments to support self-directed learning: If students are allowed and encouraged to move at their own pace, an initial view of all assessments for the whole class can give an indication of the pace with which they are moving through the domain. It is important here to be able to see how many students have attempted each assessment, not just an average mark. Does this point to intervention to increase the pace, or might there be time for deeper learning, or revisiting a weak area, if progress is better than expected?

4.2 How are students performing on an assessment?

Final marks, and statistics based on final marks, only reveal part of the picture and care must be taken with interpretation. A high average mark coupled with a majority of students having taken the assessment can be the result of a number of situations, for example:

  • a few very good students raising the average,
  • repeated attempts by the same students with high or increasing overall marks.

Similarly, a low average mark is not always an indication of a problem, particularly where students have free rein to sit assessments as often as they want. If the assessment provides good feedback to the students and they feel they are learning from them, and the assessment provides different versions of the question each time, then students may well sit the test many times, hopefully improving with each sitting.

As with all assessments, only by delving deeper into performance at a question level and beyond will rich information be revealed which can be used to modify the learning and teaching which is occurring. Again, care needs to be taken with interpretation – the picture will look very different for assessments which are sat once only, or at a specific point in the learning, when compared with those where students have free access to sit the assessments when, and as often as they like.

Performance at a Question Level: Looking at the performance of the questions in an assessment across a class can highlight areas of high attainment, but more significantly, areas where performance is patchy, or very low. Before deciding that more time needs to be devoted to the specific objective(s), it is important to review the question. Could the performance be patchy or low because the question is not valid, or poorly aligned to the desired learning objective? If the question is acceptable, further investigation is still important in order to uncover the reasons for the low performance.

Student Answers to a Question: Many reporting systems provide statistical analysis of selected response answers – for example showing the proportion of answers for each option – does a specific option stand out, and if so, what does this say about the underlying understanding of the students? However, with selected response questions the probability that the answer was obtained through guessing needs to be factored in. In addition, it is often assumed that because a student got the answer correct, that they knew why it was correct, or knew the answer without prompting.

With constructed response answers, the analysis is more time consuming, but often more enlightening. This level of analysis requires that all student answers have been recorded by the assessment system, and that they can be obtained by the teacher. The best way to review this data is often to look at the answers to a particular question from all students together. Being able to sort and collate responses can help isolate particular misconceptions. Once the misconceptions have been isolated, the problem can be dealt with for the whole class, or with identified groups. Of course, even students who got the question correct may learn from understanding what might go wrong and why.

5. Conclusion

The use of computers in education becomes more common place with every year that passes. With learning taking place on computers, it is natural that the education community also look to use computers in assessment. As e-assessment systems become more widely available to practitioners this is becoming a practical reality for some. However, the use of appropriate e-assessment requires more than just the availability of a system – it requires experience and understanding of both the possibilities and the issues involved. Some of these issues have been explored within this article, focussing on the use of e-assessment which can support learning through immediate feedback and analysis of the rich information gathered during an assessment.

Whilst it is possible to create sophisticated, formative e-assessments, current systems impose restrictions on what can be assessed and how – e-assessment is still in its infancy. It is vital we continue to strengthen and extend the e-assessment domain by encouraging and supporting users in diverse subject areas, extending the variety of assessments which can be set, and enhancing the feedback and support provided to both students and teachers. We also need to look to the future of both learning and assessment. Recent years have seen an increased focus on areas such as independent learning, on the interpretation and application of knowledge rather than its acquisition, and on the development of portfolios to demonstrate ability rather than traditional examinations. Assessment must change alongside this if it is to provide any kind of useful measure of learning - being driven by educational need rather than technical possibility. There is much still to be done!

Acknowledgements

I would like to thank Ruth Thomas, Oormi Khaled and Cliff Beevers for their advice during the review of earlier drafts of this article. I am also grateful to SCHOLAR for permission to use the example discussed in this article.

Dr Helen Ashton is Head of e-Assessment for the SCHOLAR programme [6] where she is involved in the development of e-assessment technologies and supporting those writing assessments. Prior to this Helen was a Lecturer in Computer Science at Heriot-Watt University.

Helen has a strong interest in supporting feedback and reflection in learning (both online and otherwise), and in the use of technology in education. She has worked on a number of projects in this area including PASS-IT [7] and to deliver e-NABs in Mathematics and Computing in conjunction with SQA (Scottish Qualifications Authority) [8]. Current work in the area of online assessment includes research into the usage of such systems, the development of questions and specifications, and the development of delivery and reporting systems.

References

[1] Rowntree, D. (1999) Designing an Assessment System

[2] Exploring Tangible Benefits of e-Learning: Does investment yield interest?

[3] Assertion-reasoning questions are about understanding cause and effect. A question consists of an assertion and a reason. The learner must decide whether the assertion and reason are true, and whether the reason is the cause of the assertion. An example can be seen on the CAA Centre web site.

[4] Nicol, D. & Macfarlane-Dick, D. (2004) Rethinking formative assessment in HE: a theoretical model and seven principles of good feedback practice

[5] Chickering, A.W. & Gamson, A. (1987) Seven principles for good practice in undergraduate education. AAHE Bulletin, 40(7), 3-7

[6]The SCHOLAR Programme

[7]Project on ASsessment in Scotland - using Information Technology PASS-IT

[8] eNABs (electronic National Assessment Bank). NABs are minimum competency assessments for each unit within a National Qualification. These summative assessments are carried out internally by the school. A pass should be obtained in the NAB for each unit before the student can continue to the external assessment. This project is run in conjunction with the Scottish Qualifications Authority

Other e-Assessment Resources

There are lots of useful e-assessment resources available. A few are detailed below.

Examples and guides for question writing

This article is published according to the JISC CETIS Publication Policy under a Creative Commons Attribution-NonCommercial-ShareAlike License

 

Supported by JISC Supported by CETIS
Powered by Plone