Sunday, 15 June 2014

Supporting learners through assessment and feedback using TEL

#ocTEL - Week 4 If I only do one thing...

Reflections on some experiences or expectations of e-assessment and e-feedback to support student learning -
This post responds to ocTEL Week 4 'If I only do one thing...' questions. The questions asked are represented by the sub-headings below.

A particular type of e-assessment chosen and why

A media annotation tool (MAT) was chosen to build clinical thinking skills in chiropractic students (see Colasante, Kimpton & Hallam). Designed as 'assessment for learning', or formative assessment, the students had to work through a video case study of a patient-chiropractor clinical episode delivered in two parts:

  1. patient interview / medical history
  2. physical examination of the patient.

The students worked through a guided process of individual analysis and categorisation of the video (part 1), followed by small group collective synthesis of their pooled analyses - which received teacher feedback.

The student groups then had to determine a short-list of plausible diagnoses from their initial analysis, then use this short-list as the analysis categories for the second part of the video. Each individual had to arrive at a working diagnosis for the patient, but many collaborated within their group to tease this out.

All this was achieved within the tool, and all annotations and discussion threads remained anchored to it's relevant section of video as selected by the student.

De-identified MAT site - one of the chiropractic student groups (2011)

Effectiveness / how it helped deepen knowledge and understanding

Generally, both the chiropractic students and teachers were happy with the deepening of the application of knowledge to real (at least vicarious) situations. These were second year students, but normally the students don't get to apply clinical thinking knowledge to 'real' patients (other than fellow students) until fourth year.

However, there was an issue of concern that some students weren't following the clinical thinking process adequately. This was brought to the teachers' attention upon reviewing the student group's short listed differential diagnoses, when a few diagnoses that were implausible were listed.

The teachers intervened at this point, and offered a 'feedback lecture' which had strong attendance. It involved an interactive lecture where the class worked through the clinical thinking process together to arrive at possible differential diagnoses. This helped some students to arrive at 'aha' moments, and set the student body up well to tackle the second part of the video without becoming completely lost as newbies to this skill.

Approach to create environment conducive to self-directed learning, peer support and collaborative learning, and how technology might help

If a collaborative environment is set up, then some simple but guiding principles need to be established for the class, but particularly to set expectations for those who might not know how to collaborate effectively. For example, across a number of courses integrating MAT, one of the complaints was when peers within the group gave unhelpful feedback. This might be in the form of comments that were not constructive, including light, complementary commentary that wouldn't lead to deepened reflection or change of practice (e.g. Colasante 2011).

This would tend to match with the Institute of Education London (IOE) feedback profile (still searching for the more detailed PDF reference I found on this yesterday...), where P1 'giving praise' type feedback shouldn't be overused...

My response above I know is not necessarily a technological answer as requested, but a 'setting expectations' answer. (Although one might argue the guidelines could easily be posted to MAT for visualisation each time a student logs in to MAT...)

Overall, I believe the learning design is overwhelming the most important factor to enable the processes of self-directed and peer-supported learning, regardless of the technology.

Always, always: LEARNING FIRST!

As a contrast, MAT - as used in a near 'free-form' integration in one cohort - had students alienated against it. The tool was integrated as a packaged integration as is and the students encouraged to create several videos each to upload to MAT, without explained purpose or structured guidance on how to interact with artefact or peers (see Colasante & Lang, 2012).


Opportunities and challenges this approach presents to tutors

This approach of granular media annotation offers a lot of opportunities - where learning design is effective - for critical reflection, deep learning and application of theory to practice. One example would be to aid the flipped classroom, where content from a more traditional 'delivery' learning model is available online to students to attend to before coming to classes to do what was more considered 'homework', e.g. grabbling with examples, exercises, problems, with help of peers, tutors, teachers.

Video recording of lectures has become a typical method of allowing students to receive content online. However, if the student just sits at home to watch the video, isn't this still a largely passive method of learning, but with learner control of stop-start-pause, etc.? Ways of engaging with the content may be many and varied, but one method is using a video annotation tool, such as MAT.

The image below shows a Lectopia recorded lecture (which in this case included a video of a site visit as shown in the image), which was loaded to a test site in MAT to grapple with possibilities of designing interactive annotation activities in MAT, especially to make the video more meaningful on student return visits to the recorded lecture. This was considered to have wide learning benefits (however, the additional storage issue of many recorded lecture videos, or the interfacing direct from MAT to Lectopia, have not been resolved.)

Labelled and de-identified MAT test site (C) Colasante, M (2010). Future-focused learning via online anchored discussion, connecting learners with digital artefacts, other learners, and teachers. In C.H. Steel, M.J. Keppell, P. Gerbic & S. Housego (Eds.), Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp.211-221). 

An analysis was conducted on MAT regarding challenges and opportunities (see Ruyters et al., 2012), especially from the viewpoint of all ten teachers using MAT in a multiple-case study:

Noted opportunities:

  • recommended the benefits of MAT to other teachers
  • offered that the technology, despite a few glitches, was relatively easy to use
  • suggested future uses of MAT for their own classes and for others.

Noted challenges:

  • requirement for professional development in using MAT
  • promotion of MAT to others
  • continuation and strengthening of the Community of Practice formed with MAT
  • ongoing support from the university for use of MAT (post funded project).



Unfortunately, the server specifically bought for MAT still sits in a room somewhere, with the temporary server supporting MAT restricting very limited usage. Therefore, the hardware limitations have prevented additional promotion and uptake of MAT. Ironically, because there hasn't been further uptake of MAT, a decision has been made that perhaps MAT is not needed after-all, and analysis has been undertaken to phase it out.

MAT was born under the encouragement of a VC who said "Be bold!", an educational technology director who established a creative think-tank on annotation tool requirements, and a DVC-Academic who provided funding for analysis of learning and teaching with the subsequent university-created tool, MAT. Innovative people worked on requirements, design, design testing, programming, development, developmental testing, implementation, analysis, sharing of findings, etc. Two of the first three figures mentioned have left the university, the third about to leave, all going/gone onto bigger and better things. The climate remaining is not conducive to innovation outside the standard LMS. Allow me this sober moment - some of us have been working on this for years (often in our own time, such was the enthusiasm), some since 2005...


  1. This is a really in depth look at an e-assessment tool and touches on many of the common issues that many learning technologist (or people with similar positions). The word that came to mind was sustainability of initiatives from pilots to wider adoption. Thank you for sharing this (what appears) to be a painful project in the end! (Santanu Vasant, ocTEL support tutor)

  2. Thanks Santanu. The sustainability issue is an interesting one. When I was doing the literature review for this project, I stumbled upon a number of institute-built annotations tools across the globe with varying features, many of which had already seemingly disappeared. On further hunting, I found that a small few of these had effectively moved on from a beta version to a totally new tool with a new name, applying lessons learnt from the first version. My dreamy-eyed vision was that our MAT was so good we would just need to tweak it upon feedback. Now I feel that if we did have institute support, a rebuild would be required. This could take advantage of new and improved programming software available, and redesign it as an interface tool to connect to videos and other media already stored - rather than depending on additional dedicated server space for each video to be annotated. Alternatively, I would think that if the institute doesn't feel the need to support MAT any more, then I would like to see it handed over to the open source community. This has been suggested to me at conferences around the globe, and I get the feeling other passionate people could really run with this. This latter point gives me hope :-)