This is an archive of the original site and you may encounter broken links and/or functionality

Assessment and Curriculum (3 Short Papers 0214, 0256, 0259)


13:40 - 14:40 on Tuesday, 6 September 2011 in A1

The abstract below for 0256 replaces the one published here until 12.56 on 5 September.

0214 Visualising the holistic assessment experience – The use of Google tools to support effective design. Mark Kerrigan, Rita Headington, Simon Walker
0256 Managing Change in the Development of Sustainable Online Assessment Practices Ann Liggett, Christopher Cramphorn
0259 So what? Evidencing the value of curriculum innovations with technology Rachel A Harris
0214 Visualising the holistic assessment experience – The use of Google tools to support effective design. Mark Kerrigan, Rita Headington, Simon Walker

Effective programme design and delivery has become increasing fragmented with individuals struggling to support the holistic student experience and the development of graduate attributes (GAP project). These problems are exemplified by the design of assessment and feedback, whereby there is often coursework bunching, a limited diet and poor opportunities for the effective use of feedback (The Escape Project, 2008; Higgins et al., 2002). Institutions need to take a step back and review the fundamentals of assessment and feedback design at the programme level and how, in such challenging times, effectiveness can be maximised whilst enhancing the both the staff and student experience.

As part of the newly formed central Educational Development Unit at the University of Greenwich, a process for visualising the design of assessment and feedback via an open-source Google tool was developed to help inform successful programme design based on the principles developed by the JISC funded ESCAPE project. Course leaders were asked to complete a simple course diagnostic questionnaire built using Google Forms and the data automatically feed into a programme database. This culminated in an automatic report that graphically displays assessment diet, landscape and importantly an experiential timeline. The timeline, based on a Google Motion Chart, interactively displays assessment periods, type and weighting concurrently for each course within a programme. Staff are then able to interact with the assessment parts of their programmes and see graphically, in real-time, the consequences of their design decisions. By linking this process to the face-to-face curriculum design workshops, staff are better informed about their programme’s design, thus permitting the evidential-based development of a supportive holistic curriculum that is aligned to both the staff and student experience and good assessment practice.

Here we will provide an evidence-based report on how both staff have received this process and how it is possible to systematically enhance the design of assessment and feedback practice using an easily scalable, and transferable technological solution.

0256 Managing Change in the Development of Sustainable Online Assessment Practices Ann Liggett, Christopher Cramphorn

The paper will explore successful cultural change strategies within the framework of a large scale online assessment (submission, marking, moderation and feedback) project. The project findings are summarised and offered to the wider academic community for wider dissemination in the form of a methodology for assessing scalable change requirements and including the key considerations of local context, academic regulations, organisational structures and role responsibilities.

The development of tools and software is an important aspect of online assessment practice but the engagement of academic and administrative staff to implement online assessment for student benefit at a Faculty or Institutional level can present the greater challenge. An initial literature review established that research in this area is largely focussed on the technological aspects of e-assessment implementation and practitioner skill acquisition (Ashton, Beevers and Thomas 2008; BECTA 2010; Bridge and Appleyard 2005). Staff engagement and cultural change is largely neglected in favour of student reaction and feedback.

The 2010/11 project comprised a range of pilots across two schools Business and Social Sciences in a large post 1992 university and involved 40 academic staff, 10 administrators and 10 programmes of academic study at both undergraduate and postgraduate level. Over a period of 12 months 39 module teams and student cohorts of between 10 and 150 students were involved in the transition between paper based and online practice. The aspiration to 'move towards online submission and marking for all text-based coursework’ became the overarching aim of the project. The project identified an evaluation methodology which tracked over time the reactions and adaptations made by participating practitioners and students. This process highlighted “early impact” evaluation data for project managers before moving onto the longer term consideration of cultural change and organisational transformation. A second measure of impact focussed on the organisation as a whole, as procedures, roles and responsibilities required change at module and programme levels, between stakeholders, and between schools and institutional wide services and systems. The paper will explore the challenges and strategies for addressing the engrained cultural practices embedded in conventional paper based submission, marking, moderation and feedback practices, and its transition into the virtual environment. Online assessment enthusiasts and experimenters are more likely to adapt on online assessment practices. Resistance to change by ‘non-enthusiasts’ will require a significant cultural shift in assessment practices and pedagogic worldviews to effectively challenge traditional rituals and implement new online assessment protocols and techniques.

0259 So what? Evidencing the value of curriculum innovations with technology Rachel A Harris

In designing their evaluation, projects were encouraged to think ahead to the kinds of impact they would like to achieve, to use these to derive potential ‘measures of success’, and then consider what would make credible evidence of impact. The perspective of different stakeholders was envisioned as part of this process.

Approaches to evaluation

On the basis of 15 very different projects, it is not possible to draw conclusions on the ‘best’ approach to evaluate curriculum innovations. This paper will, however, outline some of the evaluation approaches used. For example, a number of projects applied or adapted action research, and a typical evaluation cycle based in action research will be described. The potential benefits of applying appreciative inquiry, engaging an independent evaluator and of traditional formative evaluation will also be outlined.

Conclusion

One of the drivers of evaluation within the Transforming Curriculum Delivery Through Technology Programme was to learn lessons from within the programme that could be evidenced and shared outside the programme. While the programme did not necessarily advocate the strict levels of rigour associated with randomised controlled trials, there are many examples of highly effective evaluation methods, techniques and findings. Furthermore, the programme’s approach of supporting projects in self-evaluation has the potential to both provide evidence of what does and what does not work, but also to raise evaluation capacity within the sector, and hence improve future evidence generation.