Association for Learning Technology Online Newsletter
Issue 14 October 2008   Tuesday, November 4, 2008

ISSN 1748-3603

Cover Page »
Project updates
JISC Users and Innovation Programme
Scoping a vision for formative e-assessment
Think first: the Benchmarking and Pathfinder Programme 2005-2008
Conference reviews
ALT Conference 2008
ALT news
In memory of Keith Duckitt
Chief Executive's Report
An interview with Diana Laurillard
ALT Conference 2009
ALT Workshops
ALT Journal
Subscribe / Remove
Privacy policy

ALT Website

ALT-N online
Past Issues
Issue 13 July 2008
July 21, 2008
Issue 12 May 2008
May 6, 2008
Issue 11 January 2008
January 25, 2008
Issue 10 October 2007
October 19, 2007
Issue 9 July 2007
July 20, 2007

More Issues »
Think first: the Benchmarking and Pathfinder Programme 2005-2008
by Terry Mayes and Derek Morrison

Over the last three years a large-scale e-learning development programme, the Benchmarking and Pathfinder programme, has been funded to the tune of £8M by the Higher Education Funding Council for England (HEFCE). This initiative has been interestingly different from previous national initiatives; the background to the programme gives us a clue about why this has been so. Following the publication in March 2005 of the HEFCE 10-year e-Learning Strategy, the Higher Education Academy was invited to lead a Benchmarking Exercise and related Pathfinder Programme in partnership with the JISC. The Benchmarking exercise was intended to help institutions establish where they were in regard to embedding e-learning. The Pathfinder Programme, by contrast was specifically designed to help selected institutions, on behalf of the sector, to identify, implement and evaluate different approaches to the embedding of technology-enhanced learning in ways that result in positive institutional change.
The HEFCE strategy provided the strategic background to the programme, with its emphasis on placing the responsibility for, and ownership of, e-learning development clearly with the individual institution.  A significant policy context that influenced this strategy was the fallout from the closure of the UKeU, an episode which most of the sector interpreted as a failure of the Funding Councils’ attempts to exploit the potential of technology-based learning through centrally-driven initiatives. It was the residue of funds from that initiative that provided the resource for the three-year Benchmarking and Pathfinder Programme.
A great deal of material relevant to the programme can be found on the two public weblogs at:

The Higher Education Academy encouraged regular postings on general issues of interest to the programme as well as operational information about the programme. Each of the participating institutions also maintained a weblog of their project, both for Benchmarking, and where appropriate, for Pathfinder. At the recent conclusion of the programme, projects have posted a reflective account of their ‘journey’, and most have also posted briefing papers aimed at particular audiences in the sector.

Implementation of the programme started with the pilot phase of the benchmarking exercise which ran from January to July 2006. The main goal was for each institution to undertake a fundamental analysis of its own e-learning processes, provision, and practice on which its own future development decisions could be based. It was important that institutions had complete ownership of the process, and that their confidentiality was respected. There was clear opposition to the idea that a single benchmarking methodology should be adopted and imposed upon them. Although there was a view that using more than one method would make it more difficult to capture the state of e-learning across the sector, there was a greater perceived benefit in allowing institutions to choose their own approach. The Academy offered to support five benchmarking approaches that had been identified by the institutions that had responded to the initial expression of interest. By the end of the programme in July 2008, 77 institutions had undertaken an internal e-learning benchmarking exercise supported by the Academy, including some Welsh and Scottish institutions.

The methodologies were:

  1. ELTI (Embedding Learning Technologies Institutionally), originally developed as part a JISC project which completed in 2003. ELTI focuses on three key areas: Culture, Infrastructure and Expertise.
  2. eMM (e-Learning Maturity Model), developed by Stephen Marshall of the Victoria University of Wellington. eMM is based on the principle that an organisation's processes mature along a five step model of capability in e-learning, moving from 'ad hoc' processes and decision-making to an informed, engaged and reflective culture of continuous improvement.
  3. MIT90s is a conceptual framework developed at MIT in the 1990s for planning and monitoring strategic change in relation to e-learning, and represents an organisation as comprising of five interacting elements: its strategies (for technology use); its organisational structures; individuals in roles; management processes; and technologies
  4. OBHE/ACU (Observatory for Borderless Higher Education/Association of Commonwealth Universities) has developed a collaborative benchmarking methodology which offers institutions the opportunity for comparison at the process level and promotes the development of shared good practice statements.
  5. Pick&Mix was developed by Paul Bacsich, and is based on a systematic review of other approaches to benchmarking e-learning, looking for commonalities of approach but also taking a fresh start. It includes a set of core and supplementary criteria, with an option for the inclusion of local criteria specific to the needs of a participating institution.
The five methodologies were ‘works-in-progress’ at the start of the programme, and some required adaptation for implementation in the UK HE context. The Higher Education Academy recruited a team experienced in the support of such methodologies to help each institution complete the exercise. On the whole, institutions felt that the discipline imposed by undertaking an externally-coordinated exercise was of great benefit, and that the choice of methodology was of less importance than the availability (or lack of availability) of the required data about how e-learning was actually being developed within the institution. For many institutions this exercise acted as a wake-up call, while for others the exercise provided valuable insights. For a significant number these included: the realisation that they had only limited awareness of their own developments in this field; and that it is surprisingly difficult to find out about the actual student experience of e-learning, or to conclude anything about its effectiveness in enhancing learning. The main benefit of participation in the exercise was its role in raising awareness of e-learning and its potential in the changing nature of learning and teaching, both for the strategic planning of senior managers, and the on-the-ground pedagogy for academic staff.
Thirty-seven of the institutions that undertook benchmarking received funding for Pathfinder projects. After a pilot stage with nine of the original projects a number of lessons were drawn about how the main Pathfinder programme should proceed. There were three features of the designed programme that were comparatively novel and which worked well. First, a formative evaluation team were contracted by the Academy to support all the projects in conducting their own evaluations of their impact. Second, the 28 projects in the main phase were clustered into CAMEL (Collaborative Approach to the Management of E-Learning) groups, following the good practice that had by then been reported for that method, involving the reciprocal hosting of visits across a cluster aimed at the open sharing of issues and approaches. Thirdly, each of the projects in a cluster was allocated a ‘Critical Friend’ who was an experienced and respected figure in the field, asked to work from ‘inside’ each project to bring expertise and contacts not necessarily available within the institution.

Unusually for an externally funded programme, the Pathfinder projects were typically tailored to the particular needs of each host institution, those needs being revealed through a rigorous process of analysis - the benchmarking activity. In most cases they were, as hoped for in the HEFCE strategy, aimed at the deeper embedding of e-learning into the mainstream provision. This meant that many were building on areas of perceived weakness, increasing the overall deployment of e-learning rather than building on relatively isolated areas of innovation. Although each institution was developing for its own needs, the picture that emerged by generalising across the 37 projects allows us to draw some tentative conclusions about how the sector as a whole sees its priorities.

At one end of the spectrum, a few Pathfinders have produced highly specific attempts to develop particular approaches to e-learning itself, such as digital storytelling or learner-created knowledge or e-assessment. The large majority, however, have taken a range of approaches to the development and empowering of academic and support staff in e-pedagogy. In many projects this has involved the construction of new institutional infrastructures for communication to, and influence of, teaching staff in academic Schools, mostly targeting course redesign as a key opportunity. At the other end of the Pathfinder dimension lie a number of attempts to understand more deeply the nature of the changing student experience of e-learning, and to begin to grapple with some of the challenges such understanding brings to institutional policy.

In the last phase of the programme, five projects were funded to take their outputs to the wider community. These were collaborations between existing Pathfinder institutions and they represented the programme’s clearest examples of capacity building. Each has succeeded in recruiting wide participation across the sector in taking forward the following topics: quality and e-learning;  intensive course redesign events; podcasting for pedagogic purposes; and researching the student experience. There is reason to be optimistic that these will prove to be sustainable as communities of practice.

It is arguable that e-learning development is currently on the cusp of a fundamental shift. Across the programme, we see an emerging awareness that the VLE-centred approaches of recent years are increasingly out of alignment with the changing expectations of students revealed in the JISC learner-experiences of e-learning projects, and in some of the survey work carried out in the programme itself. Student expectation is increasingly setting the agenda for institutions. The logic of this is to move from a supply-side view of e-learning to a demand-side one, and institutions are currently struggling to come to terms with some huge implications for the curriculum.

There is an emerging interest across the programme as a whole in a subject-based approach to technology enhancement, with particular reference to the Subject Centres and some CETLs as both a focus for benchmarking activities and as a forum for reflecting and acting on the programme’s outcomes and recommendations. The affordances offered by technology in enhancing learning in different academic disciplines has been a focus of particular interest for some participants, and should be further pursued.

Perhaps the most distinctive feature of the programme has been its level of trust. The institutions have been trusted to devise Pathfinder projects based on their own individual strategic analysis, the projects have been trusted to conduct their projects effectively and reflectively through building trust relationships with Critical Friends and with each other, and through sharing their reflections on the challenges. The management of the programme, and its administrative style, has been light-touch, in keeping with HEFCE’s aim to achieve institutional ownership of the programme. The programme’s overall approach has been widely appreciated by the projects themselves.

Terry Mayes
Glasgow Caledonian University

Derek Morrison 
Higher Education Academy


Printer Friendly Version »

There are no letters available.

Created with Newsweaver