Exploring the mechanisms for assuring quality of e-learning courses in UK higher education institutions
Magdalena Jara [m.jara@ioe.ac.uk]
and Harvey Mellar [h.mellar@ioe.ac.uk]
Institute of Education, University of London
20 Bedford Way, London WC1H 0AL
[http://www.ioe.ac.uk]
Abstracts
English
This paper reports on a research study aimed at understanding how dual mode higher education institutions in the UK approach the application of their internal quality assurance procedures to their online courses in order to allow them to assure and enhance their quality. The research strategy aimed to identify whether the procedures implemented were capturing the aspects that characterise online courses. A case study approach allowed an examination of the procedures as well as the features of the courses under study. The results indicate that the specific quality assurance procedures most affected by the online modality were module evaluations, student representation and team meetings.
Spanish
Este artículo reporta la investigación realizada con el objetivo de comprender cómo instituciones tradicionales de educación superior en Inglaterra afrontan la aplicación de sus mecanismos internos de aseguramiento de calidad a sus cursos en línea, de manera que les permita asegurar y mejorar su calidad. La estrategia de investigación buscó identificar si los mecanismos usados eran capaces de capturar los aspectos característicos de los cursos en línea. El estudio de casos realizado permitió examinar estos mecanismos, así también como las características de los cursos estudiados. Los resultados indican que los mecanismos de aseguramiento de calidad más afectados por la modalidad en línea fueron las evaluaciones de cursos, los representantes estudiantiles y las reuniones de equipo.
An earlier version of this paper was published on the Fourth EDEN Research Workshop, Barcelona, 2006.
Keywords
Quality assurance, e-learning, online learning, quality enhancement, higher education, institutional management of quality.
Topics
- Introduction
- Quality Assurance in Higher Education
- Exploring the effectiveness of quality assurance procedures
- Results
- Conclusions
- References
Introduction
Over the last 30 years key changes have taken place in higher education institutions that have led to significant transformations in their practices and policies.
Looking at the UK, it is observed that the expansion and diversification of higher education systems has brought a more diverse student body into universities. New entrants were from a wider range of backgrounds, ages and qualifications, and also from groups traditionally disadvantaged - ethnic minorities and people with disabilities (Ashwin, 2006). This growing demand for higher education placed institutions in a new scenario where they needed to respond creatively to the needs of the new student body, within the constraints of limited funding, in order to be able to succeed. Flexible learning became the means by which institutions sought to address this challenge. Consequently, universities started to offer more flexible courses, including technology-based distance education the use of which has grown steadily over the past decade, becoming widely used by universities around the world. In the UK it has been estimated that in 1999 there were more than 70 dual mode higher education institutions (Weyers, 2000).
This process has presented a number of challenges for higher education institutions. One aspect significantly affected by these new modes of delivery is the institutional processes set up to maintain and enhance the quality of their programmes of study, as governments called for more accountable institutions through the implementation of a set of quality assurance measures (Harvey and Knight, 1996).
Quality Assurance in Higher Education
Developments in quality assurance have taken place over the past decades at different levels: institutional, national and internationally, generating intense debates over the notions of quality embedded in the different arrangements, and the role and effectiveness of these mechanisms in improving the quality of courses (Massy, 1996; Middlehurst and Campbell, 2003).
At institutional level, quality assurance mechanisms have been shown to be strongly affected by the notions of accountability and enhancement, which are particularly relevant in the UK context, where the roles of the Quality Assurance Agency (QAA) and higher education institutions internal quality management systems have evolved during the last decade in the search for more appropriate and effective mechanisms (Morley, 2003; Brown, 2004; Harvey, 2005)
A closer exploration of the internal quality assurance mechanisms in higher education institutions in the UK also shows that the tension between accountability and enhancement is not easily resolved, putting greater strain on the capacity of institutions to manage their internal procedures effectively (Middlehurst, 1997; Biggs, 2001; Harvey, 2005; Inglis, 2005). This challenge becomes more demanding as new modes of provision increasingly become part of traditional campus-based higher education provision, and as institutions try to use the same mechanisms to deal with these completely new forms of courses.
Internal mechanisms for quality assurance and quality enhancement are part of the requirements that UK higher education institutions have in place as part of their responsibility for the quality of their programmes. These internal mechanisms should be in line with the Code of Practice established by the QAA, yet each institution has the responsibility of setting up adequate internal procedures that assure the academic quality of their programmes according to their internal standards (QAA, 2002). Consequently, institutions have fairly similar internal quality assurance procedures in terms of their aims, although they may be set up differently according to their own internal organisation and structure.
The QAA recommends higher education institutions to use these mechanisms to assist them in the processes of assuring and enhancing the quality of their courses, all their courses. The question that emerges therefore is whether these internal quality assurance mechanisms are effective for every type of course that an institution provides, and particularly whether they are effective for assuring and enhancing the quality of online learning courses. The QAA have recognised the relevance of this issue by issuing a section of their code of practice's specifically about collaborative and flexible provision, including e-learning (QAA, 2004b); this code however is of a different nature to the ones referred to earlier which related to the internal management of quality and described the procedures to assure and enhance quality. The code of practice that covers collaborative provision and e-learning is defined solely as a guide for practitioners; the codes of practice related to internal management of quality are defined as a requirement with which higher education institution should comply.
The literature that discusses this question tends to agree in the judgement that the quality assurance arrangements of e-learning should be different from those of traditional distance learning and on campus delivery. Though the literature presents some differences in the level of detail with which this analysis is approached, combined it offers a clear overview of the e-learning aspects potentially impacting quality assurance arrangements (O'Shea, Bearman and Downes, 1996; CVCP, 2000; Hope, 2001; Middlehurst, 2001; Harvey, 2002; Roffe, 2002; Connolly, Jones and O'Shea, 2005).
The main arguments supporting the revision of the quality assurance arrangements are based on the differences that it is possible to identify between e-learning and campus based learning. Among the variety of elements that it is suggested are distinctive to online learning, there are four main aspects that seem relevant for the present analysis regarding quality assurance:
- disaggregated processes - e-learning courses are no longer in the charge of only one person who takes care of the whole process.
- organisation of the teams - academic staff no longer work in isolation; e-learning courses require teams to work collaboratively, and academic staff need to interact with many other professionals who are involved in the different phases of course design and delivery.
- visibility or openness to review - monitoring activities can be more in depth, continuous and unobtrusive than in face to face delivery or traditional distance learning; and
- limited access of staff to students.
Taking into account these four elements, the present research was carried out with the aim of furthering our understanding of how dual mode universities could approach the application of their internal quality assurance procedures to their online courses in order to allow them to assure and enhance their quality. The research strategy aimed to identify whether the quality assurance procedures already in place in the institutions under study were able to capture – and, if so, then to what extent - aspects of those elements that characterise online courses. A case study approach was selected as the most appropriate strategy, in that it would allow a deep examination of the quality assurance procedures as well as the features of the courses under study, whilst keeping their connection with their institutional context.
Exploring the effectiveness of quality assurance procedures
The study included four case studies of online or mixed mode courses. Each course had been under quality assurance procedures and was part of the academic offer of a dual mode higher education institution in the UK.
The first set of data gathered in each case study consisted of the quality assurance documentation related to the particular course selected. A total of 67 documents from all four case studies were coded using specific quality assurance categories devised for this project, which will be explained below. The documentation collected for each case study varied in size and contents, as the different institutions presented different ways in which they organised their records. The analysis of the quality assurance documentation gave a picture of what was actually being captured by the procedures in place in the courses under study. In order to identify the issues which were not being captured by these procedures a set of interviews were carried out with a group of participants of each of the courses.
The aim of the interviews was to get the participants' views on the quality and features of their courses, providing complementary data to the documentation already analysed. The selection of the interviewees was carried out based on their roles. The roles expected to be covered were academic staff and tutors; administrator(s); students; employers; support staff – both technical and administrative– and developers/designer(s). A total of 16 academic staff and 10 students were finally interviewed, the interviews were transcribed and coded. Supplementary data from students was gathered through an online survey.
The aim of the document analysis was to map out the aspects and quality issues that were actually being captured by the quality assurance procedures, and the extent to which they were effectively being captured. The analysis of the interviews aimed to identify the aspects and quality issues that were described by the participants with respect to their online courses. The results obtained from the analysis of the interviews were then contrasted with the results of the documents in order to get a map of the aspects mentioned by the interviewees that were not present in the quality assurance documentation.
In order to carry out this comparison, the documentary texts and the interview transcripts were coded following a predefined list of categories based on the theoretical aspects of quality assurance taken from the literature. The starting point for creating the list of codes, was the examination of the main quality assurance documents that higher education institutions are required to use when applying their internal procedures: the Quality Assurance Agency's Handbook for Academic Review (QAA, 2000c) and the relevant sections of the Code of Practice for the assurance of academic quality and standards in higher education (QAA, 2000b; QAA, 2000a; QAA, 2004b; QAA, 2004a). This list of codes was first piloted, modified to ensure clarity, and the final version – as used in this study– is presented in Table 1.
Table 1. List of codes
QA aspects |
General Categories |
Specific categories |
Administrative issues |
||
Outcomes Standards |
Course definition |
Intended learning outcomes |
Expectations |
||
Curriculum |
||
Assessment |
||
Teaching and learning |
Student capacity |
|
Staff capacity |
||
Teaching methods |
||
Student achievement |
||
Student expectations |
||
Learning Opportunities |
Student support |
Before the start of the course |
During delivery |
||
Academic support |
||
Accessibility & Equal opportunities issues |
||
Learning resources |
Staff |
|
Facilities |
||
Delivery system |
||
QA Procedures for Enhancement |
QA procedures |
External examiners |
Module evaluations |
||
Student representation, complaints and appeals |
||
Annual Review |
||
Other QA procedures |
Results
An analysis of the four case studies in terms of the number and nature of the issues which were not covered by the quality assurance procedures showed that the majority of the issues that tended to be left out from the quality assurance documentation were related to student participation and the support provided to the students. Other significant issues also missing in the documentation reviewed related to assessment strategies and organisational issues (like the relationship with the college, equivalence with on campus version, staff coverage and access to resources)
Looking at this issue from the point of view of its sources, it seems that the information missing in the quality assurance documentation was that that might mostly have come from the students, indicating that although courses had procedures in place to collect student feedback, these strategies had not been sufficient or had not been fully effective.
Each of the cases studies presented some specific features affecting the implementation of the quality assurance procedures in place, particularly those associated with the organisational context in which the courses were located. In some way it appears that online and mixed mode courses tended to become isolated from the rest of the institutional processes in place, unless there was a clear and strong leadership that would align them with these processes. This isolation was made evident in the way the institution-wide quality assurance mechanisms allowed these courses to carry on their business on their own, sometimes without significant oversight, as they had not set up any particular requirements for online courses.
The main features that characterise online courses within dual mode higher education institutions that appear to be affecting the quality assurance procedures in place in the courses under study were: the position that these courses had within their own institutions; the distributed configuration of the course teams and the distant location of students.
In terms of the specific quality assurance procedures studied across the four case studies, the analysis of the findings suggests that:
- External examiners were one of the few quality assurance procedures in place in all case studies with equal level of formalisation. The role of the external examiner was perceived as a positive one by all staff members, both because the external examiners it brought in an external view on the course and because of their support in the assessment process. Staff did not think that there was any particular distinction between the role of the external examiner in an online course and any other type of course. In two courses staff said that they had looked for someone who, as well as being an expert in the course topic, had the appropriate experience and would show sympathy for the modality. The perceived benefits of looking for these characteristics in external examiners were to get a greater understanding on their part of the way in which distance/online learning was carried out, and hence a more critical review of the course, although this was always dependant on their personality:
… it is actually helpful if they kind of want to know why you do something in case you have an entirely valid reason rather than just state this isn't happening. (Staff, CS4)
The External examiner's role was affected by the online modality only in the possibility of their meeting students, and in some cases of meeting all the tutors however staff did not consider that this restriction was affecting the quality of the feedback and support provided by the external examiners.
- Annual Reviews were also a procedure in place in all the cases under study and the value assigned to them by staff was also similar: The annual review process was perceived as a useful exercise that helped organise the paperwork, get the issues discussed and written down.
I think what's helpful…. I mean it's a real pain doing it…. it takes ages but what's helpful is having to produce that documentation for it, so yeah….. so I mean actually sitting down for a day and producing and doing what I've just said about bringing together all the disparate sources of student feedback and checking with the administrator about the student numbers and going back and checking their progression figures that they send us from [the external unit] and to check they've got that right, and I mean all that I think is quite a useful exercise to go through. (Staff, CS1)
The perceived effectiveness of the annual review process however varied. Although most staff considered the process worthwhile as an additional opportunity for reviewing the course, some staff considered annual reviews ineffective as although issues get noted, they felt that their institutions were not prepared to deal with and solve the problems identified.
Staff were more concerned with the annual review's effectiveness rather than with its appropriateness for the modality. Staff believed that there was no need for a different annual review form for e-learning courses, as the particular features of the courses would be brought up in the issues to be stated in the report.
- In contrast to the previous procedures, the strategies for collecting student feedback were a problematic area in the courses studied: module evaluations were heavily affected by the online modality. The most significant issues with this procedure were related to response rates, the tools used to collect the feedback, the moment in which it was collected and the type of questionnaire used.
Each case studied presented a different way to deal with these issues, although common to all of them was a sense of evolution in the way they had approached the issue. Two cases are good illustrations of this maturation process, as they had changed their ways to get student feedback more than once, in an attempt to get better response rates and/or higher quality of feedback. The other two case studies presented different situations regarding module evaluations. In one of these cases, the course had had in place an online survey for several years with an explicit procedure for its monitoring and reporting. In this case, despite the fact that a well established strategy was being implemented, it failed to gather enough responses or to be of sufficient quality to be considered useful. The fourth case study presented a totally different context. This course did not have a module evaluation formally in place, not because they had not planned it but because nobody asked the students to complete it:
…I think probably [the module evaluation] fell between the cracks for this session, because I thought that the student… would be sent out by the development team to all the students but it didn't go out at all, not to our students and I don't know who was responsible for sending it out… (Staff, CS3)
On the whole, module evaluations seem to be a procedure teams struggled to get right in their courses. Despite the different mechanisms by which teams attempted to collect feedback from students, one common problem was the low response rates they were getting, which staff considered to be directly affected by the distance. Students also recognised the problem of low response rates, but the reasons they gave for not completing the feedback were different from the opinions of the staff, the main issues that the students raised were related to the moment of the year in which they were asked to do it.
Overall, the issue about effectiveness of student feedback was mainly focused on how to get data rather than on the quality of the data gathered. The 'online surveys' were associated with 'tick boxing' answers that students tend to do quickly and without major reflection, while open questions could gather better quality of responses. The problem appeared when students would not respond because open questions require too much effort. Probably the challenge is to get the right balance.
- Student representation was in operation only in one case study course, which had a combination of online and mixed mode modules. The other three, fully online, courses had not implemented it, nor tried to do so. The reasons for not having student representation for staff were clear: students were spread around the country and abroad, hence they could not attend staff meetings; and/or students did not know each other, so they would not be able to select their representatives. In this sense, the modality of delivery of the courses was directly affecting the implementation of this procedure.
Even the mixed mode course that had student representatives had similar problems. The student perspective on this was somewhat surprising, the students interviewed knew about the representative but none of them had used this mechanism to put forward any issue to the staff team, they felt, in fact, that they did not need student representatives at all:
I would have just seen my own tutor… […] … to be totally honest it wouldn't cross my mind to go through that channel [the student representative]. (Student, CS2)
This raises the question whether student representation is an appropriate procedure for online and mixed mode courses. It seems that given the strong and close link that is established between students and their tutors (at least in these online courses) that student representation may not have a role to play in this context.
- Team meetings played a key role as a mechanism for coordinating, monitoring and dealing with the daily running of courses, particularly when teams were distributed. The cases under study had very different procedures for team meetings. The way in which course teams organised themselves was related to the number and location of the members of staff, and also to the style of leadership of the course directors. These factors seemed to be affecting the levels of formalisation, the meeting frequency, the mode of communication and ultimately the level of detail at which course issues were considered.
Evaluating their own team meeting's capacity as a mechanism for identifying and solving quality issues staff in all the case studies indicated that most of this work was actually done by informal contacts in casual settings (e.g. the corridor, a common office, a phone call or an occasional electronic message). They all seemed to agree however that formalisation was important, as was clearly expressed by one tutor:
…okay maybe you identified the issues through an informal conversation in the corridor but then if you don't have a structure to plug it into they somehow get lost… (Staff, CS1)
One general feature that characterised the way course teams coordinated and organised themselves was the strong reliance on online communications. There were only differences in the level of formality with which this online communication was taking place, varying from informal electronic mails to well established online discussion boards and seminars. Although this may seem a natural feature for an online course, the evidence suggested that online communications needed to be well coordinated and eventually backed up by face to face meetings in order to be fully effective.
An additional feature observed was that, although courses belong to campus-based institutions, teams were not only composed of campus-based staff, but increasingly by academics with fee-based contracts and tutors working from home or elsewhere. In this way, course teams happened to be scattered and course leaders seemed not to be aware, and hence not prepared, to cope with the coordination requirements of a distributed team. This situation was particularly evident when reviewing the mechanisms courses had for collecting feedback from tutors. In the cases where staff were mostly based on campus and hence face to face meetings were held regularly, tutors were fully integrated in the running of the course and habitually fed back their views regarding the modules and students. In contrast, in those courses with more off campus tutors, course leaders had mostly ad hoc information on which to base their decisions. As one course leader explained:
There is a atmosphere that people just let me know if there are problems and that is actually quite healthy and I suppose that you probably just relied on that and certainly well we have been still developing but I think we definitely need in… not just a feedback on materials and the pedagogical stuff that just generally for better sort of establishing feedback mechanism where we are not depending on just ad hoc people doing all things. (Staff, CS3)
The above situation suggests that leaders of online courses within campus based higher education institutions may not be prepared to deal with distributed teams, and were continuing to use the same mechanisms for coordination and feedback that were used for on-campus staff – trusting informal encounters as the main source to discuss issues related to the course.
As can be seen in the above account, from the point of view of the quality assurance procedures and their specific effectiveness to assure and enhance quality, the results suggest that three mechanisms were the strongly affected by the online modality of the courses: module evaluations, student representation and team meetings.
Conclusions
The findings suggest that the quality assurance procedures in place in dual mode higher education institutions require adaptation to be effective when applied to online courses. These findings show that the online modality and also the complex institutional environment in which the courses were located, were affecting the application of these quality assurance mechanisms, obstructing their capacity to assure and enhance the quality of the courses.
The findings also suggest that the quality assurance mechanisms in place in the cases studied were affected by the online modality of the courses to different degrees. From a general – and institutional- point of view, the results indicate that higher education institutions might need to approach the quality assurance and enhancement of their online courses from a different organisational perspective. Online courses seem to require a stronger definition of coordination, communication and planning strategies, as well as a clearly defined leadership, than face to face courses. The absence or limited clarity of any of these elements affected the effectiveness and enhancement roles of several of the procedures, in particular team meetings and students surveys.
An additional aspect suggested by the results is that institutions and course teams may need to consider with particular care the strategies to improve the amount and quality of student feedback. Online courses were particularly affected by a restricted access to students, which had a direct effect on the quantity of the feedback gathered and the appropriate representation of student views. In this sense, it seems that student representation, in its present form, is not a useful mechanism as a channel for student opinions. The data suggests however that the relationship established with tutors may be a route worth exploring for improving student representation.
References
[1] Ashwin, P. (2006), 'The development of learning and teaching in higher education'. In P. Ashwin (ed.), Changing Higher Education. The development of Learning and Teaching. Oxon: Routledge.
[2] Biggs, J. (2001), 'The reflective institution: assuring and enhancing the quality of teaching and learning'. Higher Education, 41, 221 - 238.
[3] Brown, R. (2004), Quality Assurance in Higher Education. The UK experience since 1992. London: Routledge Falmer.
[4] Connolly, M., Jones, N. and O'Shea, J. (2005), 'Quality assurance and E-learning: reflections from the front line'. Quality in Higher Education, 11 (1), 59 - 67.
[5] CVCP. (2000), The Business of Borderless Education: UK perspectives. Analysis and recommendations. London: The Committee of Vice-Chancellors and Principals of the Universities of UK.
[6] Harvey, L. (2002), 'The End of Quality?' Quality in Higher Education, 8 (1), 5-22.
[7] Harvey, L. (2005), 'A history and critique of quality evaluation in the UK'. Quality Assurance in Education, 13 (4), 263 - 276.
[8] Harvey, L. and Knight, P. T. (1996), Transforming Higher Education. Buckingham: SRHE & Open University Press.
[9] Hope, A. (2001), 'Quality Assurance'. In G. Farrell (ed.), The Changing Faces of Virtual Education. Vancouver: The Commonwealth of Learning.
[10] Inglis, A. (2005), 'Quality improvement, quality assurance, and benchmarking: comparing two frameworks for managing quality processes in open and distance learning'. International Review of Research in Open and Distance Learning, 6 (1).
[11] Massy, W. F. (1996), Teaching and Learning Quality-process Review: The Hong Kong Programme. Paper presented at the International Conference on Quality Assurance and Evaluation in Higher Education,, Beijing, China.
[12] Middlehurst, R. (1997), 'Enhancing Quality'. In F. Coffield and B. Williamson (eds), Repositioning Higher Education. Buckingham: SRHE and Open University Press.
[13] Middlehurst, R. (2001), Quality Assurance Implications for New Forms of Higher Education. Part 1: A Typology (ENQA Occasional Papers 3). Helsinki: European Network for Quality Assurance in Higher Education.
[14] Middlehurst, R. and Campbell, C. (2003), Quality Assurance and Borderless Higher Education: finding pathways through the maze. London: The Observatory on borderless higher education.
[15] Morley, L. (2003), Quality and Power in Higher Education. Berkshire: SRHE & Open University Press.
[16] O'Shea, T., Bearman, S. and Downes, A. (1996), 'Quality assurance and assessment in distance learning'. In R. Mills and A. Tait (eds), Supporting the Learner in Open and Distance Learning. London: Pitman Publishing.
[17] QAA. (2000a), Code of practice for the assurance of academic quality and standards in higher education. Section 5: Academic appeals and student complaints in academic matters. Gloucester: Quality Assurance Agency for Higher Education.
[18] QAA. (2000b), Code of practice for the assurance of academic quality and standards in higher education. Section 7: Programme approval, monitoring and review. Gloucester: Quality Assurance Agency for Higher Education.
[19] QAA. (2000c), Handbook for academic review. Gloucester: Quality Assurance Agency for Higher Education.
[20] QAA. (2002), Handbook for institutional audit: England. Gloucester: Quality Assurance Agency for Higher Education.
[21] QAA. (2004a), Code of practice for the assurance of academic quality and standards in higher education. Section 1: Postgraduate research programmes. Gloucester: Quality Assurance Agency for Higher Education.
[22] QAA. (2004b), Code of practice for the assurance of academic quality and standards in higher education. Section 2: Collaborative provision and flexible and distributed learning (including e-learning). Gloucester: Quality Assurance Agency for Higher Education.
[23] Roffe, I. (2002), 'E-learning: engagement, enhancement and execution'. Quality Assurance in Education, 10 (1), 40-50.
[24] Weyers, R. (2000), Distance learning zones: providing global information support to distance learners. Paper presented at the Libraries without walls 3 : the delivery of library services to distant users, Centre for Research in Library and Information Management (CERLIM), Manchester Metropolitan University.