Completion Rates – A False Trail to
Measuring Course Quality?
Let’s Call in the HEROEs Instead
Alastair Creelman [alastair.creelman@lnu.se], Linda Reneland-Forsman [linda.reneland@lnu.se], Department of Pedagogy, Linnaeus University, 391 82 Kalmar, Sweden [http://lnu.se]
Abstract
Statistics are often used to reveal significant differences between online and campus-based education. The existence of online courses with low completion rates is often used to justify the inherent inferiority of online education compared to traditional classroom teaching. Our study revealed that this type of conclusion has little substance. We have performed three closely linked analyses of empirical data from Linnaeus University aimed at reaching a better understanding of completion rates. Differences in completion rates revealed themselves to be more substantial between faculties than between distribution forms. The key-factor lies in design. Courses with the highest completion rates had three things in common; active discussion forums, complementing media and collaborative activities. We believe that the time has come to move away from theoretical models of learning where web-based learning/distance learning/e-learning are seen as simply emphasizing the separation of teacher and students. Low completion rates should instead be addressed as a lack of insight and respect for the consequences of online pedagogical practice and its prerequisites.
Keywords: completion rates, course design, online courses, web-based education.
The muddy waters of completion
Completion rates have been both a blindfold and a crutch in the move towards the collaborative and open-accessed higher education needed for the twenty-first century. In this article we argue that it is time to move away from a simplistic discussion centred on completion rates since they prevent us from addressing qualitative aspects of higher education regardless of distribution form.
In higher education today we find ourselves having to handle more students who in turn are less well-prepared than traditional campus students and who study fewer hours per week during an extended period of years (Sonneby, 2012). This relatively new student group is radically different from the campus population, having to combine part-time studies with full-time employment, family and other commitments. In addition we have the problem of student compliance with assigned readings which have declined dramatically (Starcher & Proffitt, 2011). Finally we lack pedagogically based approaches to courses and programs addressing the needs of an increasingly diverse student population (Bates, 2010). Could these circumstances be part of the solution rather than the problem?
In this article we present a three-part study of completion rates at Linnaeus University, Sweden and argue for a theoretical shift when discussing different distribution forms of education based on results (Creelman & Reneland-Forsman, 2012). We have performed three closely linked analyses to identify factors associated with completion rates. Online courses with low completion rates are often used to justify the inherent inferiority of online education compared to traditional classroom teaching only based on the distribution form. Our study revealed that this type of conclusion has little substance. Differences in completion rates revealed themselves to be more substantial between, for example, faculties than between distribution forms.
Previous research
In the transition to using interactive media for educational purposes two important aspects of addressing students’ meaning making often seem to challenge course designers: the role of social talk and non-linearity.
“...decentred communication has not led, in the case of e-University, to an acceptance of jointly constructed knowledge but rather to an emphasis on product guarantees. The fluidity of the internet, which media theorists proclaim as the condition for its creativity, has in fact fostered a pathological dislike of non-linearity...uncertainty has more often led to an aversion of risk rather than creativity. (Pelletier, 2005, p.17s)
Dealing with “social” has also proven more of a challenge than actually handling technology. Pitfalls are design issues of the social element such as taking communication for granted when interactive media is used and acknowledging the role of social but separating social communication from knowledge building processes (Kreijns et al., 2003). Social communication holds a key-role for implementing expectations on interactivity and dialogical conditions for learning (Jones & Issroff, 2005; Reneland-Forsman, 2009; 2012; in press).
Social talk is also a necessary ingredient in other important key factors traced to completion rates, such as creating realistic expectations and a clear structure from the outset (Mårald & Westerberg, 2006; Moore, 1993; Reneland-Forsman, 2009) as well as building a sense of community in which students will support and encourage each other to continue even when the going gets tough (Reneland-Forsman, 2009; 2012; in press). Jacqueline- Aundree Baxter’s (2012) study of completion rates at The UK’s Open University revealed that most students expected their online course to be self-study and were positively surprised when online collaboration was expected of them. Here the popular image of online education affects student expectations. Every student will bring with her/him their own preconceptions, fears, insecurities and previous academic experience and the success of each student depends on establishing a sense of belonging, a supportive environment and a can-do attitude. Students have widely varying levels of experience in online collaboration and some may feel inadequate at first. The teacher's role of setting the tone and providing reassurance to the hesitant is crucial in the early stages (cf. Reneland-Forsman, 2012).
Assessment and examination forms also contribute to the completion rates. Courses where students earn their credits step by step through regular assessed assignments had better completion rates than those where credits are earned in one final written examination (Högskoleverket, 2011a)
Online students who study mostly out of pure interest rather than for credits are often reluctant to invest the time required for a major end-of-course examination and therefore “drop out” before the exam. If they can earn their credits incrementally they are far more likely to complete the whole course (ibid).
Finally there is the issue of inactive students. The application process for university courses is long and many students who are accepted on a single optional online course may initially register for the course but then decide not to start for some reason. This can result in a course of 100 registered students actually having only 30 active students. The missing 70 never really start the course but since the Swedish government pays universities per registered student and a further sum based on course completion this has a major impact on higher education financing. At the end of the course there may be 25/30 students who are awarded credits but because the official registered number was 100 the completion rate is apparently 25 %. The Swedish National Agency for Higher Education (Högskoleverket, 2011b) reported that the proportion of campus-based inactive students is actually higher (42 %) than the percentage inactive online students (38 %).
For a richer account of what influences retention rates we also choose to address the notion of “access”. New generations of students are viewed as driving the concept of education “any time, any place and with anyone”, something that is much more achievable with today’s technology (Högskoleverket, 2008). At the same time there is a tendency to see a lower level of study skills and capacity in many students entering higher education today (cf. Sonneby, 2012). Access to technology does not in itself indicate an open road to the future since it is still to a large extent the already privileged in society that are able to fully exploit the opportunities offered by technology for empowerment and participation in adult learning. Many voices problematize a “digital divide” as having deeper and more complex implications than a divide between those with technological access or not problematizing the sustainable development of wider participation (Eynon & Helsper, 2011; Eynon & Malmberg, 2011; Jenkins et al., 2009; Livingstone et al., 2009; Livingstone & Helsper 2007; Peter & Valkenburg, 2006; Thomas, 2009). ICT risks at best increasing levels of participation within social groups that were already learning anyway (Selwyn, Gorard & Furlong, 2006). Several studies identify different aspects of digital literacy and an emerging digital differentiation based on level of knowledge rather than an access to technology. Non-participation or failures in adult and lifelong learning are deeply entrenched in ‘trajectories’ based on class, gender, generation, ethnicity and geography, which are established at an early age (Selwyn, 2009; Tuckett & Aldridge, 2009).
Survey of completion rates
The first study was a comparison of completion rates between different distribution forms. We examined the completion rates of all courses at the university for 2010 (both degree program courses and optional courses) using a variety of distribution forms (campus-based, online and blended learning with support from local learning centres). The results showed a wide range of completion rates with online courses having completion rates of 25-90 %. A major difficulty in using official completion rates is the fact that they simply measure throughput on a calendar year basis. Most courses in the autumn term are not formally completed until mid January and the credits are not registered until later on. Due to this the completion rates for autumn courses may appear to be very low whereas spring figures can often exceed 100 %. We concluded that these annual completion rates were not a reliable measurement of the success or failure of a course.
The overall tendencies we found in our study were as follows:
- There was no clear correlation between low completion rates and online education. There was similarly wide variation in completion rates between faculties and within campus-based courses.
- Two faculties had higher completion rates for their online degree programmes than their campus-based programmes.
- Optional online courses showed significantly lower completion rates in comparison to their campus-based equivalents. Courses that were part of a degree programme had higher completion rates in both campus and online formats.
Although there were many examples of online courses with a low completion rate there were also many highly successful examples with completion rates over 90 %, in some cases higher than the equivalent campus course. Courses that offered professional development in fields such as teacher training, health care or special needs education where successful completion had a direct impact on the students’ career development generally had much higher completion rates compared to courses that could be seen as simply “good to know” (e.g. basic courses in project management, psychology).
Interviews with teachers and administrators showed that this was a common situation and a clear weakness in the system. Related to this is the fact that many online students take optional courses simply out of interest in the subject and are not necessarily interested in gaining credits. These students may seem to have failed to complete the course but are perfectly satisfied with the new knowledge they have gained. This type of course can be seen as contributing to lifelong learning rather than as part of a formal education. The Swedish government has recently proposed to tighten up this loophole by insisting that universities de-register inactive students after 3 weeks of the course and that these students will not be counted in the final completion rate statistics.
Another significant factor in this analysis is the convergence of campus and online learning. The use of technology in education (VLE, recorded lectures, e-mail, instant messaging etc) has become ubiquitous and is equally important on campus as it is in distance education. Many full-time students take extra online courses in order to increase the credits they can earn for the term and we could see that a significant number of students on the universities online courses had addresses in the two campus towns. However many campus students overestimate their work capacity and discover after a few weeks that the online course is simply too demanding. As a result they drop out of the course. Is this a problem with online learning or a campus problem?
Survey of inactive students
The second analysis was a survey of this important category of registered but non-active students. We sent a short web-based questionnaire to 450 inactive “drop-out” students from five different departments at the university (from all faculties). The aim of the survey was to find out the reason for dropping the course and whether it was due to insufficient information or problems with university administration or technology. The survey completion rate was 31 % and the results showed that the most common reasons for not starting the course were:
- Difficulty combining course with work commitments
- Chose another course instead
- Family situation (illness, bereavement, stress)
- Lack of time
Over 80 % were pleased with the information they had received from the university and the admissions process and a similar number had not experienced any significant technical difficulties.
This further supports the notion that comparisons between online and campus courses are problematic. Students on online courses belong to a quite different target group to those on campus-based courses. Online students are generally over 25, already have a university degree, have families and full-time employment and are not dependent on loans or other subsidies to finance their studies (Högskoleverket, 2008). They tend to view online courses as lifelong learning and are not as highly motivated to gain credits as campus students who depend on successful course completion for the financing of their future studies. Few universities have fully grasped this vital difference and have developed specific strategies for their online prospectus.
Course design analysis
Thirdly a course design analysis was carried out on 12 courses (both degree programme courses and optional stand-alone courses).
The pedagogic design analysis was inspired by the Theory of Transactional Distance (Moore, 1993) since this model is useful to challenge an instrumental approach to technology and instead focus on the conditions for learning associated with different outcomes of course design and didactical considerations. The Theory of Transactional Distance describes pedagogical distance as a relation between structure, autonomy and communication. Courses with low interaction depend on high autonomy of students whereas courses with intense communication through motivational and structural communicative actions can compensate for lack of structure or autonomy. A further influence on the analysis has been studies on meaning making and the role of social talk for knowledge building (see Krejins et al., 2003; Reneland-Forsman, 2009; 2012).
Guiding the analysis were levels/degrees/variations in:
- level of interaction
- flexibility and change in web-based interface
- modalities used
- evidence of collaboration
- evidence of synchronous meetings
- structure and lucidity
Although based on a small material of 12 courses some tendencies were possible to observe on design aspects of courses. The courses with the highest completion rates had three things in common: active discussion forums, complementing media and collaborative activities. Courses with a higher completion rate than the national average (3 out of 12) all had some form of synchronous meeting. Alarmingly enough 10 out of 12 courses lacked structural pedagogic guidance – an orientation for students helping them to overlook the course and create meaning around it (cf. Mårald & Westerberg, 2006; Moore, 1993).
An alarming issue arising from our study was the significant lack of represented structure and orientation guiding students through a course. There are different ways of addressing the question of making visible a course structure and epistemology. This could be done using visual clues in the interface, such as a study guide or using a question approach as examples. The function however is crucial (cf. Mårald & Westerberg, 2006; Moore, 1993). This is often supplied by study guides but could also be offered in the internet interface. Designing for individual tracks makes it more difficult to use a community for structure and guidance. Students might be addressing diverse aspects of a course. If this is the case, interaction rates often drop and students become more dependent on self-autonomy with a well-presented structure keeping them active and on track. The same problems probably occur in traditional distribution forms but are often resolved in physical encounters and activities which provide orientation for students. Programs with higher completion rates than traditional distribution forms all had a long tradition of providing students with structured guidance in good time and a vocational orientation (based on the overall completion rate analysis, see also SCB 2012).
Courses and programs had a strong linear text-orientation. Few other representational modes were used for students’ active content processing. Courses therefore lacked variations in terms of resources that could be used for jointly constructed meaning (cf. Pelletier, 2005). With few exceptions PDF files were piled in virtual learning environments. When film was used these were filmed lectures that can be viewed over again. Summing up, analysis confirms known pitfalls (Jones & Isroff, 2007; Krejins et al., 2003, Pelletier, 2005). The use of technology reinforces a traditional content distribution model of teaching rather than supporting the students’ learning process. A recent Norwegian report on the use of IT in higher education indicates that technology is mostly used to support a traditional content delivery mode rather than developing collaboration and student-driven learning (Norgesuniversitet, 2011; see also Pelletier, 2005).
There is a widening digital gap between students’ use of social media to collaborate and share and universities’ inability to exploit the communicative possibilities of social media.
Some reflections
Our study of online education at Linnaeus University questioned the validity of judging a course’s success on completion rates since it was unclear what was actually being measured. Results suggest that crucial factors behind student retention are the presence of meaningful interaction (via face-to-face or virtual meeting spaces), clear timely information about course requirements, methodology and expectations and a clear link between technology and pedagogics. The study made the following proposals:
- A quality assurance system adapted to the demands of online learning should be implemented across the whole university. Firstly based on self-assessment with the option of certification later
- To create a university strategy for integrating technology in all educational activities
- To clearly define the target group for online courses and adapt administration and marketing to that group.
- Competence development and support for all teachers in IT and learning.
- Clear routines for the deregistration of inactive students after 3 weeks of a course.
- Better information and study guides for all online students.
The consumption and distribution of education
Despite the vast opportunities that social media offer to foster collaboration, sharing, creation and adaptation, the formal education sector in most countries remains trapped in the traditional classroom paradigm with the teacher/school as the provider of content and the students as consumers. Despite the affordances of technology, courses are mostly based on linear progression and content delivery and examination therefore focuses on the ability to recall and structure facts. Credits have become the hard currency of education and students are naturally focused on learning enough to earn credits rather than acquiring the skills they need for the future.
Such a system creates a state of learned helplessness where learners expect to be consumers of knowledge delivered by schools and colleges. Students in the traditional model can be seen as charter holiday tourists, consumers of a product, where courses, learning outcomes, methods, course literature and examination forms are pre-determined. At the same time, industry representatives are increasingly calling for backpackers: students who have learned how to find and filter information for themselves as well as being able to network, collaborate and be creative in virtual communities.
Technology has so far mostly been used to reinforce traditional classroom practices as shown by the popularity of lecture capture, learning management systems (virtual classrooms) and smartboards. The current wave of content-based MOOCs (Massive Open Online Courses) offered by consortia of high status universities (Coursera, edX, Udacity, etc.) may seem innovative but are still heavily based on recorded lectures and traditional course structure and difficult to compete with for less prosperous institutions.
A time for HEROEs
Many of the problems discussed here can be attributed to a lack of insight and respect for the consequences of online pedagogical practice and its prerequisites. Up till now the main applications of technology in education have been to duplicate existing practices (recorded lectures, virtual classrooms, smartboards) or provide a reasonable substitute to the physical environment which is seen as default. There are too many misconceptions about the conditions for communication and meaning making among humans ascribed to the room or the physical place itself. When it comes to the digital arena we seem unable to fully think through the consequences of our choice of digital space. The net offers a wide range of new opportunities for collaboration, creation and analysis that are not being sufficiently exploited in higher education today. This problem can be reduced by professional development initiatives based on development and exchange of experience between staff members and the acknowledgement of changed conditions for learning.
It is time to move away from theoretical models of learning where web-based learning/distance learning/e-learning are seen as simply emphasizing the separation of teacher and students. This results in the notion of campus education as a “shared environment” and norm whereas digital media are simply a means of communicating over geographical distances. This line of reasoning carries underlying assumptions on the normative character of physical environments as fully shared conditions for meaning making and human communication.
To address the challenges of the 21st century and a more diverse student population, with the implied consequences for groups that lack dispositions for self-empowerment in the new learning cultures described here, we need informed educators that can move from campus education as the unquestioned norm towards students’ meaning making processes and act upon different affordances for meaning making and learning in particular contexts (Figure 1A & 1B).
Figure 1A. The move towards the norm Figure 1B. The move towards meaning
Let’s call in the HEROEs (Highly Empowered Resourceful Online Educators) which means once and for all abandoning a consumerist approach to education applying a meaning-oriented approach. Acknowledging the design effects of learning environments means using a variety of means to trigger students’ cognitive resources. The focus on students’ active social and knowledge building processes regardless of distribution form also highlight the use of digital media in higher education as opportunities to provide experiences and orientation in a course or program. HEROES would analyze the conditions for knowledge building processes regardless of pedagogical practice. HEROEs would invite students to drag material into a learning environment, thus opening up the oyster – making students co-creators. As Wiley (2007) concurs, for the educator much of the learning, both about the subject and how to teach it, comes from the process of creating the object. Co-creation of knowledge using course wikis or by students collaborating around the creation of user-generated content are examples of such processes. Thus we move from the linear, content-based course to a fundamentally different model; the creation of a learning arena where assessment is based on successful completion of projects and where networking and dialogue are essential success factors. In such a connectivist environment the traditional learning hierarchy is evened out and students take more responsibility for their learning.
The use of completion rates to measure the relative success of a course builds on a number of assumptions based on the traditional campus-based model for higher education. It assumes that students are studying for academic credentials and that this process also has financial motivation. As soon as net-based courses are aimed at a wider audience (lifelong learners, professional development, general interest) for whom the rewards of university credits are less appealing than the learning of new skills, the concept of completing a course becomes less relevant. The low completion rates of many online courses and in particular today’s mainstream MOOCs is not a credible gauge of course quality. We need to move the focus from a simplistic head count to developing strategies for increased interaction, collaboration and collective responsibility in online learning – HEROES could do the trick.
References
- Bates, T. (2010). New Challenges for Universities: Why They Must Change. In Ehlers & Schneckenberg (eds.), Changing Cultures in Higher Education: Moving Ahead to Future Learning, (pp.15-35). Springer: London & New York
- Baxter, J. (2012). Who am I and what keeps me going? Profiling the distance learning student in higher education. In International review of research in open and distance learning, 13(4), http://www.irrodl.org/index.php/irrodl/article/view/1283/2292
- Creelman, A. and Reneland-Forsman, L. (2012). Genomlysning
av distansverksamhet vid Linnéuniversitetet. Linnaeus University. Accessed
online 26 Aug 2013 at:
https://medarbetare.lnu.se/organisation/universitetspedagogiska-enheten/1.64404/ny-rapport-genomlysning-av-distansverksamhet-vid-lnu - Eynon, R. and Malmberg, L.-E. (2011). A typology of young people’s Internet use: Implications for education. In Computers & Education, 56, (pp. 585-595).
- Eynon, R. and Helsper, E. (2011). Adults learning online: Digital choice and/or digital exclusion? In New Media Society, 13(4), (pp. 534-551).
- Högskoleverket (Swedish National Agency for Higher Education) (2008). E-learning quality: Aspects and criteria for evaluation of e-learning in higher education. Stockholm: report 2008:11R
- Högskoleverket (The Swedish National Agency of Higher Education) (2011a). Kartläggning av distansverksamheten vid universitet och högskolor. Stockholm, rapportserie 2011:2 R
- Högskoleverket (The Swedish National Agency of Higher Education) (2011b). Årsrapport. Universitet & högskolor, Stockholm, HSV Rapport 2011:8 R
- Jenkins, H.; Purushotma, R.; Weigel, M.; Clinton, K. and Robinson, A.J. (2009). Confronting the Challenges of Participatory Culture. MIT Press: London
- Jones, A. and Issroff, K. (2005). Learning technologies: Affective and social issues in computer supported collaborative learning. In Computers & Education, 44(4), (pp. 395–408).
- Kreijns, K.; Kirschner, P.A. and Jochms, W. (2003). Identifying the pitfalls of social interaction in computer-supported collaborative learning environments: a review of the research. In Computers in Human Behaviours, 19(3), (pp. 335–353).
- Livingstone, S.; Van Couvering, E. and Thumin, N. (2009). Converging Traditions of Research on Media and Information Literacies: Disciplinary, Critial and Methodological Issues. In J. Coiro, M. Knobel, C. Lankshear, & D.J. Leu (eds), Central Issues in New Literacies and New Literacies. Research Handbook of research on new literacies. New York: Routledge.
- Livingstone, S. and Helsper, E. (2007). Gradation in digital inclusion: Children, young people and the digital divide. In New Media and Society, 9(4), (pp. 671–696).
- Moore, M.G. (1993). Theory of transactional distance. In Keegan (ed), Theoretical Principles of distance education, (pp. 22-38). London: Routledge.
- Mårald, G. and Westerberg, P. (2006). Nätuniversitetets Studentnytta – slutrapport I från en 3-årig utvärdering. Umeå Centre for Evaluation Research Evaluation Reports No18 December 2006
- Norgesuniversitet. (2011). Digital tilstand i høyere utdanning. 1/2011
- Pelletier, C. (2005). New Technologies, new identities: The University in the informational age. In R. Land & S. Bayne (eds.), Education in Cyberspace. Routledge: London
- Peter, J. and Valkenburg, P.M. (2006). Adolescents’ internet use: Testing the “disappearing digital divide” versus the emerging “digital differentiation” approach. In Poetics, 34, (pp. 293-305).
- Reneland-Forsman, L. (2009). A Changing Experience: communication and meaning-making in web-based teacher training. Diss. Växjö: Växjö University Press.
- Reneland-Forsman, L. (2012).Toward a broader understanding of social talk in Web-based courses. In Text & Talk, 32(3), (pp. 349–369).
- Reneland-Forsman, L. (In Press). Students’ web-based actions when linking theory and practice. In International Journal of Web Based Communities
- Selwyn, N. (2009). The digital native – myth and reality. In Aslib Proceedings: New Information Perspectives, 61(4), (pp. 364-379).
- Selwyn, N.; Gorard, S. and Furlong, J. (2006). Adult learning in the digital age: information technology and the learning society. London: Routledge.
- Sonneby, P. (2012). Lärda för livet? En ESO-rapport om effektivitet i svensk högskoleutbildning. Rapport 2012:7. Finansdepartementet, Regeringskansliet. http://www.eso.expertgrupp.se/Uploads/Documents/8-oktober-2012/ESO2012_7_webb.pdf
- Starcher, K and Proffitt, D. (2011). Encouraging Students to Read: What Professors Are (and Aren’t) Doing About It. In International Journal of Teaching and Learning in Higher Education, 23(3), (pp. 396-407).
- Thomas, A. (2009). Community, Culture, and Citizenship in Cyberspace. In J. Coiro, M. Knobel, C. Lankshear, and D. Leu (eds.), Handbook of research on new literacies, (pp. 671-697). New York & London: Routledge.
- Tuckett, A. and Aldridge, F. (2009) Narrowing Participation. The NIACE Survey on Adult Participation in Learning. Leicester: NIACE
- Wiley, D. (2007). On the Sustainability of Open Educational Resource Initiatives in Higher Education. COSL/USU Paper commissioned by the OECD’s Centre for Educational Research and Innovation (CERI) for the project on Open Educational Resources. Accessed online on 19 April 2010 at: http://www.oecd.org/dataoecd/33/9/38645447.pdf