NEW

Best of EDEN 2018 Special issue
 

There are not any recent contributions.

There are not any recent contributions.

Archives

EURODL Mailinglist

2851 subcribers
 

EURODL Visitors

back

Quality in e-Learning from a Learner's Perspective

(Best Paper Award at the Third EDEN Research Workshop 2004, Oldenburg, Germany)
Ulf-D. Ehlers (uehlers@wi-inf.uni-essen.de)
University of Duisburg-Essen (http://wip.wi-inf.uni-essen.de)
Campus Essen; Universitaetsstr. 9; 45141 Essen; Germany

Abstract

When you really get down to analysing it, the promises of E-Learning often have yet to materialize. The question how e-learning can be successful becomes more urgent as we move from an 'early adopter' stage to a more general offering. In the discussion about the best strategy for e-learning it becomes more an more clear that e-learning has to be based on the learner. This includes the necessity to postulate in a clear way that the needs of the learners have to be determined in a concrete manner before starting the project. Important aspects are therefore the awareness of the learning biography, of individual learning preferences and of social needs.

It is important to acknowledge that quality of a learning process is not something that is delivered to a learner by an e-learning provider but rather constitutes a process of co-production between the learner and the learning-environment. That means that the product/ outcome of an educational process is not exclusively a result of the production process of an educational institution. Quality therefore has to do with empowering and enabling the learner. It has to be defined at the final position of the provision of the learning-services: the learner. The article describes learners preferences in e-learning based on empirical results of today's largest survey in this field [1]. It thus facilitates the construction of learner oriented services portfolios in e-learning.

Was macht E-Learning erfolgreich? Diese Frage steht am Anfang einer Vielzahl von Auseinandersetzungen zum Thema Qualität im Bereich des E-Learning. In vielen Diskussionen und Vorträgen zeigt sich einerseits die zunehmende Bedeutung, die dem Thema Qualität beim E-Learning im Allgemeinen - und lernerorientierter Qualitätsentwicklung im Speziellen - beigemessen wird. Andererseits ist jedoch vielfach eine große Unsicherheit in Bezug auf diese (neue?) Lernform zu spüren, sowohl bei Praktikern der Weiterbildung als auch bei Entwicklern, Autoren und Lernern: Dozenten sehen sich mit einer neuen Rolle konfrontiert, die sie zum Begleiter und Moderator von Lernprozessen macht. Softwareentwickler müssen bei der Konzeptionierung und Implementierung von Lernsoftware zunehmend stärker über die eigenen disziplinären Grenzen hinweg gehen und interdisziplinären Austausch mit Pädagogen, Autoren und Lernern suchen. Von Autoren wird verlangt, zukünftig nicht mehr in stringent aufeinander aufbauenden Lerndramaturgien zu denken, sondern Lernmodule zu erstellen, die dekontextualisiert und somit besser wiederverwendbar sind. Auf Lernerseite stellt sich in einem sich stetige ausdifferenzierenden Markt die Frage danach, welches die wichtigsten Merkmale für gute E-Learning-Angebote sind und welche Anbieter die besten Leistungen für einen angemessenen Preis bieten. Diese sehen sich wiederum mit einer immer weiter fortschreitenden Didaktisierung einer ursprünglich technologischen Entwicklung - und einer sich verstärkenden Lernerorientierung konfrontiert.

Eine umfassende Studie¹ zeigt nun erstmals, wie wertvoll es ist, einmal die Lerner zu Wort kommen zu lassen. Nicht technologieorientiert sondern lernerorientiert (und technologieunterstützt) muss erfolgreiches E-Learning zukünftig gestaltet sein. Anforderungen von Lernern an E-Learning-Arrangements bilden hier Gegenstand der Forschung. Die umfassende Forschungsarbeit geht davon aus, dass in Zukunft eine konsequente Qualitätsforschung aus der Lernerperspektive notwendig ist. Dabei kommt es insbesondere darauf an, Determinanten eines Qualitätsbegriffes aus Sicht der Lerners zu ermitteln und soweit zu konkretisieren, dass sie in die Konstruktion von konkreten E-Learning-Angebote überführt werden können. Die zugrunde liegende Fragestellung lautete daher: Was sind bedeutsame Faktoren für die Qualität beim E-Learning aus Lernersicht?

Keywords

Quality, E-Learning, Learner-orientation, Learner's preferences, empirical model, subjective quality

1 Introduction

What makes e-learning successful? This question arises at the beginning of a large number of debates on the subject of quality in e-learning. On the one hand, the increasing importance attached to the topic of quality in general is evident in many publications, discussions and lectures. On the other hand, however, there is also great uncertainty among decision-makers and managers as well as among developers, trainers and learners: instructors find themselves confronted with a new role in which they are tutors and facilitators for learning processes. Software developers more and more have to go beyond the paradigms of their own discipline when designing and implementing learning software; they are in need to seek interdisciplinary exchange with teachers, authors and learners. Authors are required to think in a new way: no longer the instructional material is built in a series of straight consecutive units where each presentation is based on the preceding one, but learning modules that are decontextualized and therefore easier to reuse are to be created. On the learner's side, the question arises which characteristics are most important for good e-learning-environments and which providers offer the best performance at a reasonable price in a market that is continuously differentiating further. Learning Management System (LMS) providers, for their part, find themselves confronted with the continually progressing didactization of the technological "delivery structure" of e-learning and are thus faced with an increasing learner orientation (cf. Ehlers et al. 2004).

2 The Learners Perspective in the Quality Debate

Finding answers to questions regarding quality in e-learning is one of the central challenges for theory and practice if e-learning is to become as important as traditional qualification measures in the future. The question arises how such a complex concept as quality, can be conceptualised systematically. Three different dimensions can be distinguished here (cf. Ehlers 2002a, 2002b, 2003a): different meanings of quality, different quality perspectives and different levels of the educational process to which quality can apply (Figure 1).

Figure 1: Multiple dimensions of the quality concept in e-learning (cf. Ehlers 2004)

First of all quality can be distinguished into several different levels. According to quality and evaluation research an educational process can be subdivided into five subsections or sub-processes [2]: context-quality, structure-quality, process-quality, output-quality or impact-quality. Quality applies to each of those sub sections differently. All the same we have to take into account the different perspectives on quality that were already referred to above (see chapter 1). And last but not least it is important to clarify different semantic understandings of what quality actually means: If quality is understood in the way of excellence we can distinguish it from quality in the sense of usability or value for money [3]. Defining quality thus means positioning oneself in this multi-dimensional space (Figure 1). There is no patent remedy and no universally applicable, standard perspective for developing or assuring quality. Quality development always has to take different perspectives and different meanings into account.

In the presented work the perspective of the learner is focussed on. This goes along with a new awareness in the field of quality research in education which emphasizes the importance of a subjective research perspective. In research as well as on the level of policy makers, in industry and private educational contexts we can observe an uprising orientation at the needs of individual competence development and learning processes. Life-long learning processes can no longer be standardized but have to be constructed along individual training biographies (Robinson/ Arthy 1998). The situation today is not only characterized by the importance of knowledge and information, but the acquisition and application of it and the ability to generate, process and communicate knowledge and information using technological tools (Castells 1996) - skills that have to be developed according to personal preferences and usage contexts.

Apart from empirical evidence there are also more general/ structural reasons to focus on quality from a learner's perspective: As well as in the area of services in general quality in learning has to be considered as a co-production process between the learning-environment and the learner - and is thus part of his/her own responsibility. A learning process is not something that is delivered to a learner by an e-learning-provider but rather constitutes a process of co-production between the learner and the learning-environment. That means that the product/ outcome of an educational process can not exclusively be influenced by the 'production processes' of an educational institution. This differentiates the field of education and services in general from the trading/market mechanisms between consumer and producer according to the conventional market paradigm: Education can not be traded or bought by the clients/ learners; learning rather constitutes a process that they have to carry out by themselves.

The definition of quality therefore has to be defined at the final position of the learning-services, as there is the learner. Of course this does not mean that the learner's perspective and preferences alone has to be taken into account: economical, organisational or even legal regulations have to be considered. Figure 2 shows that a learning environment can be conceptualised by four different components which each incorporate different perspectives on quality. Quality development from a learner's perspective therefore means to take the learners preferences as the starting point of quality development in all other areas.

Figure 2: The Learners position in a learning environment
(Paradigm of constructing and evaluation learning environments according to Fricke 1995)

Therefore the orientation on the learners needs a realistic economical basis more and more becomes a leading concept in quality research. The presented survey 'Quality from a learners perspective' analyses quality from exactly this point of view. Two goals are being aimed at: First, the research is concerned with the question (1) which dimensions are constituting quality in e-learning from a learner's perspective; that is: which quality aspects, dimensions or criteria are important for learners? Secondly, quality is no longer viewed as a concept in which the same quality approaches or quality criteria apply to all learners, but rather (2) where different learners have different preferences regarding quality in e-learning. The empirical results clearly confirm this hypothesis. For the first time at all the research project makes available an empirical based classification of subjective quality concepts in e-learning. Thus now there is a basis for learner focussed quality development in E-Learning.

The survey used qualitative data as well as quantitative data. The connection between these different approaches - also known as the methodological concept of triangulation (cf. Treumann 1998) - enables researchers to gain a more in-depth insight into the field of subjective quality preferences. The survey was conducted in two phases: First a qualitative inventory was gathered from interview data of 56 interviews with learners who had considerable experience in the field of e-learning. This inventory formed the bases for a quantitative research phase in which 2000 learners answered a questionnaire on their quality preferences for e-learning (n=1994). The data were then analysed using multivariate statistics - principal component analytical and cluster analytical methodology.

Comprehensive E-Learning Services for Learners

A learner focussed quality concept has to be more comprehensive than just focussing on aspects of instructional or technological interface design. This is clearly shown in the results of the survey. Accordingly subjective quality requirements can be structured in 7 fields of quality:

Figure 3: Model of subjective quality requirements (cf. Ehlers 2003)

The so-called subjective model of quality is organized in a three level structure. 153 factors of subjective quality form the basis of the empirical model. They are the result of an in-depth oral interview inquiry with learners. This inventory then was - on the basis of the data of a standardized online survey - structured into 30 dimensions of quality in E-learning (Principal Component Analysis). The dimensions represent bundles of factors that - empirically - belong together (correlate). On the top level the resulting 30 dimensions are then structured into seven fields of subjective quality according to thematic resemblance.

The dimensions are the result of a principal component analysis (PCA). This method allows reducing the variety of many factors (153) to few powerful bundles of factors - or: dimensions - that can explain the differences in the quality preferences of the learners. It is important to notice that the 30 preference dimensions are not all equally important to learners. They rather form a grid of dimensions that can be of relevance to a specific user. For each user the described dimensions therefore are likely to be of different importance for quality in E-Learning. In a way those dimensions are the line along which users can be different in their quality preferences.

Each of the 30 dimensions represents a set of criteria of learners preferences that are clustered to a dimension on basis of empirical evidence. In the following section a more detailed overview on the 30 dimensions of subjective quality will be given. They are presented according to the 7 fields of quality they each belong to.

Quality Field 1: Tutor Support

In this field quality preferences are represented that learners have towards the communication and cooperation with the tutor of an online course. The survey shows that tutor support is very important for learners in general - regardless their other preferences: Between 74,4% and 97,7% of learners in the different preference groups (Figure 4) value tutor support in general as 'important' or 'very important'. However - there are great differences between the learners as to how the tutor support should be performed. Their preferences differ along the following dimensions:

Learners do not only vary according to their preference sets in regard to tutorial communication and cooperation performance but also in the kind of media they wish to use for getting in contact with tutors.

Quality Field 2: Cooperation and Communication in the Course

This field contains quality requirements that learners express towards the course concerning the communication and cooperation environment with other learners in learning groups, with experts or the tutor.

Quality Field 3: Technology

The field of technical requirements can also be important to learners in regard to the following three dimensions. Technical requirements however seem to have the status of hygienic factors. That means that if technical requirements are fulfilled they do not raise the perceived quality very much - as they are taken for granted. Yet if the expected technical standards are not met the learners quality assessment decreases.

Quality Field 4: Costs - Expectations - Value

The cost-value assessment of learners is one factor that determines their quality judgment of an E-Learning-Course. It can be seen in close relation to the expectation learners have when entering a course setting, which determine their outcome assessment. The cost and effort that learners have to assign to the learning experience has to be in a subjective adequate relation to the benefits and the outcome.

Quality Field 5: Information transparency

Another field of quality preferences refers to the information possibilities learners have about a course or an institution/ organization which is offering the course. It contains the provision of formal and standard information as well as individualized counselling on course contents, learning methodology or technical advise.

Quality Field 6: Course structure

This field contains learner's requirements concerning the structure of an e-learning course. Learner's quality preferences clearly show that presence lessons as part of an e-learning-course (blended learning) are of high importance to certain groups of learners whereas others do not value them as important. The dimensions of this quality field summarizes, which functions theses presence courses have from a learners perspective.

Quality Field 7: Didactics

The research shows that a lot of quality preferences that were analysed fall into this category. It covers aspects of content, learning goals, methods and materials. Experienced e-learners are often very precise in their requirements concerning the didactical setting of an e-learning course.

Learners Quality Preferences: Individual - but Still Manageable

It is now obvious that learners quality needs can be described in terms of their preferences towards each of the 30 specified dimensions. On basis of the empirical data it is thus possible to construct an individual preference profile for each learner. Every learner has a specific value on each of the previously described dimensions. In a next research step the data was analysed to understand if it was possible to find similarities in users quality preferences. Therefore an additional analysis of the gathered data was performed with a subset of cases (n=1321) using cluster algorithms (ward & k-means).

Each users individual preference profile was analysed and compared to those of other learners in order to find resembling profiles that could be combined to groups that are alike concerning their quality needs - but which on the other hand are different to other groups (cluster analysis). The result shows that four target groups can be identified. Figure 4 shows the four target groups and gives their main characteristics. The four target groups differ very much in their demands for communication and tutor support as well as group activities and social contacts in an online course.

Figure 4: Four quality specific target groups

Conclusion

Two results have been contributed to the debate of quality in e-learning: An empirical model representing learners preferences in 30 dimensions and an analysis and description of four preference profiles. The research clearly shows that learners distinguish their quality preferences in e-learning. All learners had considerable experiences in e-learning and can be grouped into profiles according their quality preferences.

This leads to the conclusion that future quality development in e-learning has to be oriented at the learners needs and situation. No longer general criteria or the same guidelines for all learners can be applied but individual learning services are needed that support learners according to their subjective preference profile.

Annotations

[1] The complete survey will be published in July 2004: Ulf-Daniel Ehlers (2004). Qualität beim E-Learning aus Lernersicht. Grundlagen, Empirie und Modellkonstruktion subjektiver Qualität. Wiesbaden
[2] An in depth analysis of concepts from evaluation and quality research referring to Stufflebeam 1972, 2000, Auer/ Carstensen 1995, Clayton/ Cattarello 1991 and Donabedian 1980 by Ehlers 2003a shows that currently five levels of quality in education can be distinguished.
[3] For different semantic quality understandings see Ehlers 2003a.

^{1 Footnote}: Die Studie erscheint im Juli 2004 im Verlag für Sozialwissenschaften in der Reihe Bildung und Medien: Ulf-Daniel Ehlers: Qualität beim E-Learning aus Lernersicht. Grundlagen - Empirie und Modellkonstruktion subjektiver Qualität. Wiesbaden. Germany

References

1. Ehlers, U. (2002a). Qualität beim eLearning ist mehr als 'e'. Studie der Uni Bielefeld: Der Lerner steht im Mittelpunkt. In. Bundesinstitut für Berufsbildung (ed.), BWP - Berufsbildung in Wissenschaft und Praxis. Heft 3/2002. Bonn.
2. Ehlers, U. (2002b). E-Learning: Stand und Perspektiven. Gutachten für Verbraucherzentrale Bundesverband e.V. Bielefeld.
3. Ehlers, U. (2003a). Qualität beim E-Learning. Empirische Grundlegung und Modellkonzeption einer subjektorientierten Qualitätsentwicklung. Dissertation. Bielefeld.
4. Ehlers, U. (2004). Quality in E-Learning. The Learners Perspective. In: European Journal of Vocational Training. CEDEFOP. Thessaloniki.
5. Fricke, R. (1995). Evaluation von Multimedia. In Issing, L.J., Klimsa, P. (ed.) (1995). Information und lernen mit Multimedia. Weinheim.
6. Treumann, K. P. (1998): Triangulation als Kombination qualitativer und quantitativer Forschung. In: Abel, J., Möller, R., Treumann, K. P. (1998). Einführung in die Empirische Pädagogik. Stuttgart u.a., S.: 154-188
7. Castells, M (1996): The rise of the network society. Blackwell. Oxford.
8. Robinson, C & Arthy, K (1998). Lifelong learning, developing a training culture: Creating our future: A new training culture for Australia, NCVER. Adelaide.
9. Clayton, R.R., Cattarello, A. (1991). Prevention Intervention Research: Challenges and Opportunities. In: Leukefeld, C.G., Bukovsky, W.J. (ed.) (1991). Drug Abuse Prevention Intervention Research: Methodological Issues. NIDA Research Monograph 107. Rockville.
10. Donabedian, A. (1980). Explorations in Quality Assessment and Monitoring. Ann Arbor.
11. Stufflebeam, D.L. (1972). The relevance of the CIPP evaluation model for educational accountability, SRIS Quarterly, 5(1), 1972
12. Stufflebeam, D.L. (2000). The CIPP model for evaluation. In: Stufflebeam, D.L., Madaus, G.F., Kellaghan, T. (ed.): Evaluation Models. Boston.