Developing Critical Perspectives on Technology in Education: A Tool for MOOC Evaluation

Antonella Poce

back

Developing Critical Perspectives on Technology in Education: A Tool for MOOC Evaluation

Antonella Poce, Università Roma TRE, Italy

Abstract

Keywords: Online learning, MOOCs, Evaluation, Critical thinking, Lifelong Learning.

The Context

The Laboratory for experimental pedagogy (LPS) based at the Department of Education – Roma Tre University has been working, since 2010, on research focusing on the enhancement of students’ critical thinking skills to foster the development and promotion of the critical use of technology in education. A series of departmental projects, coordinated by LPS researchers, have been funded from 2011 to achieve these aims (Poce et al., 2011; Poce, 2012; Poce, 2014). The projects use specific models and coordinated approaches to teaching and learning across a range of disciplines. Students are invited to engage in learning activities, which involve analysis and reflection, individually and in groups, taking into considerations the differences in learning, according to the specific situation. Students work on the different tasks focusing on the identification of cultural and disciplinary contexts, within the lectio magistralis framework:

Distinctio – presentation of the context;
Divisio textus – analysis of the text;
Collatio – discussion;
Quaestio – critical interpretation.

The same analytical method is used on a variety of texts, including Descartes and Rousseau, working online on a dedicated platform. The same technique has then been applied to studying other disciplinary subjects and concepts accessing MOOCs, as described in the present contribution. Students are asked to evaluate the effectiveness of a massive open online course (MOOC) through their experience of learning online as outlined above. Massive Open Online Courses (MOOCs) are open access online courses, designed for distance learning involving large numbers of users. The term was employed for the first time in the “Connectivism and Connective Knowledge” module by George Siemens and Stephen Downes (2008), from the University of Manitoba (USA) and involved about 2200 online students, who did not pay any fees for their registration and attendance. As time went by, the number of open courses increased significantly, raising issues like the reliability of sources, correctness and quality of contents (Daniel, 2012; Stracke, 2014).

The main aim of the study has been to provide students with the opportunity to approach online learning in a structured way, which can be applied in a variety of contexts. The goal is to overcome short term, instrumental learning which fails to exploit the educational potential of MOOCs. This paper is part of a wider research project and focuses on the model adopted for evaluating the impact and effectiveness of online teaching and learning, enabling students to adopt a critical approach which could be extended to any online resource which they may use for their lifelong learning. Otten and Ohana, in their The Eight Key Skills Competences for Lifelong Learning (2009), a document issued under the support of the EC DG Education and Culture, focus on the identification of a set of skills needed to overcome present youth unemployment and social exclusion in developed countries. The central concepts referred to are: “critical thinking, creativity, initiative taking, problem solving, risk management, decision taking and managing feelings in a constructive manner” (p.10). There should be a closer connection between the above skills, education and digital education in particular. Technology plays a fundamental role in everyone’s life and must be approached critically, especially by young people entering the labour market for the first time. In the information society, the amount of online content is constantly increasing, and more content is becoming readily available online. Open Educational Resources (OER) are assuming an ever increasing importance in national educational policies. Between 2005 and 2007 UNESCO identified priorities for the spread of OER (OECD, 2007). As part of the aim to broaden the availability of a range of multimedia digital content, MOOCs arguably represent the most interesting digital products. The number of MOOCs is expected to grow rapidly over the next few years.

This paper describes how giving students the tools to carry out an evaluative analysis of MOOCs can enable them develop their analytical and critical thinking skills. It can also help them to gain insight into the importance of ‘learning to learn’. These students also gain the ability to characterise the impact of OERs on Higher education teaching and learning. The main scope of the present research project is that students could independently evaluate the quality of online digital resources both as learners and future educators. Doing so it is possible to overcome “brief term instrumental characteristics” of tools and promote long term evaluation processes (Vertecchi, 2012). This proposal concerns an area of research into distance learning which has not been explored in this way previously. The study does not explore the quality of learning in online environments. Rather, it investigates how students should approach the online resources at their disposal, facilitating their critical and reflective skills and adopting a model for analysis.

Hypothesis research questions and objectives

Taking into consideration national and international literature, the wider project, where this study is set, aims to design and test a new evaluation system of open access multimedia educational products such as MOOCs. The goal is to identify tools which enable the user critically to evaluate online resources and their impact on Higher Education teaching and learning.

The research tests the following hypothesis: Students who use a specific system to evaluate the quality of MOOCs are able to deepen their understanding of online teaching and learning in higher education and acquire sharper critical and analytical approaches to the evaluation of online learning.

Research objectives are the following:

to define an innovative system for the evaluation of MOOCs;
to define new quantitative and qualitative indicators to evaluate the impact of MOOCs on Higher Education teaching and learning;
to give students, as learners and future educators, quantitative and qualitative tools to freely assess a range of open access online resources;
to teach students contents about entrepreneurship education, as described, later on, in the specific example.
As mentioned above, here the focus is limited to the tool for the analysis of the effectiveness of specific examples of MOOCs.

MOOCs description

The MOOCs under investigation were created in the context of another research and training activity, carried out by LPS – Università Roma TRE, in cooperation with Salento University and DhiTECH (Apulia High Tech District). DhiTECH is a consortium established through the scientific research framework agreement signed in 2005 by the Italian Ministry of Education, The ministry to the Treasury, Region Apulia Local Authority, University of Salento, The National Centre for Research, and different private companies in the field of engineering and new technologies development. The aims of DhiTECH included training young professionals to develop their profiles as high tech innovators and entrepreneurs. The MOOCs, under investigation, here, were designed by graduate engineering students, under LPS researchers’ guidance. The aim of the MOOCs was to develop principles and generic competences, which are central to entrepreneurship education. Graduate engineers were asked to create MOOCs on a set of areas of entrepreneurship education as part of their research and training programme. The aim was to provide the engineers with the opportunity to develop their profile as high tech innovators and entrepreneurs, specialised in specific fields of knowledge. They developed and demonstrated the skills and approaches, which enabled them to transform technology or generate research results based on a new business model. The ultimate goal was to create economic and business value through technological entrepreneurship. The student engineers were divided into groups and they produced six different MOOC prototypes, but just three of them were selected for the further evaluation process to be carried out by the Education students, based in Roma Tre University. They were the following: “Social innovation and entrepreneurship”, “Business Model Canvas” and “From the business idea to the elevator pitch”. The first topic is bottom up social innovation practices in developed countries. The course is addressed to young learners interested in developing an innovative idea within cooperative learning spaces, like Fablab or Coworking. The second enables learners to use a particularly helpful tool, the Business model, to define successful business strategies. It focuses on how to draft an effective business model, which is meant as a starting point to develop new entrepreneurial ideas. The third refers to successful techniques in fundraising and how to approach a possible funder: the focus is driven on the characteristics of the so called “elevator pitch”.

Methodology

30 students, attending the first level university degree in Education – Roma Tre University, were involved in the research, on a voluntary basis and all of them participated actively in the task and filled in the evaluation questionnaire being piloted. Students were asked to participate in the three selected MOOC prototypes, as part of a compulsory internal training module (30 hours’ work), after approval of the Education degree course governing body. The task formed a compulsory curriculum module, which contributed to final certification. After taking part in the course, they had to carry out an evaluation, according to the ad hoc model devised by LPS researchers and described below.

The complete methodological plan was the following:

Students’ background variables questionnaire administration;
MOOCs completion;
Evaluation form, defined according to specific indicators created to assess the impact of MOOCs on Higher Education teaching and learning;
Final focus group with participants in order to record students’ opinions, attitudes and evaluations on the realisation of the project, and to obtain possible suggestions for improvements for a further iteration of the training module;
Final questionnaire on the experience.

The evaluation form can be considered an innovative qualitative research tool, based on specific indicators to assess open access multimedia products, like MOOCs. It was realised by adapting categories taken from the model for the assessment of critical thinking skills by Newman, Webb, Cochrane (1997). It, therefore, represents a tool which analyses the characteristics of MOOCs with reference to those generic competences which are increasingly demanded by the labour market.

The questionnaire is divided into four sections.

The first is devoted to Newman et al. (1997) categories of relevance and importance and contains indicators linked to formal characteristics of course content (e.g. comprehension, feedback effectiveness, video lecturing length and so on).
The second section is related to the categories of breadth of understanding, argumentation and justification and consists of a series of statements. Students indicate how far they agree with the statements using a five point Likert scale. The statements focus on issues such as level of understanding, step by step learning, growing difficulty, lack of information and explanation, etc.
The third section is devoted to critical evaluation, as students were asked to critically assess content, proposing issues for discussion and reflection, according to The Eight Key Competencies for Lifelong Learning (2009). Questions, here, were related to self-assessment of the competencies acquired.
The last section was designed to identify novelty. This is the only open ended section. The aim is to enable students to reflect and express their own opinion/evaluation, adding elements on already identified indicators (e.g. “briefly describe strengths of the course you participated in”; “briefly describe weaknesses of the course you participated in” “how could the course be improved” and so on).

Some Results

Some results regarding one of the MOOCs attended by Roma Tre Education students are presented and commented on below. Figures reproduced in this section refer to the evaluation of one of the MOOC prototypes under investigation: the “Business Model Canvas”. The other two MOOC prototypes under investigation were evaluated by the Education students using the same tool and received different evaluations, which are not inserted here for space reasons. These will be included in a further publication where comparisons and differences will be highlighted.

Findings presented here are mainly related to the employment of the evaluation form (questionnaire), because, according to the research group that carried out the work, it represents an innovative tool for investigation and it is a result in itself, because it could be employed to evaluate the impact of various technological instruments used in education and training.

Data from the first section of the questionnaire in this case highlighted that content was easy to understand, correct, effective, complete and quality of design and presentation were judged very positively.

Figure 1. Quality Evaluation – Relevance and importance (indicators relate to characteristics of course content)

As regards breadth of understanding, argumentation and justification general evaluation was positive: almost 90% of the students strongly agree or agree with the statements “I learnt what I expected”, “I learnt step by step (growing difficulty)”, “The course raised my curiosity and I deepened some topics”.

They do not agree, with negative statements, highlighting how the quantity of information was adequate and analysed correctly.

Figure 2. Content Analysis – Width of comprehension, argumentation and justification of solutions content

As far as critical evaluation is concerned, it is entrepreneurial “frame of mind” the transferable competence (see Otten & Ohana, 2009) which was facilitated by the course participation, followed by creativity, innovation and problem solving. Memorizing is considered less involved in these sort of learning processes.

Figure 3. Critical evaluation (skills acquired)

As mentioned above, the analysis carried out through the evaluation form was enriched by a focus group activity organised by the research group at the end of the whole experience. All the students participated and three sessions were set to discuss their experience. They showed appreciation for the initiative and gave suggestions for the improvement of the development of the activities, taking into consideration a further release of this sort of teaching and learning activity. Though in Italian and taking into consideration that the meaning of words in isolation can be confusing if out of context, it is interesting to notice the results from the word tag cloud obtained, analysing the focus group’s contributions, which are still under detailed processing activity. Most frequent words are “corso” (course), “molto” (very much, a lot), “interessante” (interesting) “gruppo” (group), “esperienza” (experience), but also “difficoltà” (difficulty) which is often linked to the concept of solution identification, thanks to the cooperative work (“gruppo”) foreseen in the various courses provided. This last piece of evidence was considered very positively by the research group, because it means that every obstacle encountered was overcome thanks to the support of the group, interacting with other subjects engaged in the same task.

Figure 4. Tag cloud – focus groups’ contributions

Conclusive remarks and further research

Data on assessment of critical thinking skills carried out through Newman et al. (1997) adapted model, already tested by the research group in other analyses (Poce et al., 2011; Poce, 2012) are being processed and will be communicated in further publications. The aim of the present contribution is to report on the evaluation system, as well as on the evaluation products devised and obtained through this research, which represents, as a whole, an innovation in the field of distance education, in general and in Higher Education, in particular.

Promotion of MOOCs is growing in different forms and settings, showing their wide educational potentialities. Actually, the same wide spread and openness, which characterize them, raised high standard methodological, evaluation and qualitative issues to be faced.

The project described here helped to define a system, which tends to match the intrinsic characteristics of MOOCS (widespread and openness) with those generic competencies, increasingly demanded by the labour market (critical thinking skills, in particular). ‘Focus group results indicate that the experience of participating in this project has made a significant contribution to the personal and professional development of students. Their ability to adopt a critical approach to technology for teaching and learning has grown.

More than half a century past by since the debate on the use of technological devices in education started and LPS – Università Roma Tre has been engaged in a deep reflection on the use of technology in Higher Education, since the very beginning. The development and the diffusion of these instruments are centred especially on the physical and affective characteristics of the technological tools, often neglecting a true scientific analysis and the application of pedagogical models and theories. This is also due to an “emphasis put on the educational potentialities of the instrument, which is too often based on light analogies and suggestions” (Vertecchi, 2012).

The educational reflection, instead, needs to overcome the instrumental short term level, in order to examine developments and implications of the long term, drafting a theoretical description, which could cover the various aspects engaged.

The insertion of technological devices in educational environments should, especially in time of crisis, aim at investments, which should be included and organised in detailed research programs. These programmes should define clear and consistent realization deadlines and evaluation methods for the sake of the costs-benefits-relation, if we want to keep in mind Comenius’ principle: omnia, omnibus, omnino (teaching everyone everything fully).

On the contrary, depending on affective aspects and not developing any didactical project, technology introduction in teaching and learning prompts a sort of “forced modernisation” (Vertecchi, 2013a) in educational practice.

Therefore, the risk of failure tied to the insertion of digital instruments in teaching and learning is very high, like many field studies demonstrate (Philip & Garcia, 2013) and investigation should concern various educational environments (sociology, psychology, linguistics, pedagogy, etc.).

That is why our research group has been working on a model for evaluation of MOOCs that could be helpful also for the development of critical thinking skills in university students, who are about to enter the hopefully productive labour world.

The LPS – Università Roma Tre group will carry on with the development of the evaluation tool and will extend the application of the system to other contexts and environments. The Department of Education – where LPS is based – has funded another term of experimentation, which will take place in the year 2014/2015.

References

Daniel, J. (2012). Making sense of MOOCs: Musings in a Maze of Myth, Paradox and Possibility. In Journal of interactive Media in education.http://www-jime.open.ac.uk/article/2012-18/pdf
Downes, S. and Siemens, G. (2008). The CCK08 MOOC – Connectivism Course, 1/4 way. In Dave’s Educational Blog.http://davecormier.com/edblog/2008/10/02/the-cck08-mooc-connectivism-course-14-way/
Hopkins, R. (2006). Critical reasoning and critical perception. In M. Kieran & D. McIver Lopes (eds.), Knowing Art, Philosophical Studies series, volume 107, (pp.137-153).
Newman, D.R.; Webb, B.; Cochrane, C. (1997). Evaluating the quality of learning in computer supported co- operative learning. In Journal of the American Society for Information science, 48(6), (pp. 484-495).
OECD (2007). Giving Knowledge for Free. The emergence of Open Educational Resources. Paris: OECD Publications. http://www.oecd.org/edu/ceri/38654317.pdf
Otten, H. and Ohana, Y. (2009).The Eight Key Competences For Lifelong Learning: An Appropriate Framework within which to Develop the Competence of Trainers in the Field of European Youth Work.https://www.salto-youth.net/downloads/4-17-1881/Trainer_%20Competence_study_final.pdf
Philip, T.M. and Garcia, A. (2013). iFiasco in LA’s Schools: Why Technology alone is never the Answer. Available at http://dmlcentral.net/blog/thomas-m-philip-and-antero-garcia/ifiasco-las-schools-why-technology-alone-never-answer
Poce, A.; Corcione, L.; Iovine, A.; Agrusti, F. (2011). Podcasting as a Teaching and Learning Tool. Experimental Evaluation of New Opportunities. ISBN 978-88-568-4136-7 Franco Angeli: Milano, (pp.77).
Poce, A. (2012). (ed.). Contributions to the definition of a critical technology. An assessment study, (pp. 81). ISBN: 978-88-204-1006-3, Milano: Franco Angeli.
Poce, A. (2014). (ed.). Promoting Science. Studies for the Definition of a Canon, (pp. 80). ISBN 978-88-204-5872-0, Milano: Franco Angeli.
Stracke, C.M. (2014). The concept of open learning for opening up education. In C.M. Stracke et al. (eds.), Changing the trajectory. Quality for opening up Education. Berlin: Logos Verlag Berlin. http://mediendidaktik.uni-due.de/sites/default/files/Heinen_Blees_Kerres_Rittberger_Connecting_Users_and_Editors_2014_0.pdf
Vertecchi, B. (2012). Parole per la scuola. Franco Angeli.
Vertecchi, B. (2013a). Modernizzazione non sempre fa rima con educazione L’analisi. Available at http://cerca.unita.it/ARCHIVE/xml/2565000/2562937.xml?key=Benedetto+Vertecchi*&first=1&orderby=1&f=fir
Vertecchi, B. (2013b). Alla scuola serve un progetto non solo qualche computer L’intervento. Available at http://cerca.unita.it/ARCHIVE/xml/2580000/2577322.xml?key=Benedetto+Vertecchi&first=1&orderby=1&f=fir

NEW

Archives

EURODL Mailinglist

EURODL Visitors

Developing Critical Perspectives on Technology in Education: A Tool for MOOC Evaluation

Abstract

The Context

Hypothesis research questions and objectives

MOOCs description

Methodology

Some Results

Conclusive remarks and further research

References

Tags

Current issue on Sciendo

EURODL is indexed by ERIC

EURODL is indexed by DOAJ

EURODL is indexed by Cabells

EURODL is indexed by EBSCO

For new referees