Flipped Classroom and Digital Learning in Action: A Case Study in Greek Primary Education
Ioannis Stefas [stefasioan@yahoo.gr], Adamantia Spanaka [madspa@otenet.gr], Logos2Learn, Hellenic Open University [www.logos2learn.edu.gr], Socratous, 152, 18648 Drapetsona, Greece
Abstract
Abstract in Greek
Η ανεστραμμένη τάξη συνιστά ένα σχετικά νέο μοντέλο εκπαίδευσης, το οποίο συνδυάζει την εξ αποστάσεως διδασκαλία με την εμπειρική μάθηση και την εφαρμογή νέων τεχνολογιών μέσα και έξω από την σχολική τάξη. Η δεδομένη έρευνα μελετά την αποτελεσματικότητα της εφαρμογής της ανεσταμμένης τάξης στην πρωτοβάθμια εκπαίδευση στην Ελλάδα με την αξιοποίηση της πλατφόρμας Edmodo. Παράλληλα, διερευνά το πώς μπορεί η ανεστραμμένη τάξη να ενισχύσει τη συμμετοχή των μαθητών στη μαθησιακή διεργασία και κατά πόσο οι μαθητές αντλούν ικανοποίηση με τη συμμετοχή τους σε αυτή τη δράση. Μεθοδολογικά επιλέχθηκε η έρευνα δράσης, χρησιμοποιώντας ποικίλα εργαλεία: ημερολόγιο καταγραφής ροής δραστηριοτήτων των μαθητών ανά διδακτική ώρα, ερωτηματολόγιο, ημιδομημένες συνεντεύξεις. Σημαντική ήταν και η συμβολή ενός κριτικού φίλου. Τα ευρήματα έδειξαν ότι η εφαρμογή της μεθόδου της ανεστραμμένης τάξης στην Στ΄ Δημοτικού έχει μόνο θετικά αποτελέσματα στους μαθητές, επιβεβαιώνοντας τα σημαντικά οφέλη που καταγράφονται βιβλιογραφικά διεθνώς. Συγκεκριμένα, ενεργοποίησε όλους τους μαθητές, ακόμη και τους αδύναμους, πέτυχε πλούσια μαθησιακά αποτελέσματα, συνέβαλε στην καλύτερη διαχείριση του διδακτικού χρόνου και στην ανάπτυξη της δημιουργικότητας των μαθητών, προσφέροντάς τους ικανοποίηση. Προτείνεται να δοκιμαστεί ως μέθοδος σε περισσότερα μαθήματα και σε περισσότερες τάξεις, ώστε να εξαχθούν ευρύτερα αποτελέσματα ακόμη και με ποσοτικές μεθόδους που θα πετύχουν γενικεύσιμα συμπεράσματα.
Key words: Flipped classroom, action research, digital learning.
Introduction
Multimedia constitutes one of the main features of Information and Communication Technologies as it enables integration of many different ways of presenting information through the exploitation of static or animated images, sounds, videos, texts, graphics and other media. The multimedia presentation of the information offers different and individualized ways of their elaboration by the user, helping in this way to preserve and consolidate knowledge in memory. The present research paper is based on this dimension of multimedia usage, which implements the innovative model of flipped classroom. According to this model the teacher provides the students with the appropriate digital educational material, which he or she has designed by enriching it with multimedia applications and clearly defining the objectives of its integration into each subject.
This material is afterwards used by the students either on their own or collaborating with other students at home (or any place convenient to them) using a computer, tablet or mobile phone, and return to the classroom, where both the teacher and the students discuss face to face and learn through the experiences the have gone through by studying the material at home.
Moreover, as Szparagowski (2014) mentions using Paulo Freire’s idea, education “shouldn’t include one person acting on another, but rather essentially combine people who work together’’. Therefore, this teaching model distance learning and technology with the empirical learning within the classroom.
The Aim and the Objectives of the Study
This study aims to identify the efficacy of flipped classroom implementation in Primary Education by exploiting the Edmodo educational platform, while enriching the existing bibliography. In particular, the results of the application of flipped classroom to the school performance of Primary School pupils in Greece were investigated by examining the following objectives:
- To what extend can flipped classroom implementation bring about rich learning outcomes?
- How can the flipped classroom alternative model enhance pupil participation in the learning process?
- How do the pupils themselves experience at the end of the study the flipped classroom as a whole?
Flipped Classroom and Blended Learning: The “Key” Combination
Blended learning is a way of teaching which combines the opportunities of student’s socialization in classroom with the activation of children in an on-line educational environment. This way of learning is not only a simple combination of various methods of teaching but it also provides opportunities for active learning and interaction among the students. Definitions of blended learning tend to vary that's why two experts on the field, Staker and Horn (2012), have classified it in four model.
The first one is the Rotation Model which “takes its name from the rotation of different learning methods for students” (Holzinger, 2016; p.5).
The second model of blended learning according to Staker and Horn (2012) is Lab Rotation: This is a model of learning during which in a particular school subject, the students according to a definite program or according to their teacher’s instructions move around different rooms in the school building. One of these rooms is the computer lab while the other rooms may be used for various ways of learning. The Lab-Rotation model is different from the Station-Rotation because the students move around and change rooms in the school building instead of staying in the same classroom for blended learning or the traditional class.
The third approach identified by Staker and Horn (op. cit.) is Individual Rotation: This is a model of rotation during which in a particular school subject, the students alternate according to an individually made program by methods of learning. One of these methods is definitely the on-line learning. The programs are made by the teacher or an algorithm. This model is different from the others because the students do not rotate either in places or in ways of learning.
The final model identified by Staker and Horn (op. cit.) is Flipped Classroom: According to this model, the students alternate (reverse) on a definite school program of teaching or project designed by the teacher in the school during the school day and there is initially an on-line presentation of the content of teaching from a remote place (for example from their home) and then practice of the content of teaching at school.
In the context of Flipped Classroom, we focus on the dissimilarity and the autonomy of the student, the cooperative learning, the acquisition of knowledge through the resolution of problems and the interaction among the students and between the students and the teacher. All these elements are parts of the methodology of Flipped Classroom which is based on the following four meanings of the word FLIP according to Flipped Learning Network (FLN, 2014).
- Flexible Environment: The methodology of Flipped Classroom is firstly based on the flexible environment. The teachers create flexible places of education where the students can choose where and when they will learn. The teachers are also flexible as far as their expectations and the content of their teaching is concerned, according to the needs and the learning profile of their students.
- Learning culture: The second important factor is the learning culture. The teachers spend a lot of time in class on the detailed investigation of issues and on the creation of opportunities of learning for their students. As a result of this, the students are able to take part in the construction of knowledge as they participate and they also evaluate their learning by personal ways of evaluation.
- Intentional content: The third factor is the educational content. The teachers decide upon what they have to teach and what kind of material the students can use. The teachers use the appropriate kind of material in order to maximize the time of teaching and they also adopt methods of active learning according to the level and the subject of learning. When we say content, we mean the kind of homework that the teacher gives to the students before the class but also the activities that the students do during the class.
- Professional educator: Finally, the fourth factor is the Professional Educator. His role is much more important and much more demanding in a Flipped class than in the traditional class. He has to watch his students in order to provide them with immediate feedback and at the same time to evaluate their work. The professional educator has a less obvious role in the flipped class but he remains the most important part of a successful way of learning.
According to Traxler (2018), the main principles of the Flipped Class can be expressed by the following questions:
- How can the teachers improve the personalization of learning? What can the students derive from that?
- How can the teachers improve the method of teaching from which the students are taught “face to face”? Can we assess the benefits that the students derive from that?
- How can the teachers improve their students’ experience in the school environment? What can the students bring to that?
- How can we deal with all the previous issues by using means of technology?
On the other hand, Herreid and Schiller (2013) have already presented some of the disadvantages of Flipped Class, having noticed for example that although the students in the beginning started working with enthusiasm at home, then they didn’t use to come appropriately prepared for the class. But all these disadvantages can be eliminated if there is a class of communication and concern which supports the cooperation among the students and urges them to share information and ideas, to exchange arguments, to discuss about their homework, to carry out projects by cooperating with their classmates, to help and respect each other. Also the teachers and all the institutions of the educational system involved, should support this model of learning. Especially as far as Greece is concerned, a survey conducted by Miminos and Spanakas (2013; pp.80-81) showed that “in the Greek educational system the opportunities for distant learning or at least for a mixed model of learning is insignificant which is perhaps due to the economical problems of the country or to the fact that these new methods are in dispute among the teachers because they have some more old-fashioned perceptions about teaching”.
Research Methodology
The research design chosen for this study was action research which focuses mainly on the practical application and more specifically on investigating “a specific school situation with a view to improving a practice” (Creswell & Plano, 2011, pp.639-640). More analytically, Mills (2000, referring to Creswell & Plano, 2011, pp.639-640) explains that the designs of action research involve systematic procedures which are carried out by teachers in order to study specific educational frameworks, to improve their teaching and the learning process of their students and finally to come closer to materializing their vision on education. Therefore, the reason why action research was chosen as research design in this study was that the teacher could participate as a researcher in the research process thus ensuring the cooperation between participants, that is the students, the teacher as well as the critical reader (a role taken over in this case by the school headmistress), as will be analysed later on.
In this research various tools of collecting data were used aiming at ensuring higher level of validity of the results. More specifically a diary of teacher-researcher was kept which provides a written account of facts during a time period, from the writer’s viewpoint (Ministry of Education/ Institute of Education Policy, 2012; p.154), which was actually used as one of the main sources of obtaining data for the investigation of aspects of this educational research in the framework of data triangulation. More analytically a diary of registering the flow of students’ activities per lesson was kept, in which an account of observations was kept regarding the development of the research, the problems faced throughout the research, the students’ response, their oral performance as well as their behaviour during the application of flipped classroom in the subject of Geography in the 6th grade of elementary school in Greece, in a classroom of 14 students.
Moreover, semi-structured interviews with the 14 students who participated in flipped classroom were taken. Each interview had an average duration of 15 minutes. The questions of the interviews regarded the students’ expectations and their whole experience of this model of teaching and learning, focusing mainly on how and why flipped classroom helped the students themselves. This tool therefore made use of predesigned questions, allowing at the same time the freedom and flexibility of addressing other issues that emerged during the conversation.
In addition, in this final stage of the research questionnaires were distributed to the students of flipped classroom with questions involving the way the lessons taught were carried out, while registering their views, impressions and suggestions on the model of flipped classroom.
Finally, observation of the teacher-researcher was made by a colleague, more specifically the school headmistress, acting as an assessor, having mutual trust, a process that enhanced the objectivity of the research.
Digital Educational Material
The educational platform that was chosen as an instrument for the application of the model of the flipped classroom in this research, was Edmodo. Edmodo, as a tool of Web 2.0 was chosen by the teacher-researcher because it is, among others, also a social media platform which is designed specifically for teachers and students. Through this platform, teachers can create a safe network for their class, while encouraging the direct exchange of opinions and messages, the participation and the teamwork, the feedback as well as the ability to incorporate a variety of digital means, quiet easily and without the need of specialized knowledge by the students. Also, Edmodo is created based on the security of the student’s personal data, which is achieved by the use of codes, who are only known by the students, ensuring their privacy and eliminating at the same time the main problem of the tools of Web 2.0, which refers to security issues. In this way, the teachers but also the students can share notes, links and files, post announcements, assign projects through it, even create a digital calendar. Furthermore, they can make digital libraries or small distance study groups and especially in digital form. All of these were thought as positive signs and they advocated for the choice of Edmodo by the teacher-researcher, as the tool that would help more in the application flipped classroom.
Consequently, a set of short videos (maximum duration of 5 minutes) were selected from YouTube and SlideBoom. The content of these videos included the content of the row of the lessons for the well-known subject of Geography that was chosen for elaboration by the teacher-researcher in 6th grade of elementary school. Regarding each video, appropriate written instructions were given on the platform of Edmodo. Additionally, the teacher-researcher has created a variety of evaluation exercises for each lesson, like quizzes with wrong or right exercises, multiple choice exercises and fill-in-the gaps exercises.
Also, the website https://online.seterra.com was used. This website contains a Greek menu and is about the subject of Geography, referring to countries, capitals, cities, lakes, rivers and many other things worldwide. For this website, the teacher-researcher created a quiz as well, in which the students had to find specific capitals of countries of Asia-Map Quiz Games (https://online.seterra.com/el/vgp/3067?c=5WGVH).
The Application of the Flipped Classroom
The application of the model of the flipped classroom took place in the classroom which is equipped with a computer, a projector and an interactive board, but also in the computer laboratory, which is equipped as well, but additionally in that laboratory each student has his own digital computer. The following educational techniques were used during the lesson in the classroom where with the physical presence of the students and the teacher.
- Practical exercises: While teacher-researcher was supervising, in the laboratory the students were browsing in the continents of Asia and Africa, while finding countries, capitals and their greatest geomorphological elements, connecting the theory with action, while only using Google earth.
- Discussion: The students asked questions that they had made notes on, while they were watching the appropriate video, which they had previously watched at home, and they got the answers from their classmates or their teacher.
- Brainstorming: On the board of the classroom are written spontaneously meanings or sentences, which are then commented on, while revealing the different aspects of a matter and aiming for the enrichment of the students’ knowledge.
- Questions and Answers: The teacher-researcher was asking questions referring to the lesson and was encouraging the students to do the same thing, while improving the students’ agency and the development of their critical thinking.
- Working groups: Three working groups were formed, each one consisting by 4-5 students, so that the participants help each other and communicate regarding topics such as what the firsts of Asia (e.g. highest mountain, deepest lake, most crowded continent), which were presented through videos and press articles, PPT, in order to promote student’s creativity.
The Results
While expanding the point on which the application of the flipped classroom for students of Primary Education can lead to rich educational results, it was detected that all of the students who participated improved their performance in the subject of Geography. Also, the method of the flipped classroom instantly sparked the interest of the students, but without the disappearance of their disbelief. This disbelief disappeared shortly, because the handiness of the educational platform Edmodo was the strong point of the whole new teaching attempt. Furthermore, for the learning and the proper use of the platform by the students, very little time was needed. This handiness is achieved thanks to the fact that it looks like social media to which the students are very accustomed.
As to how the flipped classroom can reinforce the participation of the students in the learning process, a widespread entrenchment was detected by the students in the whole process, specifically in the classroom during the lesson. A special impression was caused when the weaker students tried and succeeded their goals during the lesson. Moreover, it was noticed that the students showed the demanded maturity, as they worked on their own but also in terms, showing willingness for teamwork and information exchange. In this way, the learning process was more pleasant for the students, who felt how to do something different –in comparison to whatever they were used to doing in the past years- as far as school projects go. Additionally, the high level of interaction that the students had while they were working in teams was confirmed, as also as the interaction that was achieved between them and their teacher, proving their ability of great teamwork and mutual funds. It is also important to say that the teacher-researcher was beside them from the very beginning as an assistant and a consultant, he guided them through the difficulties that they found and he encouraged them in their attempt. An interaction was also achieved among the students, the educational material and the educational tool that was used in this research, which was simple and understandable, appropriate for the specific student’s educational needs.
Finally, all the students felt satisfied from the advantages that the flipped classroom provided them with, and they were excited as well for using something that was out of their ordinary school routine. Also, there was detected an excellent use of the digital means by the students, which made them feel very confident. However, there were also students who were unable to watch the video due to the bad internet connection and the seniority of their computer. Nonetheless, the students wanted the application of the model of the flipped classroom to be extended in other subjects of their class as well.
Besides, one of the advantages that the students harvested was connected to the additional free time that they have in their hands in the classroom, thanks to the application of the flipped classroom, in which the learning of the new knowledge is done at home. Thus, they had the chance to do activities in regard to the lesson or to acquire more knowledge besides what the school textbook provided them with. An important element that made this research stand out is also the tendency and the desire that the students had to think of a mistake as a learning experience and the will to proceed to their self-correction, as long as specific and understandable instructions were given before every exercise.
Conclusion
The application of the model of the flipped classroom, which combines the flexibility of digital learning with the immediacy of the in-class teaching, was successful in the class of this specific research. Thus, our initial concern for the level on which the application of the flipped classroom can lead to rich educational results, showed us that the students, through a variety of educational procedures, as for example was the process of the watching a short video, which had been carefully selected by the teacher-researcher, through the educational platform of Edmodo and while participating in the different activities, which had also been prepared by the teacher-researcher, they got to be more productive and to achieve clearer, better learning results, while succeeding in the subject of Geography with better performances and understanding more the meanings of this subject.
There was an acceptance of the model by the students, which led to the watching of the digital material through the computer or the tablet or even a small phone at home putting them in a good mood, while anticipating the educational object, so as we said before, they asked for the model of the flipped classroom to be extended in other educational topics too, a conclusion to which the research of Chouli (2015) had also been led, even if it was referring to the application of the flipped classroom in Higher Education.
Regarding our second question as to how the alternative model of the flipped classroom can reinforce the student’s participation in the learning process, it was found out that the flipped classroom gave the students, not only the motivation, but also the boost to participate actively and throughout, even the weaker and more “quiet” students, in the activities of the educational platform, but also in the in-class activities during the lesson and at the end of every unit, when they had to produce their own videos while working in teams and making creatively use of the learning time that they had in their hands. This is actually a conclusion similar to the one of Basal (2015), but also the literature review of Zainuddin and Halili (2016) and to the one of Aydin and Demirer (2016), which referred to tools that can be used specifically for the flipped classroom.
Concerning how the students finally experienced the flipped classroom, we get that they were extremely satisfied with the new tools and the new learning method that they were given, despite their initial disbelief – a predictable element- as it is registered by Gaughan (2014) and also by D’addato and Miller (2016) in their researches – of something initially unknown to them, which finally got them out of their everyday life and their school routine of many years. This can be concluded also by their positive comments and the posts they made on the educational platform, by their high participation in all of the duration of the lesson and by the high level of interaction that they achieved, not only with each other but also with their teacher and the digital educational material, an element which is also referred in the research of Hwang and Lai (2017).
At this point we have to point out that this specific classroom of 14 students had been a “troubled” and “difficult” one, with a frequent exchange of teachers even in the middle of the school year and students with personal differences. There was a lack of participation in the educational process, the classroom had also foreign students and the class in general was uneven. From the very start, when the teacher-researcher undertook this class from the fifth grade, there was noticed a lack of interest, nuisance, negligence, inattention, concentration and team spirit. The level of knowledge that the students had was also mediocre, as there was a profound shortage of help by their parents at home, while many of them were facing family issues. A lot of time and empathy by the teacher was needed, and also effort, persistence and patience in order to build a healthy relationship and to earn trust and communication with his students, with the purpose of changing the way that they were seeing the learning objects and the relationship between them, so that they could feel better thanks to the application of the flipped classroom in the learning process, as it is specifically recorded by Cheng and Weng (2017).
Therefore, during the application of the new educational method, the students’ active occupation with the educational subject, inside and outside of the classroom, the participation, the maturity that was shown throughout the process, the rich educational results – all of the students increased their grade on the subject, orally and writing, the satisfaction, the excitement, the willingness of the students and mainly the consideration of the mistake, which some of them did as a learning experience with the simultaneous tendency of auto correction, they emphasized the effectiveness of the model of the flipped classroom while utilizing the educational platform of Edmodo and the digital means that were included.
In the end, after the application of the flipped classroom it is concluded that the teacher has to devote a lot of personal time for the preparation of the videos and the exercises, and also for the checking of the activities, as to requirements for the model of flipped classroom are high in comparison to the conventional teaching way. Moreover, the teacher ought to keep in mind the students’ family and economical background and their environment, and also their personal issues. Although, in the restrictions of this specific research we find the small size of the sample, its restriction in just one school and only one subject, the one of Geography. Consequently, it would be interesting to further investigate the application of the method of the flipped classroom during the whole school year, as would also be the investigation of this model in smaller classrooms of the primary education as opposed to the conventional classroom, in order to track more accurately the students’ progress, their interest and participation, as well as the interactive relationships that form among them, among the students but also among the students and the teachers and among the students and the digital means which the model of flipped classroom utilizes.
References
- Aydin, B., & Demirer, V. (2016). Flipping the Drawbacks of Flipped Classroom: Effective Tools and Recommendations. Journal of Educational and Instructional Studies in the World, 6(1), 33-40. Retrieved on March 18, 2017 from http://www.wjeis.org/FileUpload/ds217232/File/04.betul_aydin.pdf
- Basal, A. (2015). The Implementation of a Flipped Classroom in Foreign Language Teaching. Turkish Online Journal of Distance Education, 16(4), 28-37. Retrieved from https://files.eric.ed.gov/fulltext/EJ1092800.pdf
- Cheng, Υ. Η., & Weng, C. W. (2017). Factors influence the digital media teaching of primary school teachers in a flipped class: A Taiwan case study. South African Journal of Education, 37(1). Retrieved from http://www.scielo.org.za/pdf/saje/v37n1/12.pdf
- Chouli, A. (2015). Flipping a Merchant Class Around (Master Thesis). Hellenic Open University, Patra. Retrieved on January 10, 2018 from https://apothesis.eap.gr/bitstream/repo/28849/1/Chouli%20Aikaterini-51614-dissertation-May%202015.pdf
- Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting Mixed Methods Research. Los Angeles: Sage Publications.
- FLN – Flipped Learning Network (2014). The four Pillars of F-L-I-PTM. Retrieved on March 16, 2019 from https://flippedlearning.org/wp-content/uploads/2016/07/FLIP_handout_FNL_Web.pdf
- D’addato, T., & Miller, L. R. (2016). An inquiry into flipped learning in fourth grade math instruction. The Canadian Journal of Action Research, 17(2). Retrieved on March 10, 2018 from http://journals.nipissingu.ca/index.php/cjar/article/view/261/128
- Gaughan, J. E. (2014). The flipped classroom in world history. The History Teacher, 47(2), 221-244. Retrieved on March 10, 2018 from http://www.societyforhistoryeducation.org/pdfs/F14_Gaughan.pdf
- Goodnough, K., & Murphy, E. (2017). The professional learning of grade six teachers of mathematics implementing the flipped classroom approach. Canadian Journal of Learning and Technology, 43(1), pp. 1-20. Retrieved from https://www.cjlt.ca/index.php/cjlt/article/view/27483
- Herreid, C. F., & Schiller, N. A. (2013). Case studies and the flipped classroom. Journal of College Science Teaching, 42(5). Retrieved on April 10, 2018 from https://www.aacu.org/sites/default/files/files/PKAL_regional/CRWG-SPEE-REF-01.pdf
- Holzinger, I. A.-Ch. (2016). The Flipped Classrooms Model for Teaching Vectors (Diploma Thesis). Linz, Austria: Johannes Kepler University Linz.
- Hwang, G. J., & Lai, C. L. (2017). Facilitating and Bridging Out-Of-Class and In-Class Learning: An Interactive E-Book-Based Flipped Learning Approach for Math Courses. Educational Technology & Society, 20(1), 184–197. Retrieved on March 10, 2018, from https://pdfs.semanticscholar.org/c9b3/d4cbaebf78c675da85922da0d0db68cb321e.pdf
- Knowles, M. S. (1975). Self-Directed Learning: a guide for learners and teachers. New York: Associated Press. Retrieved on March 14, 2018 from http://web.utk.edu/~start6/knowles/malcolm_knowles.html
- Lage, M. J., Platt, G. J.,
& Treglia, M. (2000). Inverting the classroom: A gateway to creating an
inclusive learning environment. The Journal of Economic Education, 31(1),
30-33. Retrieved on March 17, 2018 from https://www.academia.edu/340051/Inverting_the_Classroom_A_Gateway_to_
Creating_An_Inclusive_Learning_Environment - Miminos, Α., & Spanakas, Α. (2013). Distance Education Primary and Secondary Schools and Programs: A Literature Review. In A. Lionarakis (Ed.), 7th International Conference in Open & Distance Learning, November 2013 (pp. 78-90). Athens, Greece.
- Staker, H., & Horn, M. B. (2012). Classifying K–12 Blended Learning. Retrieved on March 14, 2018 from https://www.christenseninstitute.org/wp-content/uploads/2013/04/Classifying-K-12-blended-learning.pdf
- Szparagowski, R. (2014). The effectiveness of the flipped
classroom (Honors Projects). Bowling Greet State University. Retrieved on
March 17, 2018 from
https://scholarworks.bgsu.edu/cgi/viewcontent.cgi?referer=
https://www.google.gr/&httpsredir=1&article=1118&context=honorsprojects - Traxler, J. (2018). Distance Learning—Predictions and Possibilities. Education Sciences, 8(1). Retrieved on June 17, 2018 from http://www.mdpi.com/2227-7102/8/1/35/pdf
- Zainuddin, Z., & Halili, S. H. (2016). Flipped Classroom Research and Trends from Different Fields of Study. The International Review of Research in Open and Distributed Learning, 17(3). Retrieved on March 14, 2018 from http://www.irrodl.org/index.php/irrodl/article/view/2274/3766