Best of EDEN 2018 Special issue

There are not any recent contributions.

There are not any recent contributions.


EURODL Mailinglist

2851 subcribers

EURODL Visitors


E-tests -
What Can be Assessed in E-learning Applications

Thomas Risse
Institute for Informatics & Automation, Hochschule Bremen, Germany

Key words: e-learning, test, assessment, mathematics, IMS standards


In the light of different existing standards it is investigated what can be assessed by e-tests, i.e. by tests in e-learning. In this way, potential and limits of such tests are outlined and illustrated by examples most of all from mathematics and computer architecture. The results of a little field study conducted in the authors lectures are given: analysis of teacher-learner-interaction not only reveals how to map question-answer to tests of the learning outcome in e-learning applications but also in which directions these standardized tests ought to be extended. Additionally, implications for the design of intelligent tutoring systems are discussed.

1  Introduction

Integrated tests play an important role in e-learning applications both for self assessment as well as for directions given to the learner to follow on individual learning paths. By nature, these tests easily assess the correctness of some solution. But it is inherently difficult to rate approach, modeling, methodic proceedings etc. by these tests.

The objective of this paper is to analyse traditional, i.e. class lectures in order

  • to classify the interaction between lecturer and student in the light of furthering the learning process and of testing the learning outcome
  • to try to map the identified tests resp. test types onto the tests resp. test types (e.g. Basic Item Types in the IMS slang) used in e-learning applications.
In this way, benefits and shortcomings of this type of computer support should become apparent. In addition, the analysis can provide hints for the design of intelligent tutoring systems [3].

To this end, two worlds have to be bridged: on one hand, in the field study - a set of two lectures in mathematics given by the author - teacher-learner interaction in order to evaluate and test the learning outcome are analysed, and question/answer pairs are classified. On the other hand the existing standards for tests in e-learning applications are presented, their inherent limitations are identified, and in spite of these limitations it is endeavoured to match question/answer pairs to these tests.

1.1  Standardization

Since quite a few years different national and international bodies have invested considerable efforts to establish standards for e-learning applications1. Industry, e.g. the Aviation Industry CBT Committee, AICC, and special interest groups, e.g. the IEEE publish proposals for descriptions by metadata, architecture of learning technology systems, quality standards etc. in order, for example, to make possible to use learning material in different contexts. Some initiatives may serve as examples:
  • Advanced Distributed Learning, ADL is an initiative initiated by the DoD standardizing e-learning e.g. by proposing the Sharable Content Object Reference Model, SCORM, s. using XML,
  • ARIADNE, i.e. Alliance of Remote Instructional Authoring and Distribution Networks for Europe is a project of the European Union. A consortium provides educational metadata, tools, a user group etc.
  • The Dublin Core Metadata Initiative develops interoperable online metadata standards that support a broad range of purposes and business models,
  • The IMS Global Learning Consortium, Inc. was founded as a project within the National Learning Infrastructure Initiative of EDUCAUSE. Users, software producers and activists provide XML specifications of metadata, question & test, learning resources etc.
  • The Learning Technology Standards Committee, LTSC of the IEEE specifies a Learning Technology Systems Architecture, LTSA, proposes a standard for Learning Object Metadata, LOM, etc.
  • The Open Universiteit Nederland proposes the standard Educational Modelling Language, EML focusing on didactic aspects and using XML,
Standardization efforts are by no means compatible, but compliance and interoperability become more and more important issues. It is attempted either to map one standard to another or to set up comprising super standards, e.g. the Universal Learning Format, ULF

Here, we refer to the IMS standard proposals because they take other drafts like the IEEE LTSCs into consideration and offer elaborated and well documented XML implementations.

1.2  Standardized Tests in e-Learning Applications (IMS)

The common means to test the learning outcome in e-learning applications have not changed to much since IBM or Lufthansa marketed their first products of this type in the sixties [9].

Here, we list the IMS Basic Item Types [5] with the expected user action.

Logical identifier (the user is expected to click on the right radio buttons or check boxes);
X-Y co-ordinate (the user is expected to click on the right area on the screen/in an image);
String (the user is expected to type some word or text);
Numeric (the user is expected to type some number or move a slider);
Logical groups (the user is expected to drag and drop objects into appropriate positions or response holders).
Combinations of these types are of course allowed. For further reference, we list the IMS Basic Item Types in more detail.
The LID examples are:
  1. Standard True/False (text-based options) – choice-based rendering;
  2. Standard Multiple Choice (text-based options) – choice-based rendering;
  3. Standard Multiple Choice (image-based options) – choice-based rendering;
  4. Standard Multiple Choice (audio-based options) – choice-based rendering;
  5. Standard Multiple Response (text-based options) – choice-based rendering;
  6. Multiple Choice with Single Image (image-based options) – IHS2-based rendering;
  7. Multiple Response with Multiple Images (image-based options) – IHS-based rendering;
  8. Multiple Choice (slider-based options) – slider-based rendering;
  9. Standard Order Objects (text-based objects) – object-based rendering;
  10. Standard Order Objects (image-based objects) – object-based rendering;
  11. Connect-the-points (image-based) – IHS-based rendering.
The XY co-ordinate examples are:
  1. Standard Image Hot Spot (single image) – IHS-based rendering;
  2. Connect-the-points (image-based) – IHS-based rendering.
The STR examples are:
  1. Standard Single Fill-in-Blank – FIB-1>AB-based rendering;
  2. Standard Multiple Fill-in-Blank – FIB-based rendering;
  3. Standard Short Answer (text required) – FIB-based rendering.
The NUM examples are:
  1. Standard Integer Fill-in-Blank – FIB-based rendering;
  2. Standard Real number Fill-in-Blank – FIB-based rendering;
  3. Numerical entry with Slider – slider-based rendering.
The GRP examples are:
  1. Standard Drag-and-drop (multiple images) – object-based rendering.
To give a more up-to-date example, in the CBT-applications of the Cisco Networking Academy Program only LID2 questions are used in quizzes, reviews, online and final exams. Right now, besides true/false the only personalized feedback is a list of chapters corresponding to incorrect answered questions.

2  A Field Study

The field study consisted in recording a set of two lectures in mathematics (selected by chance, namely the availability of recording equipment) and in categorizing question and answers of lecturer and students.

In section 2.2 it will become obvious that the lecturer tried to make the students acquire [4]

  • declarative knowledge, e.g. what is a vector field?
  • conceptual knowledge, e.g. concept to intersect planes and graphs (partial function graphs or level curves) to visualize functions
  • procedural knowledge, e.g. how to model a 3D object?
By questioning the lecturer tried to arouse the students interest, to guide their way of thinking and to test whether or not they had learnt a certain fact, had understood concepts, or were able to set up a model, to construct an algorithm or the like. Thus, the lecturer tried to inspire different types of learning [4], i.e.
  • to make students learn facts by repetition, etc
  • to make students recognize mathematical objects, to stimulate explorative learning by using a 3D modelling and rendering program,
  • to make (groups of) students to construct mental images of 3D objects and to model more and more complex 3D objects.

2.1  Background and Objectives of the Lectures

The lectures for students at the end of their second semester dealt with the introduction of functions of several variables. These students should know basic linear algebra, basic analytical geometry, analysis of functions of one variable (including power and Fourier series) and ordinary differential equations. The objective was to work out a set of simple examples of scalar and vector valued functions of one and several variables.

The objectives can be summarized in the following table:

scalar fields vector fields
n\m 1 2 3 ...
1 y=f(x) curves* in IR2 curves* in IR3
w=f(x,y), e.g. temperature distribution on surfaces

matrix transfor- mations of IR2
surfaces* in IR3
w=f(x,y,z) or w=f(x,y,t), e.g. temperature distribution in space or on a surface in time

matrix transfor- mations of IR3, of IR2 (homogeneous coordinates), fields
w=f(x,y,z,t), e.g. time dependent temperature distribution in space


e.g. time dependent fields
* parameter representation

The case n=1 had been dealt with in a lecture preceding the set of lectures evaluated here. The students already got along some examples of scalar and vector fields. The concepts of partial function and level curves had been introduced. Parameter representation of e.g. circles in the plane or of helixes in space had been dealt with.

2.2  Question and Answers in Lectures

We now present questions posed and answers given in the set of lectures together with a categorization. The following representation is used: LQ: lecturers questions  LA: lecturers answers LT: lecturers material/text 
TA: targeted students answers - intended/expected, but not given 
SQ: students questions  SA: students answers  ST: students remarks/text 

As it turns out, for our purposes it is sufficient here to describe only the first of the two recorded lectures focusing on the question/answer type of interaction. (MAI020628 and MAI020702 on contain the full audio documentation in german.)

  1. LQ: what were the results and examples dealing with functions f:IR É D® IRm dealt with in the last lecture?  LT: objectives ...  
  2. LQ: do you want examples to refresh results of the last lecture?  SA: yes  LT: let us write a function f:IR® IRm as usual as [p\vec]:IR® IRm, e.g. [p\vec](t)=[p\vec]o+t([p\vec]1-[p\vec]o). 
  3. LQ: what kind of object is {[p\vec](t): t Î IR} ?  SA: line through [p\vec]o and [p\vec]1 
  4. LQ: what is [p\vec]1-[p\vec]o ?  SA: direction vector  LT: same not only for lines in IR3 ...  
  5. LQ: more examples?  SA: no  LT: let us deal with n=2, m=1 
  6. LQ: how do you call z=f(x,y) ?  TA: scalar field  LT: recall scalar fields like f(x,y)=x+y or f(x,y)=x y. Let us consider another example, say z=f(x,y)=sinÖ{x2+y2
  7. LQ: how is the graph of some f:IR2 É D® IR defined? and what kind of object?  SA: graph(f)={(x,y,f(x,y)): (x,y) Î D} is a surface in IR3 
  8. LQ: how does one visualize the graph of z=f(x,y)=sinÖ{x2+y2} ?  TA: sections, perspective visualizations  LT: to get some idea, note that Ö{x2+y2} is inserted into the sine function. 
  9. LQ: how do the level curves to some level zo look like?  TA: circles 
  10. LQ: which (x,y) do solve zo=f(x,y)=sinÖ{x2+y2} ?  SA: arcsinzo=Ö{x2+y2}  LT: E.g., let zo=[ 1/2] 
  11. LQ: what is arcsin[ 1/2] ?  LT: think of memory hook sinji=[(Öi)/2] for ji Î {0,[(p)/6],[(p)/4],[(p)/3], [(p)/2]}   SA: arcsin[ 1/2] = [(p)/6] 
  12. LQ: which (x,y) do solve ([(p)/6])2=x2+y2 ?  LT: think of Pythagoras or rewrite |(x,y)|=|(x,y)-[0\vec]|=[(p)/6] 
  13. LQ: wierd question?  SA: yes  LT: Recall vector length and distance. Consider the intersections with coordinate planes. [(p)/6], do call it r ...   SA: circles 
  14. LQ: how do partial functions z=f(x,0) and z=f(0,y) look like?  SA: sine functions 
  15. LQ: what kind of intersections do graphs of partial functions represent?  LT: ...   SA: intersections with coordinate planes 
  16. LQ: how does z=f(x,x) look like? What kind of intersection?  TA: graph with plane, i.e. graph(f)Ç{x=y} 
  17. LQ: what function of one variable is z=f(x,x) ?  SA: again a sine function 
  18. LQ: what kind of intersection?  LT: ...  
  19. LQ: what is the equation of the intersecting plane?  SA: x=y 
  20. LQ: why is this a plane?  TA: x-y+0·z=0  LT: ...  
  21. LQ: where is this plane in space?  SA: The plane contains the z-axis  LT: ...  
  22. LQ: how does the intersection of x=y with the xy-plane look like?  SA: the diagonal 
  23. LQ: visualize graph(f)  LT: ... think of a snapshot of a concentric water wave ...  
  24. LQ: how can we model friction?  LT: ... the amplitude should be indirect proportional to the distance to the origin. Recall ordinary differential equations with characteristic polynomial with two pairwise conjugate complex zeroes ...  
  25. LQ: recall: how do damped oscillations look like?  LT: ...  
  26. LQ: in e-¼? ¼sin(wt) fill in the exponent  SA: e-t/tsin(wt) 
  27. LQ: how can we tranfer this to our situation where damping is proportional to the distance to the origin?  LT: ...   SA: e-Ö{x2+y2}sinÖ{x2+y2} 
  28. LQ: how can we introduce time into our function?  LT: ...   LT: consider the case n=3, m=1, for example z=f(x,y,t)=Aocos2p(nt-Ö{x2+y2}/l) ...  
  29. LQ: visualize f  LT: ...   LQ: Visualize f for a fixed t  LT: ...  
  30. LQ: what is the level curve to a level of [(Ao)/2] and fixed t ?  SA: {(x,y,to):Aocos2p(nt-Ö{x2+y2}/l)=[(Ao)/2]} 
  31. LQ: solve for x and y  SA: 2p(nt-Ö{x2+y2}/l)=[(p)/3]  LT: you can do better ...   SA: nt-Ö{x2+y2}/l = [ 1/6]  LT: you can do better ...   SA: nt-[ 1/6] = Ö{x2+y2}/l  LT: you can do better ...   SA: x2+y2=l2(nt-[ 1/6])2 
  32. LQ: what are these level curves geometrically?  SA: again circles  LT: ...   LT: consider the case n=2, m=2, i.e. a function [f\vec]:IR2 É D® W Ì IR2 
  33. LQ: how is [f\vec] called?  SA: vector field  LT: here [f\vec](x,y)=(u(x,y),v(x,y)) 
  34. LQ: what are u(x,y) and v(x,y) ?  SA: scalar fields  LT: vector fields are vectors of scalar fields, e.g. matrix transforms like
    ® æ
  35. LQ: evaluate the product  SA: (xcosd-ysind,xsind+ycosd) 
  36. LQ: what is the relation between the argument vector and the function value, the product?  LT: identify, remember C , recall Euler, remember addition theorems of sine and cosine ...  
  37. LQ: what is the polar representation of (x,y) ?  SA: r(cosj,sinj) 
  38. LQ: how are r and j determined?  SA: r=Ö{x2+y2} and j = arctan[ y/x] 
  39. LQ: what do you get inserting the polar representation of the argument into the product?  SA: r(cosjcosd-sinjsind, cosjsind+sinjcosd) 
  40. LQ: what do you get applying the addition theorems?  SA: r(cos(j+d),sin(j+d)) 
  41. LQ: what happened to the argument vector geometrically?  SA: rotated by d  LT: There are important matrix transformations of the plane, e.g. scaling, shearing and even - in homogeneous coordinates - translation ... Consider in the case n=2, m=3 the vector fields [f\vec](x,y)=(u(x,y),v(x,y),w(x,y)):IR2 É D® W Ì IR3 ...  
  42. LQ: what is the dimension of D ?  SA: two-dimensional 
  43. LQ: what is - in general - the dimension of W ?  TA: two-dimensional 
  44. LQ: what do we get in W for some fixed yo ? What is {[f\vec](x,yo): (x,yo) Î D} ?  TA: a curve in IR3, co-domain of some partial function  LT: ...  
  45. LQ: what is the dimension of {[f\vec](x,yo): (x,yo) Î D} ?  SA: one-dimensional 
  46. LQ: what is the dimension of W ?  SA: two-dimensional  LT: The co-domain of [f\vec](x,y):IR2 É D® W Ì IR3 are surfaces in space in parameter representation ... For example, consider three vectors [p\vec]o, [r\vec]1, [r\vec]2 corresponding to some plane in space 
  47. LQ: what role do these three vectors play?  SA: plane through [p\vec]o with direction vectors [r\vec]1 and [r\vec]2 
  48. LQ: how do we get each point of the plane?  TA: [p\vec]o plus a linear combination of [r\vec]1 and [r\vec]2  LT: recall that planes in space as well as lines in the plane are hyperplanes ...  
  49. LQ: what are the points of the line in IR2 or IR3 through [p\vec]o with direction vector [r\vec]1 ?  SA: [r\vec](t)=[p\vec]o+t[r\vec]1 
  50. LQ: what are the points of the plane in IR3 through [p\vec]o with direction vectors [r\vec]1 and [r\vec]2 ?  SA: [r\vec](s,t)=[p\vec]o+s[r\vec]1+t[r\vec]2 
  51. LQ: what are the points of the plane in IR3 through [p\vec]o, [p\vec]1 and [p\vec]2 ?  SA: [r\vec](s,t)=[p\vec]o+s([p\vec]1-[p\vec]o)+t([p\vec]2-[p\vec]o) 
  52. LQ: how to model the intersecting plane x=y from above?  ... SA: [r\vec](s,t)=[0\vec]+s([p\vec]1-[0\vec])+t([p\vec]2-[0\vec]) with [p\vec]1=(1,1,0) and [p\vec]2=(1,1,1)  LT: not unique, not collinear, ...  
  53. LQ: why do [r\vec](s,t)=[0\vec]+s[p\vec]1+t[p\vec]2 and e.g. [r\vec]¢(s¢,t¢)=[0\vec]+s¢[p\vec]1+t¢[e\vec]z represent the same plane?  LT: ...   LQ: how to verify my claim? 
  54. LQ: difficult?  SA: yes  LQ: at home or right now here  SA: here right now  LT: ...   LT: let us summarize what we have learnt today ... in the next lecture we will consider ...  
In the second lecture, matrix tranformations were repeated, scaling and translation presented. The case n=2, m=3 was illustrated by (more) parameter representations of surfaces in space stressing the analogy between curves in the plane and surfaces in space. As examples served planes and spheres (globe) for which parameter representations were derived. Further examples were Bézier- and B-Spline surfaces used to model free form objects in generative computer graphics. It was strongly recommended to use and extend the 3D modeling and rendering software in order to model, program and render new 3D objects on ones own.

2.3  Types of Questions

Browsing this material, certain types of questions are readily identified. All questions obviously aim at triggering some action of the students. Therefore, we classify questions by the type of action they are meant to provoke. At the same time, we indicate by examples that a certain type of question accurs in many more circumstances when teaching mathematics to make up for the fact that the topic of the two lectures inevitably stressed certain actions (e.g. visualization) and neglected others (e.g. modelling).
Model, Construct
here e.g. model friction/damping, model free form 3D objects
modelling of course is the essential part in solving any problem (of problem based learning) - especially when application oriented problems posed in the real life, non-mathematical manner are to be attacked. And, geometric models are constructed as well as algorithms, bases, estimators, measures, etc.
here e.g. level curves
but also e.g. classification of differential equations to determine the method by which to solve the differential equation
Evaluate, Solve, Apply
here e.g. arcsin[ 1/2], addition theorems, Pythagoras
but also e.g. solutions of systems of linear equations, minima/maxima, integrals, zeroes of characteristic polynomials, Laplace transforms etc. - in the end, any mathematical procedure or algorithm
Identify, Recognize
here e.g. plane or space curves in parameter representation, scaling transformation
but also e.g. the other matrix transformations, time and space requirements of algorithms etc.
Typify, Find examples
here both lessons were dedicated to illustrate scalar and vector functions of one or several variables by pertinent examples
but also e.g. to get a first idea of how an algorithm works, to illustrate theorems, to find counter-examples etc.
here e.g. visualisation of vector fields by visualisation of each of its scalar fields
but also e.g. criteria for continuity or the like hold for real valued as well as complex valued functions etc.
here e.g. polar representation,
but also e.g. recalling vector rotation in the plane is essential to understand and implement CORDIC algorithms - precognition of the basics is a prerequisite to learn any mathematical procedure or algorithm.
Verify, Falsify
here e.g. check whether or not the points of the parameter representation of the sphere [r\vec](j,y) = r(cosycosj, cosysinj, siny) lie on the sphere
but also e.g. verification whether solutions do solve the given problem or any consideration of plausibility
here e.g. any scalar field or a coordinate of any vector field, 3D model
but also e.g. visualization of any solution, algorithm etc.

3  Mapping Questions to Tests

Let us try to match the different types of questions to the IMS Basic Item Types.

3.1  Inherent Limitations

The IMS Basic Item Types can be partitioned into two subsets, Basic Item Types providing a list of possible answers or solutions 'with clues' or 'without clues'.

The set of Basic Item Types 'without any clues' consists of STR1, STR2, STR3, NUM1, and NUM2. All other Basic Item Types present possible solutions and hence clues. Here, the learner is able to identify the correct answer by elimination.

Other Basic Item Types may be extended to types 'without clues', e.g.

  • LID8, i.e. Multiple Choice (slider-based options) –- slider-based rendering
    may be extended to real valued sliders: for example, to approximately set Ö2 on a slider with ticks 0, 1, 2 and 3 or to set the most frequent color in a false color image on a slider with the range of colors.
But also all fill-in-blank, FIB Basic Item Types 'without clues' need to be extended in order to offer a usefull potential for mathematics:
  • STR1, i.e. Standard Single Fill-in-Blank – FIB-based rendering
    may be extended so that formulae can be entered by using the input notations of e.g. TEX, Mathematica, Maple, mathML, OpenMath [1] etc. Then questions like Give a definition of p. can be answered by 4 arctan1 or 2arcsin1 or the like.
    Even without systems powerful as computer algebra systems JavaScript functions embedded into pdf documents allow not only to implement all sorts of numerical algorithms ( but also to check the correctness of symbolic manipulations of algebraic expressions (
The fact that the learner can guess the correct answer to Basic Item Types 'with clues' has to be accounted for by special rating schemes, e.g. wrong answers are penalized (negative point penalty as is implemented in Kangaroo, an international competition for students in mathematics, see e.g. the last two years competitions

3.2  'Limited' Mapping

In spite of these inherent limitations we set off to match the identified question/answer types to IMS Basic Types. As it turns out, some questions belong to several categories: for example, recalling a fact may at the same time serve as an example, transfer may generate a new model, visualization serves to classify and the like.
Model, Construct
Questions 24, 27, 28, 52, and in the second lecture: 3D modeling
It is obvious that the modeling process cannot be mapped to IMS Basic Item Types. The modeling result may be mapped to extended STR1 if it can be represented as e.g. set of differential equations. If the model consisted e.g. of a labeled graph the input of became even more clumsy so that LID3 represents a last resort with clues. Otherwise a graphical input had to be processed, an input graph recognized, and the isomorphy between correct graph and input graph checked.
Questions 3, 6 - 8, 33, 34
All questions of the type "What kind of function is a given one?", "What class of differential equation is a given one?" etc. are mapped to LID1 or - if one wants not to give clues - STR1. The same holds for the variant with several correct answers, e.g. "How to visualize vector fields?".
Evaluate, Solve, Apply
Questions 10, 11, 12, 19, 30, 31, 35, 39, 40, 52
As in the case of modeling, all numerical results or algebraic formulae can be mapped to extended STR1. All other results again pose the same problems, e.g. Äpply a spanning tree algorithm"
Identify, Recognize
Questions 4, 7, 15 - 18, 21, 22, 32, 36, 41 - 47, 53
If it is accepted that the recipe for success in answering these questions is training then the implementation of such question/answer pairs in e-learning applications is rather important. Fortunately, mapping to LID or STR is adequate as long as STR takes synonyms into consideration which may be preventive since e.g. a plane can be verbally described in very many ways.
Typify, Find examples
Questions 24, 29, and in the second lecture: find examples for free form surfaces
The idea to let the students come up with any free form surfaces they might think of cannot be mapped to Basic Item Types. Again, LID or STR may act as a makeshift where STR prevents automatic marking.
To separate degenerated cases from non-pathological ones input of certain model parameters may be sufficient. This can be mapped to STR or NUM Basic Item Types.
Generalize, Transfer
Questions 27, and in the second lecture: generalize e.g. Bézier-curves in the plane to Bézier-surfaces in space
Generalization and transfer save work, generate ideas and insight. However, both are based on arguments which are hard to map to Basic Item Types, e.g. real valued functions exhibit properties which complex valued functions exhibit as well, e.g.
From     ln(a b)=lna+lnb     we derive     ln1=0 and ln(a/b)=lna-lnb
can be transfered to
From     (f g)¢=f¢ g+f g¢     we derive     (f/g)¢=(f¢ g-f g¢)/g2.
Again, the result can, but the process of generalization and transfer cannot be mapped to Basic Item Types.
Questions 1, 6, 7, 20, 25, 26, 33, 36 - 38, 47 - 51
Without recall of 'dormant' knowledge there is no ascent, and no consolidation and no synthesis of old and new knowledge! Especially e-learning application can be designed to stress recall wherever needed individually. Depending on the nature of the recall the remarks of this list apply.
Verify, Falsify
Questions 7, 20
Verification checks whether or not some solution solves a given problem, i.e. whether or not a solution solves e.g. a set of linear equations, a set of differential equations, whether or not an interpolating function passes through the given points etc. Falsification invalidates some hypothesis. Depending on the input, verification and falsification are mapped to STR1 or only extended STR1.
No computer algebra systems can check whether or not an input definition is correct. Disregarding the notational problem, the question to give some definition can be mapped only to STR without automatic marking.
Questions 8, 9, 14, 16, 22, 23, 29, and in the second lecture: certain sections of the shere, certain triangles in the shere etc.
The only candidate is LID3 with clues. In general, processing any graphical input in order to automatically check correctness seems not feasible!

3.3  Dealing with the Limitations of Basic Item Types

The typical interaction between teacher and learner asked for extending the Basic Item Types in two directions: first to allow to input mathematical text using a suitable interface based on TEX, mathML, etc. or the input notations similar to those used in computer algebra systems; and second to use computer algebra systems to check the correctness of the learners input.

The ability to graphically thumbsketch say function graphs, 2D and 3D objects - even paradox ones, that is any graphical representation is essential in order to communicate ideas, designs, etc. Thinking of thumbsketches of surfaces in space is seems not feasible to test such graphical inputs on 'correctness'. Hence this example circumstantiates a severe limitation of tests in e-learning application.

In addition, it became apparent that integration of visualization, animation or simulation systems is highly desirable to depict models, algorithms, procedures in action. To some extend this is achieved by interactive documents, i.e. documents which compute [8], allowing to study diverse algorithms in action, to compare precision and performance, to identify advantages and disadvantages etc., e.g. in numerical mathematics, or in cryptography, coding and probability,

The integration of virtual experiments is near at hand in science, i.e. physics, biology, medicine or computer architecture.
How processors work and how to design (pipeline) processors is demonstrated by stepwise refinement [7]. Extending processors to process new instructions leads to a deeper understanding of the design process, of performance and of verification issues. Processor emulators, like e.g. SPIM or WinDLX, s., ostensively demonstrate how the ensemble of control, functional units, and register file processes instructions (even so, in the Knowledge Factory for Computing Systems [6] simulations are reserved to to complement lectures and exercises). Therefore, it is near at hand to ask for integration of a working, extendable emulator into a learning unit for computer architecture. Then the students can estimate cost to benefit ratios of modifications of the hardware/software interface of a processor.

Another reaction on the limitation was the inventive and creative employment of Basic Item Types in e-learning applications. For example, in Mathe Online [2] there are besides LID, GRP (called puzzles or association) also two other forms of interactive tests, namely reading off coordinates (twice NUM1 or something like the inverse of LID6) or a cross word puzzle - only in the german version - training to locate and to position objects in coordinate systems (similar to LID7). By the way, questions of type "where is the mistake made?" - again only in the german version - are implemented by LID2 in Mathe Online.

4  Conclusion

The analysis - corroborated by a field study of teaching mathematics - revealed to what extend the IMS Basic Item Types can be used to test and assess the learning outcome.
As expected, testing declarative knowledge, in many cases, is easily mapped to IMS Basic Item Types whereas conceptual and particularly procedural knowledge can only be tested by the means specified by IMS Basic Item Types if the procedure is reduced to a result, i.e. if the process is scaled down to some final state.
Extending IMS Basic Item Types especially to input e.g. mathematics broadens the applicability e.g. when testing even declarative knowledge like How is p defined? and expecting some formula as answer. The example of computer architecture indicated that more computer supported test schemes are necessary: IMS Basic Item Types should be extended to integrate simulations, virtual experiments, interactive documents, etc.
The analysis of unformatted, unstructured, untagged text is costly, but analysis images, e.g. graphical thumbsketches, is really expensive if not prohibitive. As a matter of principle, to decide whether or not a graphical sketch say of a 3D body conveys the ideal type of this body establishes a general constraint to test learning outcome in e-learning applications.


Caprotti, O.; Cohen, A.M.; Cuypers, H.; Sterk, H.: OpenMath Technology for Interactive Mathematical Documents; Eindhoven University of Technology 2002

Embacher, F.; Oberhuemer, P.: Mathe online - A Gallery of Multimedia Learning Material;

Hockemeyer, C.: References on Intelligent Tutoring Systems; University of Graz, 2001

Holzinger, A.: Basiswissen Multimedia - Lernen: kognitive Grundlagen multimedialer Informationssysteme; Würzburg 2000 ~holzinge/multimedia/multimedia-lernen-theorie.html

IMS Question & Test Interoperability: ASI Best Practice & Implementation Guide - Version 1.2;

Lucke, U.; Tavangarian, D.; Vatterrott H.R.: The Use of XML for the Development of an Adaptive Multimedia Teaching and Learning System; NAISO (Natural and Artificial Intelligence Systems Organization): Proc. of NL2002, World Congress Networked Learning in a Global Environment, May 1-4, 2002 Berlin, ISBN 3-906454-31-2

Patterson, D.A.; Hennessy, J.L.: Computer Organisation & Design - The Hardware/Software Interface; Morgan Kaufman 1997
Risse, Th: Interactive Documents -– Yet Another Way to Activate Students in Mathematics; M. Auer, U. Auer: ICL2001 Workshop 'Interactive Computer Aided Learning - Experiences and Vision'; Kassel University Press 2001

Risse, Th: e-Learning - Ideas, Intentions, Illusions vs Status, Strategies, Stakes; 18th Scientific Colloquium, University of Pécs, 2002


Dr. Thomas Risse
Institute for Informatics & Automation
Hochschule Bremen, Flughafenallee 10, 28199 Bremen


1Even for the object to be standardized there is no common term!

2Image Hot Spot


File translated from TEX by TTH, version 3.12.
On 26 Aug 2002, 10:49.



e-learning, distance learning, distance education, online learning, higher education, DE, blended learning, MOOCs, ICT, information and communication technology, collaborative learning, internet, interaction, learning management system, LMS,

Current issue on Sciendo

– electronic content hosting and distribution platform

EURODL is indexed by ERIC

– the Education Resources Information Center, the world's largest digital library of education literature

EURODL is indexed by DOAJ

– the Directory of Open Access Journals

EURODL is indexed by Cabells

– the Cabell's Directories

EURODL is indexed by EBSCO

– the EBSCO Publishing – EBSCOhost Online Research Databases

For new referees

If you would like to referee articles for EURODL, please write to the Chief Editor Ulrich Bernath, including a brief CV and your area of interest.