Etests 
What Can be Assessed in Elearning Applications
Thomas Risse
Institute for Informatics & Automation, Hochschule Bremen, Germany
Key words: elearning, test, assessment, mathematics, IMS standards
Abstract:
In the light of different existing standards it is investigated what can be assessed by etests, i.e. by tests in elearning. In this way, potential and limits of such tests are outlined and illustrated by examples most of all from mathematics and computer architecture. The results of a little field study conducted in the authors lectures are given: analysis of teacherlearnerinteraction not only reveals how to map questionanswer to tests of the learning outcome in elearning applications but also in which directions these standardized tests ought to be extended. Additionally, implications for the design of intelligent tutoring systems are discussed.
1 Introduction
Integrated tests play an important role in elearning applications both for self assessment as well as for directions given to the learner to follow on individual learning paths. By nature, these tests easily assess the correctness of some solution. But it is inherently difficult to rate approach, modeling, methodic proceedings etc. by these tests.The objective of this paper is to analyse traditional, i.e. class lectures in order
 to classify the interaction between lecturer and student in the light of furthering the learning process and of testing the learning outcome
 to try to map the identified tests resp. test types onto the tests resp. test types (e.g. Basic Item Types in the IMS slang) used in elearning applications.
To this end, two worlds have to be bridged: on one hand, in the field study  a set of two lectures in mathematics given by the author  teacherlearner interaction in order to evaluate and test the learning outcome are analysed, and question/answer pairs are classified. On the other hand the existing standards for tests in elearning applications are presented, their inherent limitations are identified, and in spite of these limitations it is endeavoured to match question/answer pairs to these tests.
1.1 Standardization
Since quite a few years different national and international bodies have invested considerable efforts to establish standards for elearning applications^{1}. Industry, e.g. the Aviation Industry CBT Committee, AICC www.aicc.org, and special interest groups, e.g. the IEEE publish proposals for descriptions by metadata, architecture of learning technology systems, quality standards etc. in order, for example, to make possible to use learning material in different contexts. Some initiatives may serve as examples: Advanced Distributed Learning, ADL is an initiative initiated by the DoD standardizing elearning e.g. by proposing the Sharable Content Object Reference Model, SCORM, s. http://xml.coverpages.org/scorm.html using XML, www.adlnet.org
 ARIADNE, i.e. Alliance of Remote Instructional Authoring and Distribution Networks for Europe is a project of the European Union. A consortium provides educational metadata, tools, a user group etc. www.ariadneeu.org
 The Dublin Core Metadata Initiative develops interoperable
online metadata standards that support a broad range of purposes
and business models,
http://dublincore.org/  The IMS Global Learning Consortium, Inc. was founded as a project within the National Learning Infrastructure Initiative of EDUCAUSE. Users, software producers and activists provide XML specifications of metadata, question & test, learning resources etc. www.imsproject.org
 The Learning Technology Standards Committee, LTSC of the IEEE specifies a Learning Technology Systems Architecture, LTSA, proposes a standard for Learning Object Metadata, LOM, etc. http://ltsc.ieee.org
 The Open Universiteit Nederland proposes the standard Educational Modelling Language, EML focusing on didactic
aspects and using XML,
http://eml.ou.nl
Here, we refer to the IMS standard proposals because they take other drafts like the IEEE LTSCs into consideration and offer elaborated and well documented XML implementations.
1.2 Standardized Tests in eLearning Applications (IMS)
The common means to test the learning outcome in elearning applications have not changed to much since IBM or Lufthansa marketed their first products of this type in the sixties [9].Here, we list the IMS Basic Item Types [5] with the expected user action.
 LID
 Logical identifier (the user is expected to click on the right radio buttons or check boxes);
 XY
 XY coordinate (the user is expected to click on the right area on the screen/in an image);
 STR
 String (the user is expected to type some word or text);
 NUM
 Numeric (the user is expected to type some number or move a slider);
 GRP
 Logical groups (the user is expected to drag and drop objects into appropriate positions or response holders).
 LID
 The LID examples are:
 Standard True/False (textbased options) – choicebased rendering;
 Standard Multiple Choice (textbased options) – choicebased rendering;
 Standard Multiple Choice (imagebased options) – choicebased rendering;
 Standard Multiple Choice (audiobased options) – choicebased rendering;
 Standard Multiple Response (textbased options) – choicebased rendering;
 Multiple Choice with Single Image (imagebased options) – IHS^{2}based rendering;
 Multiple Response with Multiple Images (imagebased options) – IHSbased rendering;
 Multiple Choice (sliderbased options) – sliderbased rendering;
 Standard Order Objects (textbased objects) – objectbased rendering;
 Standard Order Objects (imagebased objects) – objectbased rendering;
 Connectthepoints (imagebased) – IHSbased rendering.
 XY
 The XY coordinate examples are:
 Standard Image Hot Spot (single image) – IHSbased rendering;
 Connectthepoints (imagebased) – IHSbased rendering.
 STR
 The STR examples are:
 Standard Single FillinBlank – FIB^{1}>ABbased rendering;
 Standard Multiple FillinBlank – FIBbased rendering;
 Standard Short Answer (text required) – FIBbased rendering.
 NUM
 The NUM examples are:
 Standard Integer FillinBlank – FIBbased rendering;
 Standard Real number FillinBlank – FIBbased rendering;
 Numerical entry with Slider – sliderbased rendering.
 GRP
 The GRP examples are:
 Standard Draganddrop (multiple images) – objectbased rendering.
2 A Field Study
The field study consisted in recording a set of two lectures in mathematics (selected by chance, namely the availability of recording equipment) and in categorizing question and answers of lecturer and students.In section 2.2 it will become obvious that the lecturer tried to make the students acquire [4]
 declarative knowledge, e.g. what is a vector field?
 conceptual knowledge, e.g. concept to intersect planes and graphs (partial function graphs or level curves) to visualize functions
 procedural knowledge, e.g. how to model a 3D object?
 to make students learn facts by repetition, etc
 to make students recognize mathematical objects, to stimulate explorative learning by using a 3D modelling and rendering program,
 to make (groups of) students to construct mental images of 3D objects and to model more and more complex 3D objects.
2.1 Background and Objectives of the Lectures
The lectures for students at the end of their second semester dealt with the introduction of functions of several variables. These students should know basic linear algebra, basic analytical geometry, analysis of functions of one variable (including power and Fourier series) and ordinary differential equations. The objective was to work out a set of simple examples of scalar and vector valued functions of one and several variables.The objectives can be summarized in the following table:
scalar fields  vector fields  
n\m  1  2  3  ...  
1  y=f(x)  curves^{*} in IR^{2}  curves^{*} in IR^{3}  
2 


surfaces^{*} in IR^{3}  
3 




4 




The case n=1 had been dealt with in a lecture preceding the set of lectures evaluated here. The students already got along some examples of scalar and vector fields. The concepts of partial function and level curves had been introduced. Parameter representation of e.g. circles in the plane or of helixes in space had been dealt with.
2.2 Question and Answers in Lectures
We now present questions posed and answers given in the set of lectures together with a categorization. The following representation is used: LQ: lecturers questions LA: lecturers answers LT: lecturers material/textTA: targeted students answers  intended/expected, but not given
SQ: students questions SA: students answers ST: students remarks/text
As it turns out, for our purposes it is sufficient here to describe only the first of the two recorded lectures focusing on the question/answer type of interaction. (MAI020628 and MAI020702 on www.weblearn.hsbremen.de/risse/papers/ICL2002 contain the full audio documentation in german.)
 LQ: what were the results and examples dealing with functions f:IR É D® IR^{m} dealt with in the last lecture? LT: objectives ...
 LQ: do you want examples to refresh results of the last lecture? SA: yes LT: let us write a function f:IR® IR^{m} as usual as [p\vec]:IR® IR^{m}, e.g. [p\vec](t)=[p\vec]_{o}+t([p\vec]_{1}[p\vec]_{o}).
 LQ: what kind of object is {[p\vec](t): t Î IR} ? SA: line through [p\vec]_{o} and [p\vec]_{1}
 LQ: what is [p\vec]_{1}[p\vec]_{o} ? SA: direction vector LT: same not only for lines in IR^{3} ...
 LQ: more examples? SA: no LT: let us deal with n=2, m=1
 LQ: how do you call z=f(x,y) ? TA: scalar field LT: recall scalar fields like f(x,y)=x+y or f(x,y)=x y. Let us consider another example, say z=f(x,y)=sinÖ{x^{2}+y^{2}}
 LQ: how is the graph of some f:IR^{2} É D® IR defined? and what kind of object? SA: graph(f)={(x,y,f(x,y)): (x,y) Î D} is a surface in IR^{3}
 LQ: how does one visualize the graph of z=f(x,y)=sinÖ{x^{2}+y^{2}} ? TA: sections, perspective visualizations LT: to get some idea, note that Ö{x^{2}+y^{2}} is inserted into the sine function.
 LQ: how do the level curves to some level z_{o} look like? TA: circles
 LQ: which (x,y) do solve z_{o}=f(x,y)=sinÖ{x^{2}+y^{2}} ? SA: arcsinz_{o}=Ö{x^{2}+y^{2}} LT: E.g., let z_{o}=[ 1/2]
 LQ: what is arcsin[ 1/2] ? LT: think of memory hook sinj_{i}=[(Öi)/2] for j_{i} Î {0,[(p)/6],[(p)/4],[(p)/3], [(p)/2]} SA: arcsin[ 1/2] = [(p)/6]
 LQ: which (x,y) do solve ([(p)/6])^{2}=x^{2}+y^{2} ? LT: think of Pythagoras or rewrite (x,y)=(x,y)[0\vec]=[(p)/6]
 LQ: wierd question? SA: yes LT: Recall vector length and distance. Consider the intersections with coordinate planes. [(p)/6], do call it r ... SA: circles
 LQ: how do partial functions z=f(x,0) and z=f(0,y) look like? SA: sine functions
 LQ: what kind of intersections do graphs of partial functions represent? LT: ... SA: intersections with coordinate planes
 LQ: how does z=f(x,x) look like? What kind of intersection? TA: graph with plane, i.e. graph(f)Ç{x=y}
 LQ: what function of one variable is z=f(x,x) ? SA: again a sine function
 LQ: what kind of intersection? LT: ...
 LQ: what is the equation of the intersecting plane? SA: x=y
 LQ: why is this a plane? TA: xy+0·z=0 LT: ...
 LQ: where is this plane in space? SA: The plane contains the zaxis LT: ...
 LQ: how does the intersection of x=y with the xyplane look like? SA: the diagonal
 LQ: visualize graph(f) LT: ... think of a snapshot of a concentric water wave ...
 LQ: how can we model friction? LT: ... the amplitude should be indirect proportional to the distance to the origin. Recall ordinary differential equations with characteristic polynomial with two pairwise conjugate complex zeroes ...
 LQ: recall: how do damped oscillations look like? LT: ...
 LQ: in e^{¼? ¼}sin(wt) fill in the exponent SA: e^{t/t}sin(wt)
 LQ: how can we tranfer this to our situation where damping is proportional to the distance to the origin? LT: ... SA: e^{Ö{x2+y2}}sinÖ{x^{2}+y^{2}}
 LQ: how can we introduce time into our function? LT: ... LT: consider the case n=3, m=1, for example z=f(x,y,t)=A_{o}cos2p(ntÖ{x^{2}+y^{2}}/l) ...
 LQ: visualize f LT: ... LQ: Visualize f for a fixed t LT: ...
 LQ: what is the level curve to a level of [(A_{o})/2] and fixed t ? SA: {(x,y,t_{o}):A_{o}cos2p(ntÖ{x^{2}+y^{2}}/l)=[(A_{o})/2]}
 LQ: solve for x and y SA: 2p(ntÖ{x^{2}+y^{2}}/l)=[(p)/3] LT: you can do better ... SA: ntÖ{x^{2}+y^{2}}/l = [ 1/6] LT: you can do better ... SA: nt[ 1/6] = Ö{x^{2}+y^{2}}/l LT: you can do better ... SA: x^{2}+y^{2}=l^{2}(nt[ 1/6])^{2}
 LQ: what are these level curves geometrically? SA: again circles LT: ... LT: consider the case n=2, m=2, i.e. a function [f\vec]:IR^{2} É D® W Ì IR^{2}
 LQ: how is [f\vec] called? SA: vector field LT: here [f\vec](x,y)=(u(x,y),v(x,y))
 LQ: what are u(x,y) and v(x,y) ? SA: scalar fields LT: vector fields are vectors of scalar fields,
e.g. matrix transforms like
æ
ç
èx y ö
÷
ø® æ
ç
ècosd sind sind cosd ö
÷
øæ
ç
èx y ö
÷
ø  LQ: evaluate the product SA: (xcosdysind,xsind+ycosd)
 LQ: what is the relation between the argument vector and the function value, the product? LT: identify, remember C , recall Euler, remember addition theorems of sine and cosine ...
 LQ: what is the polar representation of (x,y) ? SA: r(cosj,sinj)
 LQ: how are r and j determined? SA: r=Ö{x^{2}+y^{2}} and j = arctan[ y/x]
 LQ: what do you get inserting the polar representation of the argument into the product? SA: r(cosjcosdsinjsind, cosjsind+sinjcosd)
 LQ: what do you get applying the addition theorems? SA: r(cos(j+d),sin(j+d))
 LQ: what happened to the argument vector geometrically? SA: rotated by d LT: There are important matrix transformations of the plane, e.g. scaling, shearing and even  in homogeneous coordinates  translation ... Consider in the case n=2, m=3 the vector fields [f\vec](x,y)=(u(x,y),v(x,y),w(x,y)):IR^{2} É D® W Ì IR^{3} ...
 LQ: what is the dimension of D ? SA: twodimensional
 LQ: what is  in general  the dimension of W ? TA: twodimensional
 LQ: what do we get in W for some fixed y_{o} ? What is {[f\vec](x,y_{o}): (x,y_{o}) Î D} ? TA: a curve in IR^{3}, codomain of some partial function LT: ...
 LQ: what is the dimension of {[f\vec](x,y_{o}): (x,y_{o}) Î D} ? SA: onedimensional
 LQ: what is the dimension of W ? SA: twodimensional LT: The codomain of [f\vec](x,y):IR^{2} É D® W Ì IR^{3} are surfaces in space in parameter representation ... For example, consider three vectors [p\vec]_{o}, [r\vec]_{1}, [r\vec]_{2} corresponding to some plane in space
 LQ: what role do these three vectors play? SA: plane through [p\vec]_{o} with direction vectors [r\vec]_{1} and [r\vec]_{2}
 LQ: how do we get each point of the plane? TA: [p\vec]_{o} plus a linear combination of [r\vec]_{1} and [r\vec]_{2} LT: recall that planes in space as well as lines in the plane are hyperplanes ...
 LQ: what are the points of the line in IR^{2} or IR^{3} through [p\vec]_{o} with direction vector [r\vec]_{1} ? SA: [r\vec](t)=[p\vec]_{o}+t[r\vec]_{1}
 LQ: what are the points of the plane in IR^{3} through [p\vec]_{o} with direction vectors [r\vec]_{1} and [r\vec]_{2} ? SA: [r\vec](s,t)=[p\vec]_{o}+s[r\vec]_{1}+t[r\vec]_{2}
 LQ: what are the points of the plane in IR^{3} through [p\vec]_{o}, [p\vec]_{1} and [p\vec]_{2} ? SA: [r\vec](s,t)=[p\vec]_{o}+s([p\vec]_{1}[p\vec]_{o})+t([p\vec]_{2}[p\vec]_{o})
 LQ: how to model the intersecting plane x=y from above? ... SA: [r\vec](s,t)=[0\vec]+s([p\vec]_{1}[0\vec])+t([p\vec]_{2}[0\vec]) with [p\vec]_{1}=(1,1,0) and [p\vec]_{2}=(1,1,1) LT: not unique, not collinear, ...
 LQ: why do [r\vec](s,t)=[0\vec]+s[p\vec]_{1}+t[p\vec]_{2} and e.g. [r\vec]¢(s¢,t¢)=[0\vec]+s¢[p\vec]_{1}+t¢[e\vec]_{z} represent the same plane? LT: ... LQ: how to verify my claim?
 LQ: difficult? SA: yes LQ: at home or right now here SA: here right now LT: ... LT: let us summarize what we have learnt today ... in the next lecture we will consider ...
2.3 Types of Questions
Browsing this material, certain types of questions are readily identified. All questions obviously aim at triggering some action of the students. Therefore, we classify questions by the type of action they are meant to provoke. At the same time, we indicate by examples that a certain type of question accurs in many more circumstances when teaching mathematics to make up for the fact that the topic of the two lectures inevitably stressed certain actions (e.g. visualization) and neglected others (e.g. modelling). Model, Construct
 here e.g. model friction/damping, model free form 3D objects
modelling of course is the essential part in solving any problem (of problem based learning)  especially when application oriented problems posed in the real life, nonmathematical manner are to be attacked. And, geometric models are constructed as well as algorithms, bases, estimators, measures, etc.  Classify
 here e.g. level curves
but also e.g. classification of differential equations to determine the method by which to solve the differential equation  Evaluate, Solve, Apply
 here e.g. arcsin[ 1/2], addition theorems, Pythagoras
but also e.g. solutions of systems of linear equations, minima/maxima, integrals, zeroes of characteristic polynomials, Laplace transforms etc.  in the end, any mathematical procedure or algorithm  Identify, Recognize
 here e.g. plane or space curves in
parameter representation, scaling transformation
but also e.g. the other matrix transformations, time and space requirements of algorithms etc.  Typify, Find examples
 here both lessons were dedicated to
illustrate scalar and vector functions of one or several variables
by pertinent examples
but also e.g. to get a first idea of how an algorithm works, to illustrate theorems, to find counterexamples etc.  Generalize
 here e.g. visualisation of vector fields
by visualisation of each of its scalar fields
but also e.g. criteria for continuity or the like hold for real valued as well as complex valued functions etc.  Recall
 here e.g. polar representation,
but also e.g. recalling vector rotation in the plane is essential to understand and implement CORDIC algorithms  precognition of the basics is a prerequisite to learn any mathematical procedure or algorithm.  Verify, Falsify
 here e.g. check whether or not
the points of the parameter representation of the sphere
[r\vec](j,y) = r(cosycosj, cosysinj, siny)
lie on the sphere
but also e.g. verification whether solutions do solve the given problem or any consideration of plausibility  Visualize
 here e.g. any scalar field or a coordinate
of any vector field, 3D model
but also e.g. visualization of any solution, algorithm etc.
3 Mapping Questions to Tests
Let us try to match the different types of questions to the IMS Basic Item Types.3.1 Inherent Limitations
The IMS Basic Item Types can be partitioned into two subsets, Basic Item Types providing a list of possible answers or solutions 'with clues' or 'without clues'.The set of Basic Item Types 'without any clues' consists of STR1, STR2, STR3, NUM1, and NUM2. All other Basic Item Types present possible solutions and hence clues. Here, the learner is able to identify the correct answer by elimination.
Other Basic Item Types may be extended to types 'without clues', e.g.
 LID8, i.e. Multiple Choice (sliderbased options) –
sliderbased rendering
may be extended to real valued sliders: for example, to approximately set Ö2 on a slider with ticks 0, 1, 2 and 3 or to set the most frequent color in a false color image on a slider with the range of colors.
 STR1, i.e. Standard Single FillinBlank – FIBbased rendering
may be extended so that formulae can be entered by using the input notations of e.g. T_{E}X, Mathematica, Maple, mathML, OpenMath [1] etc. Then questions like Give a definition of p. can be answered by 4 arctan1 or 2arcsin1 or the like.
Even without systems powerful as computer algebra systems JavaScript functions embedded into pdf documents allow not only to implement all sorts of numerical algorithms (www.weblearn.hsbremen.de/risse/MAI/docs/numerics.pdf) but also to check the correctness of symbolic manipulations of algebraic expressions (www.weblearn.hsbremen.de/risse/MAI/docs/vorkurs.pdf)
3.2 'Limited' Mapping
In spite of these inherent limitations we set off to match the identified question/answer types to IMS Basic Types. As it turns out, some questions belong to several categories: for example, recalling a fact may at the same time serve as an example, transfer may generate a new model, visualization serves to classify and the like. Model, Construct
 Questions 24, 27, 28, 52,
and in the second lecture: 3D modeling
It is obvious that the modeling process cannot be mapped to IMS Basic Item Types. The modeling result may be mapped to extended STR1 if it can be represented as e.g. set of differential equations. If the model consisted e.g. of a labeled graph the input of became even more clumsy so that LID3 represents a last resort with clues. Otherwise a graphical input had to be processed, an input graph recognized, and the isomorphy between correct graph and input graph checked.  Classify
 Questions 3, 6  8, 33, 34
All questions of the type "What kind of function is a given one?", "What class of differential equation is a given one?" etc. are mapped to LID1 or  if one wants not to give clues  STR1. The same holds for the variant with several correct answers, e.g. "How to visualize vector fields?".  Evaluate, Solve, Apply
 Questions 10, 11, 12, 19, 30, 31, 35, 39, 40, 52
As in the case of modeling, all numerical results or algebraic formulae can be mapped to extended STR1. All other results again pose the same problems, e.g. Äpply a spanning tree algorithm"  Identify, Recognize
 Questions 4, 7, 15  18, 21, 22, 32, 36, 41  47, 53
If it is accepted that the recipe for success in answering these questions is training then the implementation of such question/answer pairs in elearning applications is rather important. Fortunately, mapping to LID or STR is adequate as long as STR takes synonyms into consideration which may be preventive since e.g. a plane can be verbally described in very many ways.  Typify, Find examples
 Questions 24, 29,
and in the second lecture: find examples for free form surfaces
The idea to let the students come up with any free form surfaces they might think of cannot be mapped to Basic Item Types. Again, LID or STR may act as a makeshift where STR prevents automatic marking.
To separate degenerated cases from nonpathological ones input of certain model parameters may be sufficient. This can be mapped to STR or NUM Basic Item Types.  Generalize, Transfer
 Questions 27, and in the second lecture:
generalize e.g. Béziercurves in the plane to Béziersurfaces in space
Generalization and transfer save work, generate ideas and insight. However, both are based on arguments which are hard to map to Basic Item Types, e.g. real valued functions exhibit properties which complex valued functions exhibit as well, e.g.
From ln(a b)=lna+lnb we derive ln1=0 and ln(a/b)=lnalnb
can be transfered to
From (f g)¢=f¢ g+f g¢ we derive (f/g)¢=(f¢ gf g¢)/g^{2}.
Again, the result can, but the process of generalization and transfer cannot be mapped to Basic Item Types.  Recall
 Questions 1, 6, 7, 20, 25, 26, 33, 36  38, 47  51
Without recall of 'dormant' knowledge there is no ascent, and no consolidation and no synthesis of old and new knowledge! Especially elearning application can be designed to stress recall wherever needed individually. Depending on the nature of the recall the remarks of this list apply.  Verify, Falsify
 Questions 7, 20
Verification checks whether or not some solution solves a given problem, i.e. whether or not a solution solves e.g. a set of linear equations, a set of differential equations, whether or not an interpolating function passes through the given points etc. Falsification invalidates some hypothesis. Depending on the input, verification and falsification are mapped to STR1 or only extended STR1.
No computer algebra systems can check whether or not an input definition is correct. Disregarding the notational problem, the question to give some definition can be mapped only to STR without automatic marking.  Visualize
 Questions 8, 9, 14, 16, 22, 23, 29,
and in the second lecture: certain sections of the shere,
certain triangles in the shere etc.
The only candidate is LID3 with clues. In general, processing any graphical input in order to automatically check correctness seems not feasible!
3.3 Dealing with the Limitations of Basic Item Types
The typical interaction between teacher and learner asked for extending the Basic Item Types in two directions: first to allow to input mathematical text using a suitable interface based on T_{E}X, mathML, etc. or the input notations similar to those used in computer algebra systems; and second to use computer algebra systems to check the correctness of the learners input.The ability to graphically thumbsketch say function graphs, 2D and 3D objects  even paradox ones, that is any graphical representation is essential in order to communicate ideas, designs, etc. Thinking of thumbsketches of surfaces in space is seems not feasible to test such graphical inputs on 'correctness'. Hence this example circumstantiates a severe limitation of tests in elearning application.
In addition, it became apparent that integration of visualization, animation or simulation systems is highly desirable to depict models, algorithms, procedures in action. To some extend this is achieved by interactive documents, i.e. documents which compute [8], allowing to study diverse algorithms in action, to compare precision and performance, to identify advantages and disadvantages etc., e.g. in numerical mathematics, www.weblearn.hsbremen.de/risse/MAI/docs/numerics.pdf or in cryptography, coding and probability, www.weblearn.hsbremen.de/risse/MAI/docs/epuzzles.pdf
The integration of virtual experiments is near at hand in science,
i.e. physics, biology, medicine or computer architecture.
How processors work and how to design (pipeline) processors
is demonstrated by stepwise refinement [7]. Extending
processors to process new instructions leads to a deeper
understanding of the design process, of performance and of
verification issues. Processor emulators, like e.g. SPIM or WinDLX, s. www.weblearn.hsbremen.de/risse/RST,
ostensively demonstrate how the ensemble of control, functional units,
and register file processes instructions (even so, in the Knowledge Factory for Computing Systems [6]
simulations are reserved to to complement lectures and exercises).
Therefore, it is near at hand to ask for integration of a working,
extendable emulator into a learning unit for computer architecture.
Then the students can estimate cost to benefit ratios of modifications
of the hardware/software interface of a processor.
Another reaction on the limitation was the inventive and creative employment of Basic Item Types in elearning applications. For example, in Mathe Online [2] there are besides LID, GRP (called puzzles or association) also two other forms of interactive tests, namely reading off coordinates (twice NUM1 or something like the inverse of LID6) or a cross word puzzle  only in the german version  training to locate and to position objects in coordinate systems (similar to LID7). By the way, questions of type "where is the mistake made?"  again only in the german version  are implemented by LID2 in Mathe Online.
4 Conclusion
The analysis  corroborated by a field study of teaching mathematics  revealed to what extend the IMS Basic Item Types can be used to test and assess the learning outcome.As expected, testing declarative knowledge, in many cases, is easily mapped to IMS Basic Item Types whereas conceptual and particularly procedural knowledge can only be tested by the means specified by IMS Basic Item Types if the procedure is reduced to a result, i.e. if the process is scaled down to some final state.
Extending IMS Basic Item Types especially to input e.g. mathematics broadens the applicability e.g. when testing even declarative knowledge like How is p defined? and expecting some formula as answer. The example of computer architecture indicated that more computer supported test schemes are necessary: IMS Basic Item Types should be extended to integrate simulations, virtual experiments, interactive documents, etc.
The analysis of unformatted, unstructured, untagged text is costly, but analysis images, e.g. graphical thumbsketches, is really expensive if not prohibitive. As a matter of principle, to decide whether or not a graphical sketch say of a 3D body conveys the ideal type of this body establishes a general constraint to test learning outcome in elearning applications.
References
 [1]
 Caprotti, O.; Cohen, A.M.; Cuypers, H.; Sterk, H.:
OpenMath Technology for Interactive Mathematical Documents;
Eindhoven University of Technology 2002
www.win.tue.nl/~hansc/lisbon.pdf  [2]
 Embacher, F.; Oberhuemer, P.:
Mathe online  A Gallery of Multimedia Learning Material; www.univie.ac.at/future.media/moe/
 [3]
 Hockemeyer, C.: References on Intelligent Tutoring Systems; University of Graz, 2001
http://wundt.kfunigraz.ac.at/hockemeyer/itsbib.html  [4]
 Holzinger, A.: Basiswissen Multimedia  Lernen:
kognitive Grundlagen multimedialer Informationssysteme;
Würzburg 2000 wwwang.kfunigraz.ac.at/ ~holzinge/multimedia/multimedialernentheorie.html
 [5]
 IMS Question & Test Interoperability:
ASI Best Practice & Implementation Guide  Version 1.2; www.imsglobal.org
 [6]
 Lucke, U.; Tavangarian, D.; Vatterrott H.R.:
The Use of XML for the Development of an Adaptive Multimedia
Teaching and Learning System; NAISO
(Natural and Artificial Intelligence Systems Organization):
Proc. of NL2002, World Congress Networked Learning in a
Global Environment, May 14, 2002 Berlin, ISBN 3906454312
www.wwrproject.de/de/infopool/veroeffentlichungen.htm  [7]
 Patterson, D.A.; Hennessy, J.L.: Computer Organisation & Design  The Hardware/Software Interface; Morgan Kaufman 1997
 [8]
 Risse, Th: Interactive Documents –
Yet Another Way to Activate Students in Mathematics;
M. Auer, U. Auer: ICL2001 Workshop 'Interactive Computer Aided Learning 
Experiences and Vision'; Kassel University Press 2001
www.weblearn.hsbremen.de/risse/papers/ICL2001  [9]
 Risse, Th: eLearning  Ideas, Intentions, Illusions
vs Status, Strategies, Stakes; 18^{th} Scientific Colloquium,
University of Pécs, 2002
www.weblearn.hsbremen.de/risse/papers/Kolloq18
Author:
Dr. Thomas Risse
Institute for Informatics & Automation
Hochschule Bremen, Flughafenallee 10, 28199 Bremen
risse@hsbremen.de
Footnotes:
^{1}Even for the object to be standardized there is no common term!
^{2}Image Hot Spot
^{1}FillinBlank
File translated from T_{E}X by T_{T}H, version 3.12.
On 26 Aug 2002, 10:49.