• About EURODL |
•  Sections |
•  Subscription |
•  For Authors |
•  Special Issues |
•  Archives
•  

• Peer-reviewed articles |
•  Brief items |
•  Reviews

TLTP was always intended to have an impact on HEIs. The overall aim of making "teaching and learning more productive by harnessing modern technology" was expected to "help institutions to respond effectively to the current substantial growth in student numbers, and to promote and maintain the quality of their provision" (UFC Circular 8/92).

Twelve of the seventy-six TLTP projects were "institutional projects" accounting for £4.5 million over programme Phases 1 and 2. Much of this chapter is based on institutional projects file data, and interviews with participants in these projects. However the institutional dimension in TLTP is not confined to institutional projects.

"Institutional support" was a criterion for the evaluation of all TLTP bids. Furthermore the invitation to tender raises a number of institutionally relevant issues and questions, e.g. "changes in the culture of teaching and learning in the HE sector"; the extent to which TLTP projects have "been part of co-ordinated attempts on the part of institutions to raise awareness of the potential of new technology"; "the support received by projects/consortia from their host institutions" - all of which have programme-wide implications. For this reason although much of the material in this chapter is derived from "institutional" TLTP projects it also incorporates relevant information from our analyses of consortia projects.

At the most general level project proposals, progress reports and other TLTP documentation often couple the programme with "changing the organisational culture" of HE and of HEIs. The term also figured prominently during project visits and regional workshops.

 As used by TLTP participants "organisational culture" is so vague and ill-defined a term, that the temptation is to ignore it. For some it is associated with a greater priority for teaching and learning; the enthusiastic use of new technology; and for others a more coherent direction for HEIs and departments. (It is certainly very distant from the way the concept is understood by organisational analysts and social anthropologists who invented it.) Nonetheless it does signify an important debate about the current state of the HE sector that deserves "unpacking". Changing the "organisational culture" appears to be used as a shorthand for the need to pursue fundamental and systemic change in HEIs.

 For those who hold to this view, TLTP is seen as part of a broader picture - consistent with other programmes and initiatives. Those who argue for such change appear to accept that it is not possible to tinker at the margins: the "taken for granted" organisational rules of the academy are being questioned.

 The logic of mass higher education, public sector financial constraints and economic competitivity requires root and branch change. HEIs have no choice if they are to deliver quality student learning, maintain academic discourse and sustain innovative research.

General attitudes towards TLTP often reflected this broader debate about values and direction in the HEI sector. Positive or negative attitudes towards the programme appeared to be associated with positions adopted in relation to the perceived need for fundamental and systemic change in HE, as well as to the potential contribution of technology to such change. Institutional projects in TLTP have usually followed some combination of the following five implementation strategies:

• Staff training, development and support;
• Strengthening internal specialist resources;
• Awareness raising and resource access;
• Strategic coordination, and
• Technological infrastructure development.

Staff development has now begun to feature seriously in the development plans of most HEIs. Training for new teaching staff, and skill enhancement for existing staff is part of a refocussing on teaching and learning. These activities probably account for the frequency with which staff development has been employed as an implementation strategy within TLTP. The precise configuration of staff development adopted has varied considerably and has included:

• training lecturers as courseware authors;
• targeting staff in "lead" departments;
• general awareness raising workshops regarding technology in teaching and learning;
• support and advice for individual staff members.

Despite the level of activity noted this implementation strategy has not been without its problems. In particular we would highlight:

• A tendency to focus more on technology awareness and skills than on teaching and learning. This is by no mean universally so - the few cases where pedagogy is in general well covered, point up the contrast.
• Links with wider institutional staff development have sometimes been uneasy. Also as staff development is the chosen vehicle for several other HEI initiatives and programmes the lack of institute-wide staff development strategies in practice has been more problematic.
• A tendency to problematise teaching staff - "the problem is lecturers" attitudes" - can create a difficult atmosphere for staff development.
• Lack of support by Heads of Department for staff development and for technology enhanced teaching and learning has sometimes undermined individual lecturer’s commitment and ability to use their new found skills and enthusiasm.
• The incentive structure of Universities continues not to reward the development of teaching and learning. The recent Research Assessment Exercise underlined this imbalance and on occasions undermined TLTP staff development efforts.

More positive tendencies can also be discerned. In particular the TQA has evidently had a positive impact on the use of technology in support of teaching and learning. Moves toward greater cross HEI coordination has also been another consequence of TLTP. (See below)

HEIs with pre-existing specialist centres in utilising new technology in support of teaching and learning have had considerable advantages in implementing TLTP.

Indeed several institutional TLTP projects built their proposals around such specialist resources. These HEIs and others also used TLTP as a vehicle for strengthening these specialist units and the skills of their staff. The contribution of such units may be to provide technical advice and consultancy to academic departments, to implement and design specific teaching and learning "solutions", to customise courseware, to contribute to staff development, to provide specialist services across the HE sector.

Several problems can also be noted with regard to this implementation strategy. In particular,

• As in any change process, CBL (and related) specialists can easily come to be seen as the "owners" of innovation. Several projects had difficulty in arriving at a reasonable balance between "doing it for lecturers" or expecting lecturers to become experts, say in authoring or hypertext, on the basis of very limited exposure, training or practice.
• No HEI can expect to cover all aspects of technology in HE. Most HEIs start from the basis of what they know best which given the enormous diversity of specialist needs - multi-media, simulation, student assessment, WWW, authoring, evaluation, courseware development, hypermedia, pedagogics, conferencing, etc. - can only cover a small part of what is needed. Specialisation, as always, raises the question of where to locate specialist resources and what is the scale unit for such location decisions. It suggests a serious need for cross-HEI cooperation - perhaps built around the recently established TLSTN network of regional support centres. (See below section on Dissemination and Transfer).

General awareness raising via workshops, disseminating information on conferences, and acting as a "broker" for external information sources has been a common activity of TLTP projects at HEI level. This is discussed further below (see section on dissemination and transfer). At this point it should be noted that internal dissemination has been uneven across HEIs within TLTP. A lack of knowledge within HEIs of other TLTP efforts (e.g. across consortia, or between an institutional department participating in a project and a subject consortium) was surprisingly common.

Strategic coordination has been both a precursor and consequence of TLTP. As circular letter 8/92 explained: "Projects are expected to involve main teaching departments and to have active support of participating institutions senior management". Most of those HEIs reviewed and visited as part of this evaluation had on paper, at least, confirmed HEI support for the project at a senior level. Several were able to demonstrate a track record of coherent corporate developments, e.g.

• Developing teaching and learning plans and setting up associated institution wide committees;
• Funding innovations in teaching and learning;
• Serious investment in computing infrastructure;
• Merging information systems and libraries;
• Asking departments to include in plans their teaching and learning priorities;
• Creating central media/technology resource units.

In reality these developments are of varying degrees of seriousness. An "innovation fund" to support new and often technology-led developments in teaching and learning may be £20k or £200k per annum.

Vice Chancellors rhetoric about University vision, mission and plans may bear little relationship to what happens in terms of resource allocation, promotion decisions and investment in infrastructure. However it is striking how many HEIs have committed resources to TLTP follow-up: on-funding of specialist staff; further funding of courseware development; new commitments to staff development; further investment in PCs for staff and students; network improvements, etc.

Several HEIs whose institutional commitment to TLTP proved to be stronger on paper than in practice, nonetheless began to move significantly once TLTP was underway. "Many ... particularly at senior level levels are only now becoming aware of the scale of the TLT Programme and are still not fully appraised of its implications." (12 month TLTP Progress Report) Raising the awareness of senior management was one of the positive outcomes of several TLTP institutional and consortium projects we reviewed. In some cases TLTP outcomes were more specific. Setting up new innovation funds, strengthening specialist units, further investment in teaching and learning, auditing current HEI practice, infrastructural investment, - have all followed after TLTP. Senior management we interviewed recognised that TLTP had contributed to the emergence of new institution-wide reviews, policies, coordination mechanisms and investments.

This is not to overstate TLTP’s efforts. The phrase "contributed to" is used above advisedly and a number of reinforcing pressures were acknowledged - including TQA, funding cutbacks, institutional rationalisations and other HEFCE programmes such as eLib. Furthermore, in some HEIs no real progress was reported: "Senior management support is still very weak", "awareness is restricted to enthusiastic individuals", "senior management even resent having to contribute the overhead component".

Technology infrastructure is at various levels of sophistication and scale in the HEIs we reviewed and visited. Courseware requires, at a minimum, access to PCs (or Macs) and for many of the applications being developed within TLTP, far more powerful broad-band networks. The more ambitious institutional projects had already invested substantially in IT infrastructure.

 They had variously:

• Installed networks capable of high speed and high band width communications;
• Provided PCs for all staff;
• Initiated student PC access initiatives - in libraries, student residences and resources centres;
• Had successful CTI centres on-site;
• Entered into joint ventures with commercial companies to develop software, workstations, and common end-user interfaces.

In contrast, some HEIs were described as "short of hardware", had no IT support for students or staff, and had resources and know-how confined to computing and IT departments.

Among problems identified we would highlight:

• the different levels and directions that investment in IT has taken, creating de facto barriers to transfer of even stand-alone, let alone communications-based applications;
• dangers of "reinventing the wheel" and not being sufficiently aware of experience in other HEIs;
• a tendency for IT infrastructure to be engineering and systems-led rather than teaching and learning led;
• a tendency for the pace of IT innovation to overtake plans that make sense when first envisaged; an example that has been evident in TLTP is the growth of the internet and in particular WWW which has cut across various interface and communications plans.

HEIs are faced with the general problems of balancing "bottom-up" and "top down" tactics, whatever their implementation strategies. For example, it is now common in HEIs to create a central fund to which departments may bid, or to offer academics the opportunity to shape initiatives (such as TLTP and in the past EHE) in relation to their needs.

 "Ownership" by academic and Heads of Departments given the decision-making processes of HEIs is usually seen as vital. However it is only possible to go so far in decentralising initiatives before some more coherent, and often less voluntary system, is needed. This is certainly the case for teaching and learning, staff development, reward and promotion systems and infrastructure.

 One impression from the HEIs we have considered as part of this evaluation, is that the limits of this approach to programme implementation may now be being reached. Further initiatives may have to require of HEIs more than public statements of support and may also need to be resourced in a less programmatic basis.

Finally, it is noteworthy that the starting position in different HEIs supported through TLTP is enormously varied. This is reflected in apparent "value for money": a similar sum of money may pay for far more apparent output and activity in one HEI setting than in another. Usually more output and activity is associated with a greater level of pre-existing learning technology related activity in the HEI. In many ways we see this as a reasonable use of programme funds, if this is a conscious strategy to promote innovation in less advanced HEIs. (It is not always clear how far this has been an explicit and thought through strategy.)

However, two caveats are in order. First, it is sometimes easier for HEIs with a high level of pre-existing activity to appear to be doing more within a programme such as TLTP - activities from other programmes and funding sources spill over and are often difficult to disentangle. Second, HEIs with less prior experience are not only unable to repackage "spillover" results, but become genuinely preoccupied with intangible strategies, coordination and institutional management issues for which output and activity can be difficult to demonstrate.

Subject consortia are intended to support the widest possible take up of TLTP sponsored courseware (and related innovation) throughout the HE sector. The extent of take-up and use is still difficult to judge though early indications are beginning to be available. This section of the report therefore concentrates on transferability, the likelihood of transfer as much as on actual transfer and use, and on the conditions that are known to support transfer.

The consortia principle has not always been followed through in practice. We have encountered non functioning consortia, effectively dominated by one partner and others that pay as much attention to the wider subject community in the UK as to Consortia members. Materials and courseware developed is likely to reflect these consortia’s structures. Thus some consortia have produced materials for only one HEIs needs, whilst others have seen themselves as meeting on entire national subjects needs. The latter for example may have gone to considerable lengths to consult colleagues across the entire sector.

 Such consultations (via survey, piloting, roadshows etc.) do not however guarantee transfer, several significant factors mediate the direct effects of consortia intention. These include in particular:

• Subject or disciplinary traditions;
• Proposed uses of courseware;
• Numbers of potential users;
• The stability of subject knowledge.

Different subjects have different internal structures that influence the likelihood of inter HEI co-operation. Subjects that are formally regulated like law or medicine are more likely to have common course elements and modules than, say, economics or geography. Less regulated subjects such as chemistry with standard laboratory practicals and experiments, are still likely to share common course elements. Some subjects also have little tradition of cross institutional co-operation and may teach the same course content in very different ways. Some indeed have come together in "joint ventures" for the first time via TLTP. It might be possible to posit a "scale" of subjects from say Law through to Politics and Art-History that other things being equal will be more or less likely to support transfer.

However an important qualification to this argument is how technology is to be used in support of teaching and learning.

Many of the materials developed in TLTP have been free standing resource materials capable of being re-used by lectures in a variety of ways. We were frequently told during fieldwork for this evaluation that what was being produced was "to complement not substitute for lectures" Such project outputs were not embedded in a particular course structure or pedagogic model. Rather they were transportable because, for instance, they were not highly interactive, nor did they assume a common curricula. Arguably in order for the more sophisticated uses of IT to be usable and transportable across HEIs, common curricula would be needed across subject departments in sets of collaborating HEIs.

The number of potential users is also an important influence on likely take-up and transfer. Previous "open-learning" initiatives have faced similar constraints and commercial publishers involved in such initiatives are also much influenced by potential market size, given the upfront costs of courseware investment. A concentration on first year undergraduate courses is one sign of the same processes operating TLTP. Indeed a shift to first year and away from more advanced level courseware has been discernible in some TLTP projects.

 Moves towards modulisation and greater course specialisation and choice, set new limits to transferability across departments.

An associated factor is the stability of subject knowledge. Some parts of courses are relatively unchanging - the same building blocks and fundamentals continue over decades rather than years. Other parts of the self same course - for example in some scientific and technical areas - knowledge is expanding or changing very rapidly. Change may also come about because of curriculum requirements or even legal requirements (e.g. health and safety). Those we interviewed saw the likely longevity (or otherwise) of material as a key factor in transfer - and as an important consideration in course maintenance and renewal.

It is important to note that transfer does not only happen with courseware or course-related resources. Within HEIs, departments "neighbouring" other TLTP consortia department can adopt an idea or concept rather than particular courseware (for example we encountered a Physics department that copied a TLTP strategy from a Chemistry department, but outside of TLTP funding). Transfer may also come from previous research conducted under SERC or ESRC; or from taught course material being adapted to a distance teaching environment. These instances, both of which were encountered in the projects we reviewed as part of this evaluation, also raise the question of customising material, discussed further below.

Few of the TLTP projects we encountered had comprehensive dissemination plans. Dissemination appears to have been considered for the first time mid-way through many projects term. Evidence from similar initiatives tends to confirm that unless dissemination is built in from the beginning it does not happen. Although comprehensive dissemination plans are scarce, considerable efforts have gone into dissemination.

These efforts have included:

• Use of existing professional/academic networks;
• "Roadshows" and visits to other HEIs;
• Surveys of the subject community;
• Establishing databases of potential users;
• Involvement of many in "Beta" testing and piloting;
• Demonstrations at TLTP conferences;
• Use of World Wide Web;
• Commercial Partnerships;
• Design of subscription scheme for HEIs;
• Issuing newsletters.

Those who saw themselves engaged in a continuing activity were understandably concerned with the on-funding of TLTP projects. Arrangements in the UK whereby TLTP materials are available free to the UK HE sector means that there is little incentive to market and sell TLTP products in the UK. On the other hand, there is no reason to believe that a more commercial strategy would have led to greater uptake of TLTP materials. The more systematic inclusion of commercial partners (software houses, publishers, etc.) in consortia might have helped identify commercial market niches more reliably. As it was, projects have relied mainly on further funding from institutions and foreign sales. For marketing, projects have tended to rely on existing resource centres, in particular CTI centres which have been highly rated by most of those we interviewed and visited.

More recently and beyond the terms of reference for this evaluation, the establishment of regional TLSTNs have created a new resource for dissemination and transfer. Clearly, there is a need for centres of expertise in this area - able to raise awareness, provide training, customise material and analyse needs. The very limited budgets and uncertain future of TLSTNs suggest caution about this existing model. We have been impressed by the Scottish Higher Education Funding Council’s LTDI (Learning Technology Dissemination Initiative) as a more extensive attempt to create appropriate regional resources. The success of TLTSNs and of LTDI ultimately depends on the model of inter-HEI cooperation expected in the HE sector of the future. Arguably, the ethos of Scottish universities together with a history of inter-HEI collaboration and, more recently, the establishment of Metropolitan Area Networks (MANs) makes it easier to pursue such a model in Scotland than in other parts of the UK.

Evidence of actual usage of TLTP materials is sparse. Most project staff we visited told us of plans, piloting, likely consortia uptake and early stage commercialisation plans. Some projects were able to offer "proxy" measures. For example, a video disc project, the uptake of which depended on the purchase of a particular playback machine, was able to present figures for the purchase of such equipment.

In general, however, TLTP projects as "producers" of courseware were often more optimistic about future uptake than HEI departments in the same subjects. In one university, Heads of Departments were systematically asked about their familiarity with and likely usage of TLTP products.

A quantitative summary of these responses, in the table below, shows that in the fourteen departments surveyed there were six cases of TLTP use, nine of non-use and five of materials review of "trialing". (These totals adding up to twenty in the fourteen departments cover more than one project in several of the departments concerned).

Qualitatively, their replies indicated that:

• There was general awareness of the existence of TLTP products but this had not always extended to review/testing;
• Direct involvement in a consortium (but not as a "sleeping partner") made it more likely that materials would be taken up;
• Assessments of TLTP products was not always favourable when compared with other already existing subject material.

In only a few cases had detailed assessment of materials, involvement in design and attendance at workshops taken place.

In another HEI, an ad hoc "trawl" through TLTP mailing lists and those interested in Computer Assisted Learning elicited ten responses from nine departments. All respondents knew of potentially relevant TLTP material. Six departments were either using TLTP materials or planned to do so, two did not plan to use the available material and/or were still uncertain. Concerns about both content and technical quality were identified by those who were not planning to use TLTP products or who remained uncertain.

A more extensive evaluation of TLTP could usefully repeat and extend such HEI based surveys of take-up and usage once more TLTP products are available.

Finally, it is worth reviewing the positive factors and the barriers to TLTP uptake identified by those we interviewed.

Positive factors that contributed to TLTP uptake included:

• The right level of material;
• Portable and usable software;
• The existence of an effective CTI in the subject area concerned;
• General support in the HEI for technology supported learning;
• Discipline/subject tradition that favours cooperation;
• Growth of World Wide Web;
• An active and relevant staff development unit.

Barriers to TLTP uptake included:

• Lack of specialist resources especially to advise and customise;
• Lecturers too overstretched to innovate;
• Inadequate access to PCs;
• Poor incentives to disseminate;
• Lack of staff skills;
• Over-specificity of learning context presented.

Innovative programmes, particularly in the field of new learning technologies where there remains considerable uncertainty about the most effective ways of designing and implementing technology-mediated products and services within the curriculum, can greatly benefit from evaluation. Evaluation would desirably engage with the development process and help to shape its direction as well as to contribute to the continuous improvement of exploitable outputs. Evaluation can also capture the learning from a programme that results from reflecting on difficulties and failures as well as from achievements and successes.

Despite the innovative nature of the TLTP programme, evaluation has not been a significant feature, lacking as it does a coherent framework and appropriate resourcing for evaluation at a programme level. Our view, echoed by many of those whom we interviewed, is that an effective evaluation strategy would have enhanced the value of the programme and contributed to the achievement of its objectives. Its relatively low profile in TLTP thus represents a major missed opportunity for learning about the process, what worked and did not work, and what the implications are for future programmes.

Whilst there was support for project level evaluation at the outset of the programme and this was built in as a requirement to the project bids, it would seem that evaluation budgets were an expendable item when it came to making budget cuts as part of the funding review. In the view of some HEIs, the lack of strong central support for evaluation made it an obvious target, particularly in those projects where there was no advocate for evaluation among members of the Steering Group or on the part of the project manager. We heard of several instances where evaluation was not carried out because of the need to make savings or because it was no longer perceived as a requirement. Not all projects responded in this way, however, and a substantial number proceeded with their evaluations as planned and in some cases the HEIs contributed additional resources.

We came across some good, if isolated, examples of project level evaluations. In general, though, evaluations were scanty, partial and showed little evidence of state-of-the-art thinking about evaluation theory and practice in the field of learning technologies. Software design is an established domain of practice, and one that many TLTP project teams are engaged in. Nonetheless, despite the advances of embedding evaluation into the software design process, the TLTP project participants seemed unaware of such developments beyond the more standard practices of prototype and ß testing.

Across the whole programme, project evaluation reports provide relatively little information beyond formative assessments that have contributed to the design and continuous improvement of their products. Thus we find few examples of the lessons that could be useful to others in some future programme or project, and relatively little about the actual use, and learning outcomes of these innovative teaching and learning technologies. The lack of summative evaluations at a project level has rendered this external summative evaluation less comprehensive and coherent than it might otherwise have been. Ideally, projects would have been requested to provide project level data that could contribute to the programme wide evaluation as well as meeting their own specific local data collection needs.

Projects could perhaps be excused for not taking evaluation more seriously. The strong impression we gained from HEI respondents was that evaluation reports and findings, like regular progress reports, were seen as fulfilling a contractual duty but that they were not acted upon by the central management team and were certainly not used as a basis for feedback and dialogue. Rather, it appeared to us that "flagging up" problem areas by the central programme co-ordinator was ad hoc and informal - experienced as supportive by the projects themselves but arguably not an effective approach for ensuring that maximum benefit was derived from these innovative, experimental projects.

Approaches to evaluation. We found a number of different approaches taken to evaluation by the TLTP projects which were represented at our meetings and visits or which we looked at through the documentation file reviews.

• first, a sizeable proportion of projects (about half) had no evaluation at all, other than what might be described as routine project management;
• second, a significant sub-group of projects conducted some form of piloting of their products (as prototypes or ß-testing) which was aimed primarily at user reactions and acceptance of the courseware interface and curriculum content; in a minority of cases, piloting was restricted to the technical aspects;
• third, small groups of projects undertook more extensive usability testing in field trials and pilots, drawing on HCI expertise in some cases;
• finally, there were a very few cases where some attempt was made to look at genuine usage of the LT product in a real organisational and curriculum context.

One reason for the paucity of evaluations of this fourth kind is the time scale problem. Although TLTP was presented as an implementation programme using stable technologies, most of the non-institutional projects had a substantial software/courseware development component and the time scale of the programme partly precluded a summative assessment of learning outcomes and effects. IN at least one project, an evaluation of this kind is now underway with support from the Teaching and Learning Innovation Fund in the lead partner’s host HEI.

Focus of evaluation. Most commonly, evaluation was concerned with obtaining user feedback on the TLTP product as part of a process of continuous improvement so as to increase its acceptability and uptake. Formative evaluations of this kind were often quite extensive, involving numerous test sites in different institutions and substantial numbers of student and staff users.

 One project, for example, undertook the following evaluative activities: i) piloting of the courseware material with students in terms of acceptability, style and quality; ii) extensive use of the informal network of subject-based teaching staff to run and evaluate clips of the courseware; iii) an assessment questionnaire to departments asking about their activities, intentions and ratings of the materials.

 Far less common were evaluations which included the management of the projects and the process of implementation within their scope. Those few instances where there was a process evaluation of this kind tended to be institutional projects which lent themselves more obviously to an assessment of their strategies for institutional change and the development of IT/LT skills in staff and students alike. One product-based consortium was however exceptional in its emphasis on constructing a coherent account of the project and the design process methodology. Whilst we did not find many instances of reflexivity, i.e. project capability for reviewing the nature of their innovation, the development path it was taking, the need for any reorientation or redirection, we did come across examples of projects which opened project management up to scrutiny and sought to make changes deemed necessary. In one instance, for example, the evaluation revealed shortcomings in project communication with partners, resulting in a switch from e-mail to workshops as a way of involving everyone more fully.

Location of evaluators and their expertise. We found considerable diversity in the way in which evaluation was organised and the evaluation skills they were able to call on. A considerable number of projects embedded evaluation internally in the project, as a responsibility of all the partners. Where this was the only evaluation undertaken by the project, it tended to be restricted in its scope to fairly basic user feedback as part of pilot testing. Evaluations of this kind did not require sophisticated evaluation expertise, relying mostly on simple questionnaires.

 Another sizeable group of projects undertook a more substantial internal evaluation drawing on evaluators with considerable expertise in computer based teaching and learning. Even so, many of these evaluations were not more far-reaching in their scope than assessing user acceptance of the tools and courseware being developed.

 A small number of projects had an external as well as an internal evaluation. External evaluations were sometimes carried out by another department in the same university (such as, for example the Institute of Social and Applied Psychology, or the Centre for Computer Based Learning); other projects however approached an experienced external evaluator from another institution with no connection to the project at all. In at least one project, the external evaluation took the form of "authoritative peer review" or "connoisseurship".

 It would appear that external evaluators brought both evaluation expertise to the task as well as more specific pedagogic, HCI or curriculum expertise and our general impression is that their contributions were highly valued by projects.

 Mention was also made of the Funding Council’s support for greater involvement by the CTIs in evaluating software, but we did not come across instances were this had occurred.

Use of evaluation findings. Beyond using evaluation findings to improve the various outputs of TLTP projects, many projects seemed to be unaware of how they might potentially make effective use of evaluation findings. An exception here was a respondent who argued that "proper evaluation" can give leverage within an institution as well as help with the diffusion and uptake of software. On the more negative side, we came across one project where the project co-ordinator and team embers were unaware of evaluation activities which had been carried out although it later became clear that member sites had been involved in quite substantial evaluation activities. We were not able to ascertain whether this reflected poor management and control or changes in a project directorate.

Evaluation methods. The evaluation activities of projects included a diverse array of methods. The predominant method used was the questionnaire, but this was supplemented in a number of projects by one or more methods including observation, protocol analysis, interviews, user logs and limited use of focus groups. Some projects incorporate on-line data capture into their software, thereby providing a record of usage patterns and sometimes including user profiles as well.