FAQ's | Site Map | Contact Us | Home  


 

STRUCTURE, COST & PEDAGOGY
Evaluating Change
Pat Latham and Michael Raggett

CONTENTS:

This paper, which supports a presentation made to the EvNet '98 Conference in Montreal, is based on interim findings from ongoing evaluations of distance courses in Retail Management, produced and delivered by Ryerson Polytechnic University. These evaluations were concerned with the interface between education and technology, and with the question of whether educational needs were driving the technology rather than vice versa. In the course of the evaluations, it became apparent that the development (and delivery) of distance courses was forcing change in a number of areas, and ultimately these changes would impact on the whole institution.

 

Assumptions

Before beginning a discussion of the evaluations, it is important to make explicit some taken for granted assumptions that form the contextual background to the arguments and findings presented. While some of these assumptions can, and probably will, be contested by some observers, we believe that most of the following would receive widespread agreement.

Our main underlying assumptions are:

  • that knowledge can now be acquired from many more sources, more easily, than ever before
  • that new knowledge is being acquired faster than ever before
  • that knowledge can become obsolete much faster than ever before
  • that education is now a lifelong process for many people
  • that technology has freed educational delivery from time and space, both of which are scarce resources for most educational consumers
  • that higher (post-secondary) education is at the beginning of a huge paradigm shift
  • that distance education is a global and growing market
  • that the distance education market is increasingly competitive
  • that conventional universities now face the difficult challenge of having to operate in real AND virtual space in order to remain viable in the educational marketplace

Top  

 

The Aim of the Project

The purpose of the evaluation was ultimately to inform the creation of a "toolkit" for educational decision makers who want to develop "technology-enhanced" courses, and which would address the attitudinal and design issues relevant to the effectiveness of technology-enhanced learning. The toolkit would consist of qualitative, quantitative and comparative instruments, and strategies that an educational practitioner could use to select appropriate technologies with confidence.

Phase one of the project consisted of the design of a prototype model, grounded in previous research and development initiatives in the area of new learning technologies.

Phase two would consist of testing the model to improve its usefulness in delivering technology-enhanced courses.

The final phase will involve applying the model to corporate training to evaluate the use of new learning technologies in the context of work-based training.

At this moment the project is entering the final stage of Phase two.

Top  

 

The Methodology

The evaluation methods consisted of:

  • An initial literature search of relevant sources of information on developing and evaluating technology-enhanced courses
  • Observation of the development and delivery of 3 technology-enhanced courses presented by Ryerson Polytechnic University in 1997 and 1998
  • Personal monitoring of all course sessions, whether delivered synchronously or asynchronously
  • Questionnaires sent to students on these courses
  • Personal interviews conducted with the students on these courses and with members of the course development teams
  • Analysis of chat line logs - where applicable.

The evaluation was to be carried out by an organization external to the delivering university. At the outset, this evaluation was expected to be of a summative nature, so that the evaluators would retain objectivity. However, it soon became clear that much valuable information would be lost and the focus shifted very quickly to formative evaluation. This involved the evaluators from Sims Latham Associates more closely with the development of the courses, and provided feedback from their general experience of distance and conventional course design and delivery at this stage, rather than later feedback based solely on the evaluation of the designated courses. The evaluators were aware of the danger of too close association with course development teams, which is well documented by Dr. Robin Mason [Mason (1996) "Evaluating Technology-Based Learning" The Open University]. However, all critical comments were welcomed by course teams and SLA's objectivity was clearly respected in this case.

It should be noted here that, taken individually, none of these courses involved large enough student numbers to make any analysis statistically significant. However, the findings were able to provide indicators for further study, and there is already evidence that indicators derived from the findings of the first iteration are being confirmed in the second. Therefore, while the findings may not be significant, some of them will likely be "highly indicative".

Top  

 

The Courses and the Delivery Technology

The courses to be evaluated were part of Ryerson Polytechnic University's Certificate and Degree Program for the Canadian Retail Sector, and were deliberately innovative and experimental in the choice of delivery technologies.

CZRS 101 - Retail and Services Management This course used a very wide range of technologies, including: videoconferencing, audioconferencing, computer conferencing and print. It was delivered mostly in synchronous formats.

CZRS 108 - Issues and Innovations in Retailing 1 This course used a narrower range of technologies, including computer conferencing (e.g. internet searches, bulletin boards, chatlines and e.mail) and print It was also delivered mostly in synchronous form.

CZCM 101 - Introduction to Retail Management Communication This course will use the same range of technologies as CZRS 108 but will be delivered totally asynchronously. The use of this wide range of technologies, and the changes from synchronous to asynchronous delivery, will enable a systematic evaluation of the technology to take place. More importantly it will enable the evaluation to determine how these technology ranges and their degree of synchronicity affect the educational aims and outcomes of the courses. For example, it will help to test the belief held by many educators that face-to-face contact is necessary for a true development of critical thinking.

Top  

 

A Focus on Educational Considerations: A Model for Evaluation

The evaluators were well aware of the dangers of over-concentration on technology at the expense of essential educational considerations. The evaluation model was therefore based on five essential educational concerns that were identified by the evaluators as:

  • What are stakeholders* trying to teach/learn? = GOALS
  • What methods and technologies are being used to achieve those teaching/learning goals? = MEANS
  • What content material is being used to achieve those teaching/ learning goals? = CONTENT
  • How are other factors in the process of teaching/learning, impacting on the achievement of those teaching/learning goals? = CONTEXT
  • How well were those teaching/learning goals achieved? = OUTCOMES

* There were deemed to be several possible stakeholders in the educational process: course designers, teachers, learners, and sponsors, (who may be the state, the educational institution, employers or other benefactors)

A model was therefore constructed to be used as a conceptual tool for the evaluation. It was constructed on the basis of the key factors needing to be studied, to ensure that the evaluation had a sound theoretical base.

The evaluation model (see figure 1) therefore identified the investigative areas where questionnaires, interviews, observation etc would focus.


Top  

Figure 1.
  An Evaluation Model for Technology Enhanced Education
(Click to Access)

 

Means

For the purpose of this paper we will focus on the area of MEANS because this typifies the attempt to balance pedagogy and technology. However, this does not mean that the overall research is ignoring other aspects, such as Goals or Outcomes.

This evaluation of a series of technology-enhanced courses was fully conscious of the need to locate the means of achieving educational goals squarely in its educational context. To this end, SLA looked at the pedagogy used to deliver conventional courses and then considered how this might be affected by the use of new technology.

Models of Pedagogy

In order to study the relationship between educational aims, pedagogy and the use of technology, we devised a simple typology of pedagogical methods:

  • Didactics

    In this method, we suggest, a perceived set of facts, or theories are imparted in a one-way interaction from teacher to student(s). The active participant is the teacher and the learner passively receives teacher output. An example is the university lecture.

    This method allows one teacher to impart information to a number of students. In the traditional classroom, this number is limited by the size of the room. Once technology is used, this can be extended to any number, over any distance, and at any time.

    While this can be a very cost-effective method for imparting knowledge, in terms of staff costs, it may not be effective for teaching skills and may have little effect in changing attitudes. While ideal situations in the traditional classroom might be thought to provide for satisfying interpersonal relations, in practice the face-to face classroom situation is more usually that of one professor to too many students to allow any meaningful interpersonal interaction to take place. This is important to remember in comparing conventional and virtual classrooms.

  • Discourse

    In this method, we suggest, facts or theories are imparted through a two way interaction between teacher and student(s). An example is a seminar. The teacher and students are both now active participants in the educational process. This is relatively easy to translate into more distanced applications, through audioconferencing or on-line chat groups, which can retain the active participation of students.

    This can be cost-effective in terms of teaching staff costs but may be more costly in terms of technical support staff and, again, will require more time for planning, which must also be costed.

    Discourse can be private (usually between teacher and individual student) such as discussing grades or educational problems, or public, such as in the form of seminars or chat lines.

  • Discovery

    In this method, we suggest, experiences, facts or theories are explored through an interaction between the student(s) and a variety of sources. In this scenario students have taken active control of the educational process. In its purest form, it could obviate the need for an educational institution at all, unless the student requires some form of certification of their endeavours.

    The problem for the student is that the quality of educational experience that can be acquired for “free” may well not be satisfactory for the student’s needs. For example, while the Internet had free knowledge as an original ideal, this is being challenged by the fact that it costs money to acquire and organize knowledge, and these costs may need to be recouped. In addition, unstructured as opposed to structured discovery may be a very inefficient way to acquire knowledge and skills.

It seems likely that, while the need for the certification of educational effort and the organization of knowledge acquisition still exists, there will continue to be a role for educational institutions, where teachers take on a facilitative role and aid students in the organization of their acquisition of knowledge, skills and attitudes.

It should be noted that, as the cost of education becomes more manifest to students, and as technology allows for more effective distanced learning, commercial organizations have also entered the learning facilitation market via the Internet. For more details of this see "The Stone Age University"

While the focus is shifting from teaching to learning, and the role of teacher is changing from teaching to facilitating, the popularity of didactics is supposedly declining. Yet the problems of costs and time may be working against this shift in favoured methods. Unstructured discovery can be very costly for the student in terms of time, and time also costs money. On the other hand, well structured discovery can be costly in terms of teaching, both in time and financial terms.

This is an important consideration because an accompanying paradigm shift is an increasing emphasis on lifelong education (or learning), where students are more likely to be studying part-time and may well be holding down a full-time job as well. While the joys of discovery may be legion, especially for younger students, those already in the workforce may well prefer the short cut of “talented didactics” to acquire new bodies of knowledge. A mixture of pedagogies may still be the best option for achieving successful outcomes and customer satisfaction.

     As Tom Creed of St. John’s University describes it:

"I view my role in the course, then, as setting up the environment so that my students will most likely learn the material as well as possible, but I do spend some time playing expert and transmitting information ."

The three pedagogic methods outlined above are, of course, ideal-typical concepts that in real life tend to merge into one another, or be used in conjunction with one another in order to achieve specific educational aims. All three methods can be used effectively in technology-enhanced education. Course teams designing technology-enhanced delivery systems need to decide which (or which combination) will be best for achieving the specified educational goals, and then how technology can be used to maximise their effectiveness. For a brief discussion of this see "Technology and the 3Ds"

Top  

 

The Use of Technology

Once the basic educational aims and methods of the course have been decided upon, consideration can then be given to the use of technology, both to enhance the educational process and product and also to achieve other aims. These other aims may be to reduce costs by reaching more students through the use of virtual classrooms or by freeing the course from time constraints. In other words, the educational needs should dictate the selection of the technology not the reverse:

"What do we want to accomplish in our courses, and can technology advance our teaching goals?" [Creed (1996) ibid]

In this regard, it is clear that making decisions about technological enhancement of courses needs to be a collaborative process, involving both educators (including teachers and administrators) and technologists, with effective communication between the two. A common problem here is that both sets of participants have their own set of technical terms, often viewed as "jargon" by the other. Education, for example is a multi-disciplinary social science that is riddled with esoteric terminology. Similarly, technology has its own special language. So teachers and technologists will not necessarily be on the same wavelength. For example, it may be a rare academic who can cope with such technological acronyms as MOOS and MUDS!

With this in mind, it is best to discuss matching educational aims with the use of specific technologies in pragmatic terms, couched in everyday language where possible. This approach has been adopted in the following discussion

We would argue that an important distinction is between the intrinsic and extrinsic features of technology. Therefore choosing a suitable technology will depend on consideration of:

     a) the intrinsic nature of that technology and

     b) the extrinsic factors present in the context in which it will be used.

Intrinsic Features of Technology

In order to determine suitable technology, we identified some basic questions that need to be asked:

  • Audience and Direction

    Do the features of the technology allow communication with the size and type of audience you want to reach and in the way you need to interact with that audience?

    The most usual variations of type of audience and interaction will be:

    One to a group or groups

    where a teacher needs to communicate information to a limited number of students but does not need a reply, and so the interaction is one way

    One to unlimited

    where the teacher needs to communicate information to a very large number of students or all groups of students but does not need a reply

    One and One

    where a teacher or student needs to interact with one other person to exchange information

    One and group(s)

    where one person (student or teacher) needs to obtain or exchange information with a group or limited number of groups

    One and unlimited

    where one person (student or teacher) needs to obtain or exchange information with an unlimited number of people

    group and group

    where a group of people needs to obtain or exchange information within the group or between groups of people

  • Time and Space Sharing

    The question here is whether the technology allows for the sharing of space and time in the way that is required.

    The usual combinations are:

    Sharing real space in synchronous time

    This is the conventional classroom situation, where teacher and students are both present in the same physical space at the same time.

    Sharing virtual space in synchronous time
    This is where teacher and students are separated in physical space but share a virtual space (e.g. through videoconferencing, audioconferencing or chat lines) in synchronous time.

    Sharing virtual space in non-synchronous time
    This is where the student learning has become space and time- independent, such as the use of bulletin boards for discussion. These can have some specific advantages over interacting in real space:

    "class discussions over computer networks leave a record, unlike verbal discussions (every student knows what it's like trying to decipher his or her class noted after a particularly heated debate). If you want to refute a previous point or refer to an early lecture in an essay, they are immediately accessible." [Nayman (1997) "Getting Connected with Connect Ed." Educom Review Vol 32 No 3]

    Sharing real space in non-synchronous time
    While this would seem to be rare there are documented cases, such as that of students who had access to a shared social space and, in spite of the ability to use electronic chat lines, would leave each other written messages on a physical bulletin board! [Knowledge Connection (1996) "Innovations in Course Delivery for a Distributed University: The Evaluation of the Interactive Learning Connection-University Space Network Pilot Project"]

    Sharing neither Time nor Space
    The most time and space independent of all media is one of the oldest - print! Although the use of laptop computers is beginning to change that, you still don't need a battery to read a book. In addition reading large amounts of text from a computer screen remains non-ergonomic for the human eye.

  • The means of communication & the need for recordkeeping
    The educational aims and the preferred type of pedagogy will suggest the need for certain means of communication, and whether or not a record of the communication is required:

    Visual Communication
    Do participants need to have visual contact in one or both directions? Does there need to be a record of the interaction? Will the educational institution provide the record or will it be up to the student. This has cost implications for both parties. If the visual need is for illustration purposes rather than an aid to personal interaction, the need for real time transmission may be obviated.

    Aural Communication
    Do participants need to hear each other in one or both directions? If so, does there need to be a record, and who should be responsible for providing it? Depending on the nature of the educational need, once again real time transmission may not be necessary.

    Written Communication
    Can the interaction take place in writing? Is this in fact better for the educational purpose? Is it better via physical print or electronic format? There are advantages and disadvantages to both options. Physical print is costly to produce and costs are difficult to justify or recoup unless student numbers are very large, and it is also next to impossible keep up to date on a regular basis. Physical print certainly provides an automatic record, but it is now also possible to print from electronic delivery formats.

    There are many choices of technologies or combinations of technologies that will provide these various means of communicating to students. They range from video-, audio- or computer-conferencing, through the use of television video-tapes, radio and audio-tapes, to good old-fashioned books. All have implications in terms of cost and effective usability for both students and staff.

  • Linearity
    The desirability for non-linear performance is an important consideration in choosing suitable technology.

    This feature relates very much to accommodating different student learning styles and tasks. While print can be considered a linear type of activity (i.e. you start at the beginning and go through to the end), it is still possible to browse or dip into a book to find specific piece(s) of information.

    This search capability is made even easier if the print is in electronic form. However, there are sometimes limitations in the design of electronic formats for presenting information, and students can sometimes be forced to proceed through an electronic presentation without the ability to stop and review parts without going right back to the beginning.

  • Interaction/Routing
    The degree of interactivity is another important issue. While some technologies seem to provide interactivity (i.e. students can receive a response to an input, this is sometimes merely routing rather than true interaction. For example, a student is asked to respond with a yes/no answer. If the answer is yes, they are routed through one section of the structure, if the answer is no, they are routed through a different structure.

    True interactivity is capable of dealing with different types and large numbers of responses, and not simply preset procedures for dealing with a very limited set of responses. But it will not always be necessary for achieving a particular course aim.

  • Costing
    The cost of using a specific technology is not so simple to determine as it may first appear. There are a whole variety of factors to take into consideration, and in addition there are two main complications:
    1. The question of hidden or" sunk" costs as opposed to overt costs; and
    2. The problems of cost distribution, not only between the students and the educational institution, but also over the shelf life of the educational course.

For example, planning and support time seems to be a high cost associated with the use of technology, but this may simply be that planning costs are for the first time becoming manifest rather than hidden within the teaching role.

In addition, though technology-rich courses may seem to have high planning costs, the ability of technology to update the course content at lower costs allows the shelf life of a course to be extended as long as the demand exists, and this can reduce costs considerably over time. Some courses can have very long shelf life (mathematics, for example) while others by their very nature require more frequent updating. One such course was one of the two already evaluated: "Issues and Innovations in Retail." This course would obviously benefit from being available in an easily updatable format.

The costs of conventional courses have often become embedded in long existing structures and processes that are no longer open to scrutiny, but this is not true for new technology-enhanced distanced courses, where new equipment, staffing and operational costs are all clearly obvious. Performing a cost-benefit analysis of conventional and distanced courses is therefore not an easy task.

Another problem with assessing cost will relate to the number of students taking a particular course. As technology frees educators and students from space and time, many more students may be able to access a particular course with the same number of teaching staff. This usually means much lower marginal costs. In that situation it is difficult to assess costs until the student enrolment numbers for a course are finally known, and this can make a critical difference to the financial viability of a course.

The whole question of educational costs is, in fact, such a vast subject in itself that within the confines of this paper it is not possible to examine it in the depth it needs and deserves. Only some of the issues associated with costs have been raised here and we are currently investigating cost issues, and assessing methodologies that will allow educational institutions to make more informed decisions on costing their technology enhanced and distanced courses.

Comparing technology-rich courses with more conventional ones, not just in the area of cost but also in efficacy, may be like comparing apples and oranges if the conventional course is no longer viable for nontechnical reasons, such as the lack of a market for space and time-dependent courses that have become dated. Unfortunately, such content-obsolete courses are not as rare as one might expect. While universities have a monopoly on accreditation, the pressure to remove them from the list of required courses is not always sufficient. Technology, with its constant and increasing rate of change, can leave some faculty members unable to keep pace with new developments and, therefore over-reliant on old content or skills. Indeed we know of at least one faculty that has negotiated with its university that no existing faculty member shall be required to use new technology. [Young (1997) "Canadian University Promises It Won't Require Professors to Use Technology" The Chronicle of Higher Education October 3 ] In the short term this type of defensive measure may secure jobs but, with increasing competition from off-shore universities offering accredited distance courses, it will be self-defeating as students transfer to more techno-savvy educational providers.

The choice of a particular technology or set of technologies is a time-consuming task that requires coordination and communication between teachers and technologists, as well as educational administrators. It is further complicated by the addition of extrinsic factors that also affect the successful implementation of educational courses.

Extrinsic Factors in the Use of Technology

As technology frees the teaching learning process from space and time, so the context in which it is used becomes ever more variable, and less and less under the control of the educational institution. This means that very careful consideration must be given to the likely context in which the chosen technology will operate.

This is further complicated by the fact that the context is a moving target, continually affected by new advances in technology, and other social and economic changes. Some considerations that have emerged in this research, which we believe important are outlined below.

  • Level of availability
    How available is the technological hardware, software, support and user know-how among the population from which the students are likely to be drawn? If any of these are not generally available this can cause problems for a course. For example, there can be a problem in accessing the Internet in the more rural areas, where even a national provider may not be available, unless expensive long distance charges are incurred.

  • Level of Compatibility
    While technology has made great strides in the area of compatibility, there are still problems in this regard, such as the ability to read attachments. Different versions of software can also pose problems. While possibilities for more sophisticated activities may be available from newer software, if some students have earlier versions this may have to be foregone.

  • User familiarity
    While an increasing number of people are using computers at work, less have them at home and even less use them to access the Internet. Company employees do not generally have access to the Internet, although some may have access to the company Intranet. The likelihood of many students from the retail trade (or indeed from other industry sectors other than IT) having a good working knowledge of the Internet is not very high. If Internet use is to become an essential part of a course, thought and planning will need to go into orienting students to its use BEFORE this skill is required to undertake research or assignments. In addition, the use of a specific technology for EDUCATIONAL purposes may require a specified set of protocols/practices that differ from more casual use. Even experienced technology users may require time for this adjustment.

  • Place of operation
    If a course is to take place in virtual space, the conditions of this space become an important factor in the educational process. This may be a remote site where some students come together, or it might be an individual student's home or workplace. Either way, the educational institution will need to take account of conditions at these sites in its course planning. What special equipment is needed there? Are staff available to operate it? Are they fully trained in its use? What other facilities are needed? What allowances (if any) will they make for problems that arise from the location? For example, a student using a computer at work may not be able to access the Internet because of a company firewall.

  • Timing
    Many students, especially part-time ones who are active in the workforce, will be attracted to courses that allow for asynchronous response (i.e. free(r) of time constraints). But this is also a factor that must be taken into account when planning the course, especially if teamwork is a course requirement. Team coursework or assignments that require synchronous communication between team members may be difficult if some students are using computers at work and are available only during working hours, while those using computers at home are only available out of working hours.

  • Availability of technical support
    Given the generalized lack of technical know-how amongst the likely student population, the provision of technical support is almost certainly an essential part of any technology-based course. How that support will have to be costed and provided is another important consideration for a course planning team. For example, providing technical support in office hours only will not meet student need in a distanced and continuous educational setting, where most students will be in the workforce and have to study in the shoulder hours. There will be a need to consider what business calls a "24/7" service.

  • Cost
    While it is possible to download costs from the educational institution to the student, this may be self-defeating when it comes to marketing the course. If the student has to acquire expensive new hardware and software, as well as pay the course fees, this will serve to reduce the demand for this type of course. While educational niche markets are pursued by commercial enterprises, the traditional university has traditionally had to cater for a more general educational consumer who may be much more cost-sensitive.

  • Time
    Most students for this type of course are already in the workplace, with severe constraints on the time available for continuing education. This means that the technology must make their time spent on the course more effective rather than less. Having to struggle with unfamiliar technology that adds to their sense of stress can be seriously demotivating. Either the technology chosen must be familiar or technology orientation must be part of the course.

In general the more widely available and well tested the technology the better. Also restricting the course to the use of one or two technologies may be preferable if planning time is at a premium.

Technology-based courses will certainly take up a lot more planning time than conventional ones. They also require good communications between educators and technologists. The lead time to convert from a conventional to a technology-based course, or to create a technology-based course from scratch, will depend on the resources available, but will not in any event be short.

Careful planning for technology-based courses is essential, but the resources put into planning can repay the institution in the longer term. Customer satisfaction will be higher and this will increase the demand, which can in turn reduce net costs. Added to this, well planned courses can have a longer life, again a cost reducing factor.

In any event, the institution may have no choice but to plan for distanced courses. This is where the growth in educational demand is to be found. In the USA, for example, there can be seen to be 6 distinct groups of students making up the higher education population. [Twigg (1997) "Notes for NLII-ITP Symposium on Creating and Delivering Collegiate Learning Materials"]

Group Ages Enrolment Location Student type Workforce Numbers
1 17-24 full-time campus trad/
1st Degree masters/PhD
not 3.9 million
2 22-34 full-time campus trad/
1st Degree masters/PhD
not 0.65 million
3 17-24 part-time campus semi-trad
1st Degree
non-career entry level 2.9 million
4 22-34 part-time campus semi-trad masters/PhD career (on-campus) 0.48 million
5 25> full/part on/off non-trad 1st Degree career workforce 5.3 million
6 25> full-part on/off non-trad masters/PhD career workforce 0.88 million

Totals: Full-time traditional
Part-time semi-traditional
Full/part-time non-traditional
  4.55 million
3.38 million
6.18 million

This demonstrates the demand by non-traditional students for post-secondary education in the USA. It seems reasonable to assume that the situation is similar in most of the developed world because the pressures driving the demand are the same. Add to this the fact that ,while the traditional student numbers are not rising in most of the developed world (with the possible exception of California, Florida and Texas [Daniel (1996) Mega-Universities and Knowledge Media Kogan Page]), the numbers of non-traditional students is likely to rise significantly, due to advances in technology that require frequent and continued re-training and re-qualification.

In addition, in the developing world the demand for education is still rising and distance education is likely to provide the answer to this demand, which must satisfy large numbers at the lowest possible cost.

 

Top  

Some Issues and Implications from Interim Findings

Some Emerging Implications for pedagogy

Faculty and familiarization with new technology
The students on both evaluated courses were highly motivated by the chance to discover and use the technology itself - the medium was as important to them as the message. This should not be surprising. Many workplaces now depend on technology, and experience of technology is an important job skill to have on a resume for many people. This will continue to be important in the educational workplace because technological change is a constant.

We would suggest that faculty members who attempt to cut themselves off from the use of new technology (even if this is officially sanctioned) may nevertheless find themselves under increasing pressure to come to grips with it in order to remain relevant to their students, who will likely be drawn more and more from non-traditional educational populations.

Conversely, faculty who enjoy and are skilled at imparting new knowledge with the new media will have the chance to become academic "stars", rather like TV news anchors, with a whole production department supporting them behind the scenes. This may mean that teaching skills will at last become as valued and important as research skills, and as equally scarce a resource.

Synchronicity, faculty workload and optimal student numbers
On the second course evaluated, the range of technology was narrower than the first but there was still a considerable portion of the course conducted synchronously. The professor was also more distanced from her students, who were also much larger in number than on the previous course. As we monitored the course, it became clear that the number of students was causing problems in two areas.

First, the use of synchronous chat lines was proving problematic. There were too many students for a meaningful and sufficiently ordered discussion (in spite of the fact that students were engaged with the subject matter and were obviously participating in the discussions with interest). An analysis of the chat logs showed that questions and answers were totally out of sequence and group chats were interfering with class discussion. This situation improved somewhat as students learned to use group chat lines instead of the main line for team communications, and as the professor provided a clear protocol for class discussion. However, the numbers still remained a problem and in interviews with the students, we found that the sheer numbers trying to use the technology was interfering with their ability to conduct a meaningful discussion.

Second the amount of tutor workload was considerable. A whole morning was devoted to synchronous delivery of the course; much extra time was spent outside the virtual classroom in responding to student concerns (and these were not always of a strictly educational focus, often being problems related to using the technology). In the remaining time available the marking of assignments was becoming overwhelming, and would certainly become unsustainable without the delegation of at least some of this task to another academic. The problem of assessment was more a function of the number of students than the sharing of real time. So distance courses, with large numbers of students, will obviously require fewer teachers but many more assessors.

Fragmentation of academic roles in distance education delivery
Synchronous distance courses are time greedy and will be limited in terms of the number of students who can be effectively accommodated in educational terms. From student feedback, and from our observations in monitoring the course, it quickly became clear that 39 students on a synchronous chat line was too much for truly effective learning.

Asynchronously delivered courses, if successfully promoted, will be associated with much larger student numbers. The need to provide extra assistance to a "lead" professor, in terms of student support and assessment, will grow as courses become more truly distanced and the numbers of enrolled students increase. This will be another pressure for role "fragmentation", and new specializations that relate to activities as much as to subject specialisms. A gifted teacher will be best used in "front of the camera" and others more camera shy could develop another expertice, such as the assessment of distance students or research in educational technology. This, along with willingness to use new technology, will serve to promote the "star" system.

Team membership and educational concerns
It also emerged very early in the evaluation that an academic will quickly become outnumbered by technologists and even administrators on course teams. This seems to be a normal composition of this type of team. For example, Professor Robert Kowalski, Head of the Department of Computing at Imperial College delivered a lecture in September 1997 on computational logic and legal reasoning. This was broadcast live on the Internet as an interactive webcast. It was delivered to a live audience at a conference in Dresden. (For those interested it is still available on the web - kmi.open.ac.uk/compulog/story.) For this broadcast there was one academic and five technologists. While this was a delivery course team, a design course team has much the same composition. This raises the concern that educational/intellectual issues will begin to take a back seat to technological issues. Part of the problem is that academic staff who do not know what the technology can do are at a distinct disadvantage in this situation. There is much to be said for devoting some resources to staff training in the new technologies, enough that they have a reasonable knowledge of the options open to them in achieving their educational aims at a distance or enhanced by technology.

This seems a more positive approach than refusing to use the technology, if only because technology has much to offer in achieving educational aims.

Academic freedom and the use of technology
Traditional academic freedom becomes increasingly problematic in the distanced learning environment. For example, if a number of academics, none of whom necessarily had a hand in designing the course, are marking sets of students on the same course, there is a need to standardize the level of marking to ensure fairness to the students. This means that the assessment procedure must be monitored. This will be an anathema to many traditional academics. For those of us who have taught at distance universities and are used to this procedure, there are benefits to be obtained from the process. Feedback from other academics can be very welcome in a distance environment where this is unusual compared to a conventional university.

Personal Interaction with students
This is a difficult and unresolved issue. There is evidence from our evaluations that the use of technology does have an effect on how an academic interacts with students. For example, we would categorize the professor on the first two courses as a gifted teacher who enjoyed the personal interaction with students and was good at communicating with them. Nevertheless, it became a challenge to retain the quality of the personal interaction over increasingly distanced delivery methods. For example, less expansive gestures had to be adopted to avoid jerkiness in camera movements in the videoconferencing sessions. The professor herself was concerned that this type of problem was affecting her ability to engage students in the same way as she could in a conventional classroom. This in spite of the fact that she was enthusiastic about trying the new technology and soon learned to have plans A, B and C ready for occasions when the technology was causing a problem.

In order to address her concerns we probed students in the interviews about their reactions. It was interesting to discover that, while students on the first course were generally satisfied that they had had good personal interaction with their professor, they could not generally imagine being as satisfied with a more distanced approach. Students on the second course, which used the same professor but a more impersonal, distanced delivery, were nevertheless generally satisfied with their interaction with their professor, and yet they could not imagine being satisfied with an even more impersonal, distanced delivery.

This seems to suggest that students adjusted to the type of delivery mode with less of a feeling of remoteness than one would have expected. One clue to the reason for this came from observations and several responses by students on the first course. In this course the professor operated from two sites alternately. This meant that on every other week the students at one site were distanced from her by videoconference or audioconference, while the other students interacted with her face-to-face. We noticed that, when she was at a site in person, students sat up to the tables and were generally less relaxed and more engaged than when she was not present in person. Several students told us that they felt "deprived" when she was at the other site. Yet in interviews all students said they were very satisfied with their interaction with their professor. One student's comment suggested an answer to this apparent contradiction. "I don't think I would have felt deprived if I had known the other group was also at a distance."

It will be interesting to discover just how satisfied students on the third course will be. This course will have only web-based interaction between students and professor. Sadly we will not be able to compare delivery methods using the same professor. It may be that videotapes, or video on web (currently still problematic because of bandwidth or modem problems), will be needed to establish sufficient "rapport" between students and teacher(s). Or it may turn out to be the contact between students, rather than between students and faculty, that is important to successful educational outcomes. We also suspect that familiarity with the technology will also be an important factor in this issue.

Implications for technology choices

Student anxiety about using technology
Using a reflective technique we found that even students who were outwardly enthusiastic towards new technology could still have underlying anxieties about its use. This means that educational decision makers would be wise to consider the familiarity of the proposed technology to the population they are targetting.

This also has implications for the range of technologies used on a course. If these are likely to be largely unfamiliar to the students, it is better to keep to a fairly narrow range so that students do not have too many new technologies to cope with at once. It may also be necessary to provide some form of technology orientation or prerequisite, for those who need it, before they are required to use the technology to work on or submit assignments. This request came up again and again in the interviews with both groups of students.

In addition, it became clear from monitoring the course, analysing the chat logs and interviewing the students, that the educational use of this technology needs to have clearly specified protocols or "netiquette" if it is to be educationally effective. Even if students are experienced in the technology, they may still require some orientation to its use in an educational setting. This will require the input of educationalists and not just technologists in the preparation of any such courses or sessions.

Effective learning will clearly relate to the ease with which students can use the delivery technology. The most expensive technologies, which are often those which come closest to approximating the qualities of face-to-face contact, and which we found to be generally popular with students, may not be necessary or the most effective for successful learning. When asked what they considered the important learning points on the course, and which technology they had used to learn it, print and videotapes were more frequently mentioned as effective than the more popular technologies. This suggests that more impersonal and distanced technology can be associated with effective learning.

Implications for Structures

Course teams and their composition
One question that arose from findings about the need for continued student technological support during the course was: for how long should course teams be constituted? Developing a course may require one set of activities and people, delivering a course a different set of people, maintaining a course another set of people/activities, while supporting a course (i.e. its students) quite another, and assessing course outcomes yet another.

Many of these functions have been traditionally contained in the one role of "professor". With technology-enhanced courses, especially if they involve distanced education, the role of the professor is likely to change from a multi-role, subject specialist to a single role task/subject specialist. We have outlined below some possible course team structures for the various stages of the distance educational process. These partly depend on the technologies used for delivery and assessment. If, for example, the course is totally web-based, broadcast specialists would not be needed. If assessment is totally computer-based, the number of academics on a delivery team will be reduced. However, representatives from four main areas will generally be required on all teams:

academics
educational technologists
technologists
administrators.

In addition, other people with required specializations, such as marketing or print buying, may need to be present from time to time.

 

Course Design/Creation Team
Role Activity
Academic Content specialist(s) Research and organization of content
Educational technologist Learning structure of course/assessment procedures
Technologist(s)/Media Specialists Development of delivery mechanisms
Production specialist(s) Plan production schedules
Financial Planner Cost projections/budgeting
Administrator Plan administration of proposed delivery
Market Researcher Provide feedback on market conditions
Coordinator Process communications and requirements of team
 
Course Production/Delivery Team
Role Activity
Academic Delivery specialist Interaction with students (online/broadcast etc)
Academic Assessment specialist(s) Assessment procedures
Educational Technologist Monitor assessment/standards
Production specialist(s) Print/broadcast/online production
Administrator(s) Registration/financial organization/processing
Coordinator Communications/reports/records
 
Course Maintenance/Support Team
Role Activity
Academic(s) Revisions to content/educational student support
Technologist(s) Technology troubleshooting
Educational technologist Assessment problems/monitor assessment process
Administrator(s) Administration of student support services
  Administration of course maintenance
Coordinator Team communications and reports

In addition to the above one might also need a Summer School Team and an Exam Team. It is easy to see how a full open univesity course team can reach 40 in number!. While not every role will necessarily require a separate person (for example, an academic may also be able to act as an educational technologist), it will be important to have enough people to cover the range of activities. This also relates to the costing of courses.

Implications for Institutional Structures
When people come together in teams of the above type, they almost always cut across existing departmental boundaries. If attempts are made to confine them within existing structures this will soon impact on the institution's ability to effect economies of scale. As the financial case for distance courses rests largely on the low marginal costs (especially as the institution has to operate in real and virtual space simultaneously), economies of scale become essential.

While distance or online courses for the non-traditional educational consumer may well originate in a university's school or department of continuing education, they will require technical and other support that could also cover such courses elsewhere in the university. Sooner or later, pressure will build to change existing university structures to accommodate this possibility and take full advantage of expertise gained in the creation of such courses for more general benefit.

The balance of academic to non-academic staff
As technology-enhanced courses are adopted, the need for academic staff will diminish in relation to the need for technical staff, which will increase. As with all technological change, jobs will disappear and other jobs will take their place but the personnel will not necessarily be able to make the transfer from one type of job to another.

While some new academic jobs will become important, such as that of educational technologist (a person who is largely concerned with the way teaching and learning is made effective), they will not necessarily relate to subject specialisms. "Star" specialists will become even more important and scarce than they are at present but the specialism may now also relate to the ability to TEACH as much as to create knowledge. At the same time technology will enable these stars to reach many more students than previously. The problem for the average academic will be more serious. They will be less needed to teach but will still be required for assessment and also for student support services. These activities will not be location specific and many academics may find themselves teleworking.

A second and critical area of concern is the power relationship within a course team. If the course is to retain academic credibility, it is essential to ensure that the academic(s) has the POWER to ensure academic/content issues are not diluted by so-called technological imperatives.

Implications for Costs

Technology support services and costs
If students require help and assistance with the technology (and we have found that even those who say they are familiar with the technology still need this type of assistance from time to time), this has cost implications. In addition, the timing of that need is not likely to be during office hours if the course has substantial asynchronous components. This impacts on staffing arrangements as well as on costs.

Another important factor here is that the problems many students experienced were not necessarily under the control of the institution or the student. For example, several students on both iterations complained of having problems in submitting, or having enough time left for, or communicating with other team members for assignments, because their ISP was "bunching" their e.mail and it was not reaching them until much later than the sender expected. Having "expensive" educators or technologists dealing with this type of problem is not cost-effective and lower cost administrative staff will need to be involved.

Costing Courses
There is no doubt that traditional management accounting practices will not provide the quality of information that is needed to make informed decisions about distance education courses. Three major criticisms of traditional accounting cited by Greville Rumble [Rumble (1997) The Costs and Economics of Open and Distance Learning  Kogan Page] are:

  • the failure to deal adequately with overhead costs (as opposed to direct costs) which often form the major cost areas;
  • the over focussing on the costs of direct labour (ignoring the importance of other factors) and thus causing an over focus on this area for attempted savings; and
  • the fact that overheads are often driven by factors that do not relate to volume of production and services delivered, and so an inadequate attention to the details of overheads results in failure to understand the real workings of the cost structure.

Rumble's suggestion for a better cost accounting system is to use "activity-based costing". This is based on the logic of "considering the design of the system and identifying everything that might result in a cost being incurred." [Rumble (1997) ibid] As a course team structure is built on the activities of the members, this makes a lot of sense.

The problem for conventional universities is making the necessary comparisons between the costs of conventional and virtual courses. The costs of conventional courses typically hide sunk costs that are made manifest in the costing of virtual courses and this results in inaccurate cost comparisons. New costings for conventional courses, which are also based on activity-based cost accounting will be needed for a true and meaningful comparison.

The popularity of technology, development costs & financial resources
The demand for technology-enhanced courses (which our findings to date suggest is a real factor) stems in large part from interest in the technology itself. This may encourage the production of more technology-enhanced courses, which can have lower marginal costs (depending on the technology used for delivery).

With a demand for technology-enhanced courses, academic institutions will find a heavy drain on their resources as they attempt to satisfy this new market as well as their traditional classroom based clients. Technology-enhanced courses are generally acknowledged to be more expensive to develop than conventional ones. This will pressure universities either to divert resources away from conventional courses, which seems unlikely, or to seek other funding sources, such as commercial partnerships.

The need to operate in real and virtual space is a serious problem for universities that cannot be understated. The merger and acquisition fever that has gripped, and continues to grip, the commercial world is a real possibility for universities. There are several models that can be suggested for the survival of the university and all of them involve radical changes in structure. This whole issue deserves much more space than can possibly be devoted to it in this paper. We are currently researching this area, and those who are interested will be able to access our discussion papers on the web.

Top  

Conclusions

As universities move to using new communication technologies to service campus students, and some more distanced students, they will inexorably be drawn into new structures and financial management systems.

Faculty will find they are not only required to offer some awareness of new technology, but will inevitably become involved in new, more fractured roles in technology mediated, non-location specific course design.

The new "dual" universities, operating in real and virtual space, serving local and distant students, will be subject to great stresses, which will probably require collaboration and cooperation between them to enable a successful shift to this new mode of operation.

As Sir John Daniel, Vice-Chancellor of the Open University in U. K. has suggested:

"The key challenge for campus universities is to change from a teacher - centred model of operation to an approach that emphasizes learning productivity - and to carry students with them. "

 

Copyright

Users of this material are permitted to make a copy for their own research/study purposes. However, no part of it may be stored in a retrieval system or transmitted in any form or by any means, without permission in writing from the copyright holders.

© 2001 Sims Latham Associates

Top  

 
- © 2001 Sims Latham Associates. All rights reserved.
Contact Us