|
STRUCTURE, COST & PEDAGOGY
Evaluating Change
Pat Latham and Michael Raggett
CONTENTS:
The Methodology
The Courses & Delivery Technology
Focus on Educational Considerations
Provisional Evaluation Model
The Use of Technology
Issues and Implications
Conclusions
This paper, which supports a presentation made to the EvNet '98 Conference in Montreal, is based on interim findings from ongoing evaluations of distance courses in Retail Management, produced and delivered by Ryerson Polytechnic University. These evaluations were concerned with the interface between education and technology, and with the question of whether educational needs were driving the technology rather than vice versa.
In the course of the evaluations, it became apparent that the development (and delivery) of distance courses was forcing change in a number of areas, and ultimately these changes would impact on the whole institution.
Assumptions
Before beginning a discussion of the evaluations, it is important to make explicit some taken for granted assumptions that form the contextual background to the arguments and findings presented. While some of these assumptions can, and probably will, be contested by some observers, we believe that most of the following would receive widespread agreement.
Our main underlying assumptions are:
- that knowledge can now be acquired from many more sources, more easily, than ever before
- that new knowledge is being acquired faster than ever before
- that knowledge can become obsolete much faster than ever before
- that education is now a lifelong process for many people
- that technology has freed educational delivery from time and space, both of which are scarce resources for most educational consumers
- that higher (post-secondary) education is at the beginning of a huge paradigm shift
- that distance education is a global and growing market
- that the distance education market is increasingly competitive
- that conventional universities now face the difficult challenge of having to operate in real AND virtual space in order to remain viable in the educational marketplace
Top
The Aim of the Project
The purpose of the evaluation was ultimately to inform the creation of a "toolkit" for educational decision makers who want to develop "technology-enhanced" courses, and which would address the attitudinal and design issues relevant to the effectiveness of technology-enhanced learning. The toolkit would consist of qualitative, quantitative and comparative instruments, and strategies that an educational practitioner could use to select appropriate technologies with confidence.
Phase one of the project consisted of the design of a prototype model, grounded in previous research and development initiatives in the area of new learning technologies.
Phase two would consist of testing the model to improve its usefulness in delivering technology-enhanced courses.
The final phase will involve applying the model to corporate training to evaluate the use of new learning technologies in the context of work-based training.
At this moment the project is entering the final stage of Phase two.
Top
The Methodology
The evaluation methods consisted of:
- An initial literature search of relevant sources of information on developing and evaluating technology-enhanced courses
- Observation of the development and delivery of 3 technology-enhanced courses presented by Ryerson Polytechnic University in 1997 and 1998
- Personal monitoring of all course sessions, whether delivered synchronously or asynchronously
- Questionnaires sent to students on these courses
- Personal interviews conducted with the students on these courses and with members of the course development teams
- Analysis of chat line logs - where applicable.
The evaluation was to be carried out by an organization external to the delivering university. At the outset, this evaluation was expected to be of a summative nature, so that the evaluators would retain objectivity. However, it soon became clear that much valuable information would be lost and the focus shifted very quickly to formative evaluation. This involved the evaluators from Sims Latham Associates more closely with the development of the courses, and provided feedback from their general experience of distance and conventional course design and delivery at this stage, rather than later feedback based solely on the evaluation of the designated courses. The evaluators were aware of the danger of too close association with course development teams, which is well documented by Dr. Robin Mason [Mason (1996) "Evaluating Technology-Based Learning" The Open University]. However, all critical comments were welcomed by course teams and SLA's objectivity was clearly respected in this case.
It should be noted here that, taken individually, none of these courses involved large enough student numbers to make any analysis statistically significant. However, the findings were able to provide indicators for further study, and there is already evidence that indicators derived from the findings of the first iteration are being confirmed in the second. Therefore, while the findings may not be significant, some of them will likely be "highly indicative".
Top
The Courses and the Delivery Technology
The courses to be evaluated were part of Ryerson Polytechnic University's Certificate and Degree Program for the Canadian Retail Sector, and were deliberately innovative and experimental in the choice of delivery technologies.
CZRS 101 - Retail and Services Management
This course used a very wide range of technologies, including: videoconferencing, audioconferencing, computer conferencing and print. It was delivered mostly in synchronous formats.
CZRS 108 - Issues and Innovations in Retailing 1
This course used a narrower range of technologies, including computer conferencing (e.g. internet searches, bulletin boards, chatlines and e.mail) and print It was also delivered mostly in synchronous form.
CZCM 101 - Introduction to Retail Management Communication
This course will use the same range of technologies as CZRS 108 but will be delivered totally asynchronously.
The use of this wide range of technologies, and the changes from synchronous to asynchronous delivery, will enable a systematic evaluation of the technology to take place. More importantly it will enable the evaluation to determine how these technology ranges and their degree of synchronicity affect the educational aims and outcomes of the courses. For example, it will help to test the belief held by many educators that face-to-face contact is necessary for a true development of critical thinking.
Top
A Focus on Educational Considerations: A Model for Evaluation
The evaluators were well aware of the dangers of over-concentration on technology at the expense of essential educational considerations. The evaluation model was therefore based on five essential educational concerns that were identified by the evaluators as:
- What are stakeholders* trying to teach/learn? = GOALS
- What methods and technologies are being used to achieve those teaching/learning goals? = MEANS
- What content material is being used to achieve those teaching/ learning goals? = CONTENT
- How are other factors in the process of teaching/learning, impacting on the achievement of those teaching/learning goals? = CONTEXT
- How well were those teaching/learning goals achieved? = OUTCOMES
* There were deemed to be several possible stakeholders in the educational process: course designers, teachers, learners, and sponsors, (who may be the state, the educational institution, employers or other benefactors)
A model was therefore constructed to be used as a conceptual tool for the evaluation. It was constructed on the basis of the key factors needing to be studied, to ensure that the evaluation had a sound theoretical base.
The evaluation model (see figure 1) therefore identified the investigative areas where questionnaires, interviews, observation etc would focus.
Top
Figure 1.
An Evaluation Model for Technology Enhanced Education
(Click to Access)
Means
For the purpose of this paper we will focus on the area of MEANS because this typifies the attempt to balance pedagogy and technology. However, this does not mean that the overall research is ignoring other aspects, such as Goals or Outcomes.
This evaluation of a series of technology-enhanced courses was fully conscious of the need to locate the means of achieving educational goals squarely in its educational context. To this end, SLA looked at the pedagogy used to deliver conventional courses and then considered how this might be affected by the use of new technology.
Models of Pedagogy
In order to study the relationship between educational aims, pedagogy and the use of technology, we devised a simple typology of pedagogical methods:
- Didactics
In this method, we suggest, a perceived set of facts, or theories are imparted in a one-way interaction from teacher to student(s). The active participant is the teacher and the learner passively receives teacher output. An example is the university lecture.
This method allows one teacher to impart information to a number of students. In the traditional classroom, this number is limited by the size of the room. Once technology is used, this can be extended to any number, over any distance, and at any time.
While this can be a very cost-effective method for imparting knowledge, in terms of staff costs, it may not be effective for teaching skills and may have little effect in changing attitudes. While ideal situations in the traditional classroom might be thought to provide for satisfying interpersonal relations, in practice the face-to face classroom situation is more usually that of one professor to too many students to allow any meaningful interpersonal interaction to take place. This is important to remember in comparing conventional and virtual classrooms.
- Discourse
In this method, we suggest, facts or theories are imparted through a two way interaction between teacher and student(s). An example is a seminar. The teacher and students are both now active participants in the educational process. This is relatively easy to translate into more distanced applications, through audioconferencing or on-line chat groups, which can retain the active participation of students.
This can be cost-effective in terms of teaching staff costs but may be more costly in terms of technical support staff and, again, will require more time for planning, which must also be costed.
Discourse can be private (usually between teacher and individual student) such as discussing grades or educational problems, or public, such as in the form of seminars or chat lines.
- Discovery
In this method, we suggest, experiences, facts or theories are explored through an interaction between the student(s) and a variety of sources. In this scenario students have taken active control of the educational process. In its purest form, it could obviate the need for an educational institution at all, unless the student requires some form of certification of their endeavours.
The problem for the student is that the quality of educational experience that can be acquired for “free” may well not be satisfactory for the student’s needs. For example, while the Internet had free knowledge as an original ideal, this is being challenged by the fact that it costs money to acquire and organize knowledge, and these costs may need to be recouped. In addition, unstructured as opposed to structured discovery may be a very inefficient way to acquire knowledge and skills.
It seems likely that, while the need for the certification of educational effort and the organization of knowledge acquisition still exists, there will continue to be a role for educational institutions, where teachers take on a facilitative role and aid students in the organization of their acquisition of knowledge, skills and attitudes.
It should be noted that, as the cost of education becomes more manifest to students, and as technology allows for more effective distanced learning, commercial organizations have also entered the learning facilitation market via the Internet. For more details of this see "The Stone Age University"
While the focus is shifting from teaching to learning, and the role of teacher is changing from teaching to facilitating, the popularity of didactics is supposedly declining. Yet the problems of costs and time may be working against this shift in favoured methods. Unstructured discovery can be very costly for the student in terms of time, and time also costs money. On the other hand, well structured discovery can be costly in terms of teaching, both in time and financial terms.
This is an important consideration because an accompanying paradigm shift is an increasing emphasis on lifelong education (or learning), where students are more likely to be studying part-time and may well be holding down a full-time job as well. While the joys of discovery may be legion, especially for younger students, those already in the workforce may well prefer the short cut of “talented didactics” to acquire new bodies of knowledge. A mixture of pedagogies may still be the best option for achieving successful outcomes and customer satisfaction.
As Tom Creed of St. John’s University describes it:
"I view my role in the course, then, as setting up the environment so that my students will most likely learn the material as well as possible, but I do spend some time playing expert and transmitting information ."
The three pedagogic methods outlined above are, of course, ideal-typical concepts that in real life tend to merge into one another, or be used in conjunction with one another in order to achieve specific educational aims. All three methods can be used effectively in technology-enhanced education. Course teams designing technology-enhanced delivery systems need to decide which (or which combination) will be best for achieving the specified educational goals, and then how technology can be used to maximise their effectiveness. For a brief discussion of this see "Technology and the 3Ds"
Top
The Use of Technology
Once the basic educational aims and methods of the course have been decided upon, consideration can then be given to the use of technology, both to enhance the educational process and product and also to achieve other aims. These other aims may be to reduce costs by reaching more students through the use of virtual classrooms or by freeing the course from time constraints. In other words, the educational needs should dictate the selection of the technology not the reverse:
"What do we want to accomplish in our courses, and can technology advance our teaching goals?" [Creed (1996) ibid]
In this regard, it is clear that making decisions about technological enhancement of courses needs to be a collaborative process, involving both educators (including teachers and administrators) and technologists, with effective communication between the two. A common problem here is that both sets of participants have their own set of technical terms, often viewed as "jargon" by the other. Education, for example is a multi-disciplinary social science that is riddled with esoteric terminology. Similarly, technology has its own special language. So teachers and technologists will not necessarily be on the same wavelength. For example, it may be a rare academic who can cope with such technological acronyms as MOOS and MUDS!
With this in mind, it is best to discuss matching educational aims with the use of specific technologies in pragmatic terms, couched in everyday language where possible. This approach has been adopted in the following discussion
We would argue that an important distinction is between the intrinsic and extrinsic features of technology. Therefore choosing a suitable technology will depend on consideration of:
a) the intrinsic nature of that technology and
b) the extrinsic factors present in the context in which it will be used.
Intrinsic Features of Technology
In order to determine suitable technology, we identified some basic questions that need to be asked:
- Audience and Direction
Do the features of the technology allow communication with the size and type of audience you want to reach and in the way you need to interact with that audience?
The most usual variations of type of audience and interaction will be:
One to a group or
groups
where a teacher needs to communicate information to a limited number of
students but does not need a reply, and so the interaction is one way
One to unlimited
where the teacher needs to communicate information to a very large number of
students or all groups of students but does not need a reply
One and One
where a teacher or student needs to interact with one other person to
exchange information
One and group(s)
where one person (student or teacher) needs to obtain or exchange information
with a group or limited number of groups
One and unlimited
where one person (student or teacher) needs to obtain or exchange information
with an unlimited number of people
group and group
where a group of people needs to obtain or exchange information within the
group or between groups of people
- Time and Space Sharing
The question here is whether the technology allows for the sharing of space and
time in the way that is required.
The usual combinations are:
Sharing real space in synchronous
time
This is the conventional classroom situation, where teacher and students are
both present in the same physical space at the same time.
Sharing virtual space in synchronous time
This is where teacher and students are separated in physical space but share a
virtual space (e.g. through videoconferencing, audioconferencing or chat lines) in
synchronous time.
Sharing virtual space in non-synchronous time
This is where the student learning has become space and time- independent, such
as the use of bulletin boards for discussion. These can have some specific advantages
over interacting in real space:
"class discussions over computer networks leave a record,
unlike verbal discussions (every student knows what it's like trying
to decipher his or her class noted after a particularly heated debate).
If you want to refute a previous point or refer to an early
lecture in an essay, they are immediately accessible."
[Nayman (1997) "Getting Connected with Connect Ed."
Educom Review Vol 32 No 3]
Sharing real space in non-synchronous time
While this would seem to be rare there are documented cases, such as that of
students who had access to a shared social space and, in spite of the ability to use
electronic chat lines, would leave each other written messages on a physical bulletin
board! [Knowledge Connection (1996) "Innovations in Course Delivery for a
Distributed University: The Evaluation of the Interactive Learning Connection-University
Space Network Pilot Project"]
Sharing neither Time nor Space
The most time and space independent of all media is one of the oldest - print!
Although the use of laptop computers is beginning to change that, you still don't need a battery
to read a book. In addition reading large amounts of text from a computer screen
remains non-ergonomic for the human eye.
- The means of communication & the need for recordkeeping
The educational aims and the preferred type of pedagogy will suggest the need
for certain means of communication, and whether or not a record of the communication
is required:
Visual Communication
Do participants need to have visual contact in one or both directions? Does
there need to be a record of the interaction? Will the educational institution provide the
record or will it be up to the student. This has cost implications for both parties. If the visual
need is for illustration purposes rather than an aid to personal interaction, the need for real
time transmission may be obviated.
Aural Communication
Do participants need to hear each other in one or both directions? If so, does
there need to be a record, and who should be responsible for providing it? Depending on
the nature of the educational need, once again real time transmission may not be necessary.
Written Communication
Can the interaction take place in writing? Is this in fact better for the
educational purpose? Is it better via physical print or electronic format? There are advantages
and disadvantages to both options. Physical print is costly to produce and costs are difficult
to justify or recoup unless student numbers are very large, and it is also next to
impossible keep up to date on a regular basis. Physical print certainly provides an automatic
record, but it is now also possible to print from electronic delivery formats.
There are many choices of technologies or combinations of technologies that will
provide these various means of communicating to students. They range from video-, audio- or
computer-conferencing, through the use of television video-tapes, radio and audio-tapes,
to good old-fashioned books. All have implications in terms of cost and effective usability
for both students and staff.
- Linearity
The desirability for non-linear performance is an important consideration in choosing suitable technology.
This feature relates very much to accommodating different student learning styles
and tasks. While print can be considered a linear type of activity (i.e. you start at the
beginning and go through to the end), it is still possible to browse or dip into a book to find
specific piece(s) of information.
This search capability is made even easier if the print is in electronic form.
However, there are sometimes limitations in the design of electronic formats for presenting
information, and students can sometimes be forced to proceed through an electronic presentation
without the ability to stop and review parts without going right back to the beginning.
- Interaction/Routing
The degree of interactivity is another important issue. While some technologies
seem to provide interactivity (i.e. students can receive a response to an input, this is
sometimes merely routing rather than true interaction. For example, a student is asked to respond
with a yes/no answer. If the answer is yes, they are routed through one section of the
structure, if the answer is no, they are routed through a different structure.
True interactivity is capable of dealing with different types and large numbers
of responses, and not simply preset procedures for dealing with a very limited set of responses.
But it will not always be necessary for achieving a particular course aim.
- Costing
The cost of using a specific technology is not so simple to determine as it may
first appear. There are a whole variety of factors to take into consideration, and in
addition there are two main complications:
- The question of hidden or" sunk" costs as opposed to overt costs; and
- The problems of cost distribution, not only between the students and the
educational institution, but also over the shelf life of the educational course.
For example, planning and support time seems to be a high cost associated with the use of technology, but this may simply be that planning costs are for the first time becoming manifest rather than hidden within the teaching role.
In addition, though technology-rich courses may seem to have high planning costs,
the ability of technology to update the course content at lower costs allows the shelf life of a
course to be extended as long as the demand exists, and this can reduce costs
considerably over time. Some courses can have very long shelf life (mathematics, for example)
while others by their very nature require more frequent updating. One such course was one
of the two already evaluated: "Issues and Innovations in Retail." This course would
obviously benefit from being available in an easily updatable format.
The costs of conventional courses have often become embedded in long
existing structures and processes that are no longer open to scrutiny, but this is not true for
new technology-enhanced distanced courses, where new equipment, staffing and
operational costs are all clearly obvious. Performing a cost-benefit analysis of conventional and
distanced courses is therefore not an easy task.
Another problem with assessing cost will relate to the number of students taking
a particular course. As technology frees educators and students from space and time,
many more students may be able to access a particular course with the same number of
teaching staff. This usually means much lower marginal costs. In that situation it is difficult to
assess costs until the student enrolment numbers for a course are finally known, and this can
make a critical difference to the financial viability of a course.
The whole question of educational costs is, in fact, such a vast subject in itself
that within the confines of this paper it is not possible to examine it in the depth it needs
and deserves. Only some of the issues associated with costs have been raised here and we
are currently investigating cost issues, and assessing methodologies that will allow
educational institutions to make more informed decisions on costing their technology enhanced
and distanced courses.
Comparing technology-rich courses with more conventional ones, not just in the area of
cost but also in efficacy, may be like comparing apples and oranges if the conventional course is
no longer viable for nontechnical reasons, such as the lack of a market for space and
time-dependent courses that have become dated. Unfortunately, such content-obsolete courses are not as rare
as one might expect. While universities have a monopoly on accreditation, the pressure to
remove them from the list of required courses is not always sufficient. Technology, with its constant
and increasing rate of change, can leave some faculty members unable to keep pace with
new developments and, therefore over-reliant on old content or skills. Indeed we know of at
least one faculty that has negotiated with its university that no existing faculty member shall be
required to use new technology. [Young (1997) "Canadian University Promises It Won't
Require Professors to Use Technology" The Chronicle of Higher Education October 3 ] In the short term this type of defensive measure may secure jobs but, with increasing competition from
off-shore universities offering accredited distance courses, it will be self-defeating as students
transfer to more techno-savvy educational providers.
The choice of a particular technology or set of technologies is a time-consuming task
that requires coordination and communication between teachers and technologists, as well
as educational administrators. It is further complicated by the addition of extrinsic factors that also
affect the successful implementation of educational courses.
Extrinsic Factors in the Use of Technology
As technology frees the teaching learning process from space and time, so the context
in which it is used becomes ever more variable, and less and less under the control of the
educational institution. This means that very careful consideration must be given to the likely context
in which the chosen technology will operate.
This is further complicated by the fact that the context is a moving target, continually
affected by new advances in technology, and other social and economic changes. Some
considerations that have emerged in this research, which we believe important are outlined below.
- Level of availability
How available is the technological hardware, software, support and user know-how among the population from which the students are likely to be drawn? If any of these
are not generally available this can cause problems for a course. For example, there
can be a problem in accessing the Internet in the more rural areas, where even a national
provider may not be available, unless expensive long distance charges are incurred.
- Level of Compatibility
While technology has made great strides in the area of compatibility, there are still problems in this regard, such as the ability to read attachments. Different versions of software can also pose problems. While possibilities for more sophisticated activities may be available
from newer software, if some students have earlier versions this may have to be foregone.
- User familiarity
While an increasing number of people are using computers at work, less have them
at home and even less use them to access the Internet. Company employees do not
generally have access to the Internet, although some may have access to the company Intranet.
The likelihood of many students from the retail trade (or indeed from other industry
sectors other than IT) having a good working knowledge of the Internet is not very high. If
Internet use is to become an essential part of a course, thought and planning will need to go
into orienting students to its use BEFORE this skill is required to undertake research
or assignments. In addition, the use of a specific technology for EDUCATIONAL purposes
may require a specified set of protocols/practices that differ from more casual use.
Even experienced technology users may require time for this adjustment.
- Place of operation
If a course is to take place in virtual space, the conditions of this space become
an important factor in the educational process. This may be a remote site where some
students come together, or it might be an individual student's home or workplace. Either way,
the educational institution will need to take account of conditions at these sites in its
course planning. What special equipment is needed there? Are staff available to operate it? Are
they fully trained in its use? What other facilities are needed? What allowances (if any) will they make for problems that arise from the location? For example, a student using a
computer at work may not be able to access the Internet because of a company firewall.
- Timing
Many students, especially part-time ones who are active in the workforce, will be attracted to courses that allow for asynchronous response (i.e. free(r) of time
constraints). But this is also a factor that must be taken into account when planning the course,
especially if teamwork is a course requirement. Team coursework or assignments that
require synchronous communication between team members may be difficult if some students
are using computers at work and are available only during working hours, while those
using computers at home are only available out of working hours.
- Availability of technical support
Given the generalized lack of technical know-how amongst the likely student population, the provision of technical support is almost certainly an essential part of
any technology-based course. How that support will have to be costed and provided is
another important consideration for a course planning team. For example, providing technical
support in office hours only will not meet student need in a distanced and continuous
educational setting, where most students will be in the workforce and have to study in the
shoulder hours. There will be a need to consider what business calls a "24/7" service.
- Cost
While it is possible to download costs from the educational institution to the
student, this may be self-defeating when it comes to marketing the course. If the student has
to acquire expensive new hardware and software, as well as pay the course fees, this will
serve to reduce the demand for this type of course. While educational niche markets are
pursued by commercial enterprises, the traditional university has traditionally had to cater for a
more general educational consumer who may be much more cost-sensitive.
- Time
Most students for this type of course are already in the workplace, with severe constraints on the time available for continuing education. This means that the
technology must make their time spent on the course more effective rather than less. Having to
struggle with unfamiliar technology that adds to their sense of stress can be seriously demotivating. Either the technology chosen must be familiar or technology orientation must be part of
the course.
In general the more widely available and well tested the technology the better. Also
restricting the course to the use of one or two technologies may be preferable if planning time is at a premium.
Technology-based courses will certainly take up a lot more planning time than
conventional ones. They also require good communications between educators and technologists. The
lead time to convert from a conventional to a technology-based course, or to create a
technology-based course from scratch, will depend on the resources available, but will not in any event be
short.
Careful planning for technology-based courses is essential, but the resources put into
planning can repay the institution in the longer term. Customer satisfaction will be higher and this
will increase the demand, which can in turn reduce net costs. Added to this, well planned courses
can have a longer life, again a cost reducing factor.
In any event, the institution may have no choice but to plan for distanced courses. This
is where the growth in educational demand is to be found. In the USA, for example, there can
be seen to be 6 distinct groups of students making up the higher education population.
[Twigg (1997) "Notes for NLII-ITP Symposium on Creating and Delivering Collegiate Learning Materials"]
| Group |
Ages |
Enrolment |
Location |
Student type |
Workforce |
Numbers |
| 1 |
17-24 |
full-time |
campus |
trad/ 1st Degree masters/PhD |
not |
3.9 million |
| 2 |
22-34 |
full-time |
campus |
trad/ 1st Degree masters/PhD |
not |
0.65 million |
| 3 |
17-24 |
part-time |
campus |
semi-trad 1st Degree |
non-career entry level |
2.9 million |
| 4 |
22-34 |
part-time |
campus |
semi-trad masters/PhD |
career (on-campus) |
0.48 million |
| 5 |
25> |
full/part |
on/off |
non-trad 1st Degree |
career workforce |
5.3 million |
| 6 |
25> |
full-part |
on/off |
non-trad masters/PhD |
career workforce |
0.88 million |
| Totals: |
Full-time traditional
Part-time semi-traditional
Full/part-time non-traditional |
|
4.55 million
3.38 million
6.18 million |
This demonstrates the demand by non-traditional students for post-secondary education
in the USA. It seems reasonable to assume that the situation is similar in most of the
developed world because the pressures driving the demand are the same. Add to this the fact that ,while
the traditional student numbers are not rising in most of the developed world (with the
possible exception of California, Florida and Texas [Daniel (1996) Mega-Universities and
Knowledge Media Kogan Page]), the numbers of non-traditional students is likely to rise significantly, due
to advances in technology that require frequent and continued re-training and re-qualification.
In addition, in the developing world the demand for education is still rising and
distance education is likely to provide the answer to this demand, which must satisfy large numbers at
the lowest possible cost.
Top
Some Issues and Implications from Interim Findings
Some Emerging Implications for pedagogy
Faculty and familiarization with new technology
The students on both evaluated courses were highly motivated by the chance to discover
and use the technology itself - the medium was as important to them as the message. This should
not be surprising. Many workplaces now depend on technology, and experience of technology is
an important job skill to have on a resume for many people. This will continue to be important in
the educational workplace because technological change is a constant.
We would suggest that faculty members who attempt to cut themselves off from the use
of new technology (even if this is officially sanctioned) may nevertheless find themselves
under increasing pressure to come to grips with it in order to remain relevant to their students, who
will likely be drawn more and more from non-traditional educational populations.
Conversely, faculty who enjoy and are skilled at imparting new knowledge with the
new media will have the chance to become academic "stars", rather like TV news anchors, with
a whole production department supporting them behind the scenes. This may mean that
teaching skills will at last become as valued and important as research skills, and as equally scarce a resource.
Synchronicity, faculty workload and optimal student numbers
On the second course evaluated, the range of technology was narrower than the first
but there was still a considerable portion of the course conducted synchronously. The professor
was also more distanced from her students, who were also much larger in number than on the
previous course. As we monitored the course, it became clear that the number of students was
causing problems in two areas.
First, the use of synchronous chat lines was proving problematic. There were too
many students for a meaningful and sufficiently ordered discussion (in spite of the fact that students
were engaged with the subject matter and were obviously participating in the discussions with
interest). An analysis of the chat logs showed that questions and answers were totally out of sequence
and group chats were interfering with class discussion. This situation improved somewhat as
students learned to use group chat lines instead of the main line for team communications, and as
the professor provided a clear protocol for class discussion. However, the numbers still remained
a problem and in interviews with the students, we found that the sheer numbers trying to use
the technology was interfering with their ability to conduct a meaningful discussion.
Second the amount of tutor workload was considerable. A whole morning was devoted
to synchronous delivery of the course; much extra time was spent outside the virtual classroom
in responding to student concerns (and these were not always of a strictly educational focus,
often being problems related to using the technology). In the remaining time available the marking
of assignments was becoming overwhelming, and would certainly become unsustainable without
the delegation of at least some of this task to another academic. The problem of assessment was
more a function of the number of students than the sharing of real time. So distance courses, with
large numbers of students, will obviously require fewer teachers but many more assessors.
Fragmentation of academic roles in distance education delivery
Synchronous distance courses are time greedy and will be limited in terms of the number
of students who can be effectively accommodated in educational terms. From student feedback,
and from our observations in monitoring the course, it quickly became clear that 39 students on
a synchronous chat line was too much for truly effective learning.
Asynchronously delivered courses, if successfully promoted, will be associated with
much larger student numbers. The need to provide extra assistance to a "lead" professor, in terms
of student support and assessment, will grow as courses become more truly distanced and the
numbers of enrolled students increase. This will be another pressure for role "fragmentation", and
new specializations that relate to activities as much as to subject specialisms. A gifted teacher will
be best used in "front of the camera" and others more camera shy could develop another expertice, such as the assessment of distance students or research in educational technology.
This, along with willingness to use new technology, will serve to promote the "star" system.
Team membership and educational concerns
It also emerged very early in the evaluation that an academic will quickly become
outnumbered by technologists and even administrators on course teams. This seems to be a normal
composition of this type of team. For example, Professor Robert Kowalski, Head of the Department
of Computing at Imperial College delivered a lecture in September 1997 on computational logic
and legal reasoning. This was broadcast live on the Internet as an interactive webcast. It was
delivered to a live audience at a conference in Dresden. (For those interested it is still available on the
web - kmi.open.ac.uk/compulog/story.) For this broadcast there was one academic and
five technologists. While this was a delivery course team, a design course team has much the
same composition. This raises the concern that educational/intellectual issues will begin to take a
back seat to technological issues. Part of the problem is that academic staff who do not know what
the technology can do are at a distinct disadvantage in this situation. There is much to be said
for devoting some resources to staff training in the new technologies, enough that they have a
reasonable knowledge of the options open to them in achieving their educational aims at a distance or
enhanced by technology.
This seems a more positive approach than refusing to use the technology, if only
because technology has much to offer in achieving educational aims.
Academic freedom and the use of technology
Traditional academic freedom becomes increasingly problematic in the distanced
learning environment. For example, if a number of academics, none of whom necessarily had a hand
in designing the course, are marking sets of students on the same course, there is a need to
standardize the level of marking to ensure fairness to the students. This means that the assessment
procedure must be monitored. This will be an anathema to many traditional academics. For those of us
who have taught at distance universities and are used to this procedure, there are benefits to be
obtained from the process. Feedback from other academics can be very welcome in a distance
environment where this is unusual compared to a conventional university.
Personal Interaction with students
This is a difficult and unresolved issue. There is evidence from our evaluations that the use of
technology does have an effect on how an academic interacts with students. For example,
we would categorize the professor on the first two courses as a gifted teacher who enjoyed
the personal interaction with students and was good at communicating with them. Nevertheless,
it became a challenge to retain the quality of the personal interaction over increasingly
distanced delivery methods. For example, less expansive gestures had to be adopted to avoid jerkiness
in camera movements in the videoconferencing sessions. The professor herself was concerned
that this type of problem was affecting her ability to engage students in the same way as she could in
a conventional classroom. This in spite of the fact that she was enthusiastic about trying the
new technology and soon learned to have plans A, B and C ready for occasions when the
technology was causing a problem.
In order to address her concerns we probed students in the interviews about their reactions.
It was interesting to discover that, while students on the first course were generally satisfied
that they had had good personal interaction with their professor, they could not generally
imagine being as satisfied with a more distanced approach. Students on the second course, which used
the same professor but a more impersonal, distanced delivery, were nevertheless generally
satisfied with their interaction with their professor, and yet they could not imagine being satisfied with
an even more impersonal, distanced delivery.
This seems to suggest that students adjusted to the type of delivery mode with less of
a feeling of remoteness than one would have expected. One clue to the reason for this came
from observations and several responses by students on the first course. In this course the
professor operated from two sites alternately. This meant that on every other week the students at one
site were distanced from her by videoconference or audioconference, while the other
students interacted with her face-to-face. We noticed that, when she was at a site in person, students
sat up to the tables and were generally less relaxed and more engaged than when she was not
present in person. Several students told us that they felt "deprived" when she was at the other site. Yet
in interviews all students said they were very satisfied with their interaction with their professor.
One student's comment suggested an answer to this apparent contradiction. "I don't think I
would have felt deprived if I had known the other group was also at a distance."
It will be interesting to discover just how satisfied students on the third course will be.
This course will have only web-based interaction between students and professor. Sadly we will not
be able to compare delivery methods using the same professor. It may be that videotapes, or
video on web (currently still problematic because of bandwidth or modem problems), will be needed
to establish sufficient "rapport" between students and teacher(s). Or it may turn out to be the
contact between students, rather than between students and faculty, that is important to
successful educational outcomes. We also suspect that familiarity with the technology will also be
an important factor in this issue.
Implications for technology choices
Student anxiety about using technology
Using a reflective technique we found that even students who were outwardly
enthusiastic towards new technology could still have underlying anxieties about its use. This means
that educational decision makers would be wise to consider the familiarity of the proposed
technology to the population they are targetting.
This also has implications for the range of technologies used on a course. If these are
likely to be largely unfamiliar to the students, it is better to keep to a fairly narrow range so that
students do not have too many new technologies to cope with at once. It may also be necessary to
provide some form of technology orientation or prerequisite, for those who need it, before they
are required to use the technology to work on or submit assignments. This request came up again
and again in the interviews with both groups of students.
In addition, it became clear from monitoring the course, analysing the chat logs and
interviewing the students, that the educational use of this technology needs to have clearly specified
protocols or "netiquette" if it is to be educationally effective. Even if students are experienced in
the technology, they may still require some orientation to its use in an
educational setting. This will require the input of educationalists and not just technologists in the preparation of any such
courses or sessions.
Effective learning will clearly relate to the ease with which students can use the
delivery technology. The most expensive technologies, which are often those which come closest
to approximating the qualities of face-to-face contact, and which we found to be generally
popular with students, may not be necessary or the most effective for successful learning. When
asked what they considered the important learning points on the course, and which technology they
had used to learn it, print and videotapes were more frequently mentioned as effective than the
more popular technologies. This suggests that more impersonal and distanced technology can
be associated with effective learning.
Implications for Structures
Course teams and their composition
One question that arose from findings about the need for continued student
technological support during the course was: for how long should course teams be constituted? Developing
a course may require one set of activities and people, delivering a course a different set of
people, maintaining a course another set of people/activities, while supporting a course (i.e. its
students) quite another, and assessing course outcomes yet another.
Many of these functions have been traditionally contained in the one role of "professor".
With technology-enhanced courses, especially if they involve distanced education, the role of
the professor is likely to change from a multi-role, subject specialist to a single role
task/subject specialist. We have outlined below some possible course team structures for the various
stages of the distance educational process. These partly depend on the technologies used for
delivery and assessment. If, for example, the course is totally web-based, broadcast specialists would
not be needed. If assessment is totally computer-based, the number of academics on a delivery
team will be reduced. However, representatives from four main areas will generally be required on
all teams:
academics
educational technologists
technologists
administrators.
In addition, other people with required specializations, such as marketing or print
buying, may need to be present from time to time.
| Course Design/Creation Team |
| Role | Activity |
| Academic Content specialist(s) | Research and organization of content |
| Educational technologist | Learning structure of course/assessment procedures |
| Technologist(s)/Media Specialists | Development of delivery mechanisms |
| Production specialist(s) | Plan production schedules |
| Financial Planner | Cost projections/budgeting |
| Administrator | Plan administration of proposed delivery |
| Market Researcher | Provide feedback on market conditions |
| Coordinator | Process communications and requirements of team |
| |
| Course Production/Delivery Team |
| Role | Activity |
| Academic Delivery specialist | Interaction with students (online/broadcast etc) |
| Academic Assessment specialist(s) | Assessment procedures |
| Educational Technologist | Monitor assessment/standards |
| Production specialist(s) | Print/broadcast/online production |
| Administrator(s) | Registration/financial organization/processing |
| Coordinator | Communications/reports/records |
| |
| Course Maintenance/Support Team |
| Role | Activity |
| Academic(s) | Revisions to content/educational student support |
| Technologist(s) | Technology troubleshooting |
| Educational technologist | Assessment problems/monitor assessment process |
| Administrator(s) | Administration of student support services |
| | Administration of course maintenance |
| Coordinator | Team communications and reports |
In addition to the above one might also need a Summer School Team and an Exam Team. It is easy to see how a full open univesity course team can reach 40 in number!. While not every role will necessarily require a separate person (for example, an
academic may also be able to act as an educational technologist), it will be important to have enough
people to cover the range of activities. This also relates to the costing of courses.
Implications for Institutional Structures
When people come together in teams of the above type, they almost always cut
across existing departmental boundaries. If attempts are made to confine them within existing
structures this will soon impact on the institution's ability to effect economies of scale. As the financial
case for distance courses rests largely on the low marginal costs (especially as the institution has
to operate in real and virtual space simultaneously), economies of scale become essential.
While distance or online courses for the non-traditional educational consumer may
well originate in a university's school or department of continuing education, they will require
technical and other support that could also cover such courses elsewhere in the university. Sooner or
later, pressure will build to change existing university structures to accommodate this possibility
and take full advantage of expertise gained in the creation of such courses for more general benefit.
The balance of academic to non-academic staff
As technology-enhanced courses are adopted, the need for academic staff will diminish
in relation to the need for technical staff, which will increase. As with all technological change,
jobs will disappear and other jobs will take their place but the personnel will not necessarily be able
to make the transfer from one type of job to another.
While some new academic jobs will become important, such as that of educational
technologist (a person who is largely concerned with the way teaching and learning is made effective), they
will not necessarily relate to subject specialisms. "Star" specialists will become even more
important and scarce than they are at present but the specialism may now also relate to the ability to
TEACH as much as to create knowledge. At the same time technology will enable these stars to
reach many more students than previously. The problem for the average academic will be more serious. They will be less needed to teach but will still be required for assessment and also for
student support services. These activities will not be location specific and many academics may
find themselves teleworking.
A second and critical area of concern is the power relationship within a course team. If the course is to retain academic credibility, it is essential to ensure that the academic(s) has the POWER to ensure academic/content issues are not diluted by so-called technological imperatives.
Implications for Costs
Technology support services and costs
If students require help and assistance with the technology (and we have found that
even those who say they are familiar with the technology still need this type of assistance from time
to time), this has cost implications. In addition, the timing of that need is not likely to be during
office hours if the course has substantial asynchronous components. This impacts on staffing
arrangements as well as on costs.
Another important factor here is that the problems many students experienced were
not necessarily under the control of the institution or the student. For example, several students
on both iterations complained of having problems in submitting, or having enough time left for,
or communicating with other team members for assignments, because their ISP was "bunching"
their e.mail and it was not reaching them until much later than the sender expected. Having
"expensive" educators or technologists dealing with this type of problem is not cost-effective and lower
cost administrative staff will need to be involved.
Costing Courses
There is no doubt that traditional management accounting practices will not provide
the quality of information that is needed to make informed decisions about distance education courses.
Three major criticisms of traditional accounting cited by Greville Rumble
[Rumble (1997) The Costs and Economics of Open and Distance
Learning Kogan Page] are:
- the failure to deal adequately with overhead costs (as opposed to direct costs) which often form the major cost areas;
- the over focussing on the costs of direct labour (ignoring the importance of other
factors) and thus causing an over focus on this area for attempted savings; and
- the fact that overheads are often driven by factors that do not relate to volume of production and services delivered, and so an inadequate attention to the details of overheads results in
failure to understand the real workings of the cost structure.
Rumble's suggestion for a better cost accounting system is to use "activity-based costing".
This is based on the logic of "considering the design of the system and identifying everything that
might result in a cost being incurred."
[Rumble (1997) ibid] As a course team structure is built on the activities of the members, this makes a lot of sense.
The problem for conventional universities is making the necessary comparisons between
the costs of conventional and virtual courses. The costs of conventional courses typically hide
sunk costs that are made manifest in the costing of virtual courses and this results in inaccurate
cost comparisons. New costings for conventional courses, which are also based on activity-based
cost accounting will be needed for a true and meaningful comparison.
The popularity of technology, development costs & financial resources
The demand for technology-enhanced courses (which our findings to date suggest is a
real factor) stems in large part from interest in the technology itself. This may encourage the
production of more technology-enhanced courses, which can have lower marginal costs (depending on
the technology used for delivery).
With a demand for technology-enhanced courses, academic institutions will find a
heavy drain on their resources as they attempt to satisfy this new market as well as their
traditional classroom based clients. Technology-enhanced courses are generally acknowledged to be
more expensive to develop than conventional ones. This will pressure universities either to
divert resources away from conventional courses, which seems unlikely, or to seek other funding
sources, such as commercial partnerships.
The need to operate in real and virtual space is a serious problem for universities that
cannot be understated. The merger and acquisition fever that has gripped, and continues to grip,
the commercial world is a real possibility for universities. There are several models that can
be suggested for the survival of the university and all of them involve radical changes in structure.
This whole issue deserves much more space than can possibly be devoted to it in this paper.
We are currently researching this area, and those who are interested will be able to access our
discussion papers on the web.
Top
Conclusions
As universities move to using new communication technologies to service campus students, and some more distanced students, they will inexorably be drawn into new structures and financial management systems.
Faculty will find they are not only required to offer some awareness of new technology, but will inevitably become involved in new, more fractured roles in technology mediated, non-location specific course design.
The new "dual" universities, operating in real and virtual space, serving local and distant students, will be subject to great stresses, which will probably require collaboration and cooperation between them to enable a successful shift to this new mode of operation.
As Sir John Daniel, Vice-Chancellor of the Open University in U. K. has suggested:
"The key challenge for campus universities is to change from a teacher - centred model of operation to an approach that emphasizes learning productivity - and to carry students with them. "
Copyright
Users of this material are permitted to make a copy for their own research/study purposes. However, no part of it may be stored in a retrieval system or transmitted in any form or by any means, without permission in writing from the copyright holders.
© 2001 Sims Latham Associates
Top
|