Journal Home | Close window

Vol.7, No.1 April 2001
Editorial | Contents

Assessing and Evaluating Student Contribution to Electronic Discussions

Refereed Paper

Authors

Allan Herrmann BSc. (Qld) Dip. Ed. (Qld) M.Ed. (UNE)
Senior Lecturer
Centre for Educational Advancement
Curtin University of Technology

Dr Jill Downie, RN, PhD
Senior Lecturer
School of Nursing
Curtin University of Technology

Associate Professor Bev O'Connell, RN, PhD, FRCNA (Contact Person)
School of Nursing
Deakin University
Waterfront Campus
Geelong 3217

Phone (03) 522 78402
Email bevoconn@deakin.edu.au

Abstract

Tutors in face-to-face teaching and learning contexts, evaluate students' participation in order to provide assessment that contributes towards the students' final grade. Similarly, in on-line learning environments, there is a perceived need to reward the quantity and quality of student interactivity. However, the different nature of the context presents new challenges. Specifically, without the visual cues and immediate feedback, so important in face-to-face communication, the evaluation of students' contributions to on-line learning activities and interaction demands new instructional and assessment skills. A unit of study at an Australian university, using computer mediated communication, was reviewed to address questions related to the appropriateness of an on-line evaluative process.

Introduction

Tutors in face-to-face teaching and learning contexts, evaluate students' participation in order to provide assessment which contributes towards the students' final grade. Similarly, in on-line learning environments, there is a perceived need to reward the quantity and quality of student interactivity. However, the different nature of the context presents new challenges. Specifically, without the visual cues and immediate feedback, so important in face-to-face communication, the evaluation of students' contributions to on-line learning activities and interaction demands new instructional and assessment skills.

A unit of study at an Australian university, using computer mediated communication, was reviewed to address questions related to the appropriateness of an on-line evaluative process. In addition students and staff were interviewed to identify the skills students need to communicate and the perceived appropriateness of the learning environment. This paper addresses some of the issues arising from the feedback and reports the use of interactive components, including the mechanism for assessing student contributions to the learning process with particular emphasis on the electronic discussion.

Context: Illustration 1

(http://www.curtin.edu.au/learn)

Within the many faceted electronic learning environment of Curtin University of Technology, Curtin Learning Link (CLL) (Illustration 1) provides an infrastructure for distance education and open learning students within which is offered a number of units of study. Overall, the CLL site offers a range of generic support services designed to meet perceived distance and open learning student needs and administrative requirements (Table 1).

Table 1: Curtain Learning Link site offerings

Linked to this site are a number of units of study. One of these, the Nursing Honours Program (Illustration 2) was designed to act as both an information starting point and a communications interface for students and staff involved in the unit. The two main communication forums used in this unit are the Nursing Conference Room (NCR) - a discussion group - and Thoughts in Action (TA) - a ‘write to the web’ facility for students. Both of these are designed for asynchronous use, i.e. participants do not have to be on-line at the same time but rather providing a place where messages and information are left to be accessed at times suitable for the user; thereby allowing the students and staff to organise their involvement according to their needs and availability. The NCR and TA are both passive forums in that the students and tutors must choose to go to the site to interact. This is in contrast to the intrusive devices such as listserves which send the messages to the users.

Conceptual design

Many teaching/learning web sites confuse information provision with the process of teaching and learning. The premises on which this site was designed are good practice:

• encourages contacts between students and faculty

• develops reciprocity and co-operation among students

• uses active learning techniques

• gives prompt feedback

• emphasises time on task

• communicates high expectations

• respects the diverse talents and ways of learning.

Chickering and Ehrman (1996)

Whilst this paper is not focused on the technical issues of CLL which are dealt with elsewhere (Boyd etal. 1996), it is necessary to identify those technical design concepts relevant to this paper: the need for a transparent interface; the need for the site to be fully accessible to students with minimal web expertise and hardware/software; a visual representation of the discussion (threading), follow-up and links to provide clues to assist students to follow the discussion; and the ability to evaluate the quantity of students’ participation. All of which need to support the main aims of the unit which are:

To fulfil these objectives, interactions were conducted via the web with only three on-campus sessions. The main aim of the campus sessions was to offer support and to monitor and evaluate the new unit on the web.

Illustration 2

(http://www.curtin.edu.au/learn/unit/NursingHonours/)

Computer Conferencing Learning Contexts

The use of Computer Mediated Communication (CMC) as a teaching learning tool is now well documented (see Berge and Collins 1995 for some initial articles). Similarly, the issue of the neutrality or otherwise of technologies in general and computers in particular has been subject to a deal of debate. (For an example of the range of research in this and the related area of tele-education, see the ‘no significant difference’ web site http://tenb.mta.ca/phenom/phenom.html) However, within a specific learning environment some forms of mediation provide greater benefits.

Kaye (1990) summarises the strengths of CMC contexts:

• the convenience of an asynchronous communication mode, which liberates users from both time and space constraints;

• its value as a medium of written communication, within a system in which students are graded essentially on the quality of their written work;

• the enhanced levels of interactivity between and amongst students, tutors, course developers, and other members of a widely dispersed learning community; and

• the reduction of the isolation felt by many distance learners and the potential of CMC for collaborative learning. (p.228)

Schwan (1997) also argues that CMC is not a neutral transmitter of information because of the influence of a myriad of factors that impact on the teaching learning process. For example, the large amount of information that can be presented simultaneously to a large audience of participants makes it a less controlled environment. He goes on to emphasise that, particularly with respect to asynchronous discussions, both tutors and students can encounter difficulties in following the various threads. In fact, a far more sophisticated understanding of the discussion is needed than for the linear face-to-face tutorial context. Students and tutors require skills to match the new environments and tutors need to be provided with efficiency tools to support their work both in teaching and assessment.

A study conducted by Burge (1994) that studied how students learned using CMC identified three types of learning skills exhibited by students (p.29):

Also three types of peer behaviour were required (p. 30):

Study design

Seven Honours students were enrolled in the unit with three of these completing the semester successfully. As not all students had remote access initially, computing facilities were in an Honours room, established within the School of Nursing. The facilities comprised three computers, all with Netscape Navigator 3. All students were invited to participate in the evaluation by completing an open-ended questionnaire three times; first prior to undertaking the unit, second mid semester, and finally at the conclusion of the unit. The questionnaire was designed to provide data regarding students’ general satisfaction with the unit and any changes in skills and attitude which might appear over the semester. The perceived appropriateness of the learning environment was also canvassed with a view to further development and improvement. Students were asked to select an unique four letter code for themselves which was entered on the questionnaire each time they completed it so that, while remaining anonymous, all responses could be matched and support the tracking of any changes.

Outcomes and discussion

A number of aspects related to this project were studied, this paper addresses two issues: changes in student perceptions over the semester and assessing student participation.

Changes in student perceptions

The first time the questionnaire was completed, students identified problems associated with their operational skills, for example only four of the seven students had previously used the web. They expressed concern that the environment was foreign to them and they were largely unaware of either the computer software or hardware necessary to support their interaction on-line. Whilst this initially caused some student anxiety, anecdotal evidence suggests that it was not directly the cause of students withdrawing from the unit. Rather, students who withdrew from the unit identified work pressures and over commitment as opposed to lack of skill or a dislike of the method of delivery as the cause.

Initially, students described their computer skills as ‘average’ or ‘not very good’. However, completion of the unit resulted in a considerable positive change in students perceptions of their computer skills to ‘very good’ or ‘average’. Given this was the only unit of study being undertaken by these students and levels of web use in other aspects of their life had not changed dramatically, we infer that skills were either enhanced through their participation in the unit or the level of self understanding or self esteem with respect to their computer, technical, information handling and communications skills had increased.

Students experienced initial problems associated with remote access, slow downloading of data and the congestion on the internet. During the semester, they suggested that access to a remote connection made the unit very flexible and indicated that early resolution of these difficulties would be most beneficial for future students. By the completion of the unit attitudes to the utility of the unit had also changed to the extent that they were enthusiastic, interested and keen to resolve any continuing and emerging technical problems and learn more. According to Gagne’s conditions of learning (Gagne, Briggs & Wager 1988), it is essential that students are motivated and interested in order for learning to take place. Students commented that they particularly enjoyed the combination of reading the text and interacting on the web. This stimulated them to think critically about their assignments relating to nursing theory and practice.

Experience negotiating the Web

Students described their initial lack of familiarity with the new learning environment in terms of their poor skills for using the web as an information resource. They had difficulty in finding the correct search words, not knowing where to start and not understanding ‘how it all worked’. This is contrasted with later responses which characterised a re-focusing of their use of the web site to seeing it in more of a communications role. Previous internet experience for the students comprised using the e-mail facility (4) or browsing the web for leisure (3). In the mid-semester questionnaire, students reported significant increased use of the NCR and to a lesser extent the e-mail facility. All reported being comfortable using the web for a range of activities such as literature searches. The students described a range of skill development and typified the Web as being interesting, exciting, a vehicle for debating topics, searching for information and a forum for enhanced interactivity among students and researchers.

Interactivity of the Web

As noted above, initially students did not characterise the web as a useful tool for teaching and learning. However, in the mid-semester questionnaire they commented on the usefulness of posting their publications to the web recognising the importance of both formal and informal feedback from their peers and tutors through written communication. At the end of the semester all students responded positively to the interactive nature of the learning environment and their ability to easily communicate with fellow students, tutors and others to enhance their learning.

Although the students became enthusiastic about using the site and believed it to be an appropriate mechanism to meet their learning needs, they felt that they had worked very hard to develop skills and attitudes to enable them to fully exploit the medium. It is possible that as students present to the university with more web experience and higher levels of existing skills, an optimal operational level for discussion will be achieved earlier in the semester. Discussions in the NCR were not perceived favourably by students initially, mainly because of their multiple or parallel nature. These problems of increased complexity of the multiple discussions with a number of strands active at one time is discussed in the literature and was experienced by both the students and the tutor. To overcome these initial difficulties, the students required clear explanations and regular messages from the tutor. This encouraged the students to actively participate. The other major difficulty perceived by students was the increased time needed to establish relationships with tutors and other students on-line. In line with what Eastmond (1995) learned from his study, our students also indicated that they found face-to-face sessions useful in being able to understand their student peers better as they were able to use visual cues not available through the on-line discussions.

Overall, in the final responses to the questionnaire, students rated the unit very highly. We believe that, contrary to early indications, they developed enthusiasm for the unit, particularly the nursing theory and research component. This could be related to some extent to the collegial relationships engendered through the more collaborative working environment.

Mechanisms for assessing student participation

While slower in the beginning, over a period of approximately three weeks, student activity in the NCR and TA increased dramatically, totalling 89 student contributions to the NCR and 21 to the TA page over the semester. As an indicator, this is an average of 6 contributions to each site over the 14 week semester. The fact that significant levels of interaction did not occur until week 3 or 4 further highlights this increase. To be added to this is the unknown amount of direct email interaction between students arising form the NCR and TA which was unmonitored. Illustration 3 provides an example of the NCR discussion list. The most obvious point is the visual representation of the discussion threads assisting the students to follow the multiple, non-linear discussions.

Illustration 3

The size of each contribution to the NCR varied from a few words to approximately five paragraphs, while the largest entry in the TA, which were student summaries of their readings was twelve paragraphs. Students noted that they often spent considerable time in re-drafting both types of contributions before uploading them to the site. This was substantiated by tutor observations and reflects the literature that in contrast to the impromptu nature of face-to-face tutorial interactions, the increased complexity of the asynchronous, on-line discussion with the perception of permanence, encouraged fewer, more reflective sophisticated responses from participants.

Given the time and effort apparently demanded by these activities, and as an incentive for student participation, 40% of the final mark was allocated for these activities. Considerable time was spent on developing a process to evaluate student participation. Maor (1998) pointing to a similar dilemma adds to it the problem of dealing with large numbers. Even with three students the problem of reviewing 89 contributions is not insignificant. The solution was to develop a web site which re-configured the discussion list (Illustration 4) so that contributions from each student were listed under the student’s name.

Illustration 4

This provides the tutor with the opportunity to count the number of contributions from each student and investigate the consistency over time. Further, by sampling the contributions the tutor may make judgments regarding the quality of the participation. It is clear that this is not a perfect assessment procedure. However, in an interview with the tutor, it was apparent that she felt more secure in making this assessment than she might have felt in the past in making similar assessments of student participation in face-to-face tutorials. Not only that, but should any disputation arise tangible evidence was available. Because the discussion re-configuration site is ‘created on the fly’, the tutor can take ‘snap shots’ of student participation during the semester in order to identify low participation students and take action to encourage them to participate more fully. Students were also supportive of receiving credit for their participation as most believed that they had worked more consistently and at a higher level in this environment, a perception echoed by the tutor. Whether or not it was related to the use of the web site, the students expressed support for a less subjective assessment process.

Conclusion

The effort to develop a robust pedagogy for electronic learning environments is still in its early stages. Data such as tutor and student perception and usage levels combined with professional, but still somewhat intuitive, responses help to focus continuing research and development on this goal within such a quickly changing environment. Our evaluation of the implementation of this teaching and learning site and the mechanisms for evaluating student participation finds support in the current literature. It also identifies the need for and evaluation mechanism to be easily used and as transparent as possible to gain the support of the students and tutors.

List of References

Berge, Z. and Collins, M. (1995) Computer mediated communication and the online classroom. (3 Volumes) Cresskill, NJ, Hampton Press.

Boyd, A., Chisholm, G., Dougiamus, M., Fox, R., Herrmann, A., Kevill, R & Thornton, D. (1996) Developing a CMCS for Technophobes. In C. McBeath and R. Atkinson (Ed) The learning superhighway: New world? new worries? Selected papers from the 3rd International Interactive Multimedia Symposium, 21 - 25 January 1996, Perth.

Burge, E. (1994) Learning in computer conferenced contexts: The learners’ perspectives. Journal of Distance Education IX, 1 pp (19 - 43).

Chickering, A. & Ehrman, S. (1996) Implementing the seven principles: Technology as a lever. AAHE Bulletin Vol. 49. No 2. October.

Eastmond. D. (1995) Alone but together: Adult distance study through computer mediated conferencing. Cresskill, NJ, Hampton Press.

Gagne, R., Briggs, L. and Wager, W. (1988). Principles of Instructional Design (3rd ed.). Florida, Holt, Rinehart and Winston Inc.

Kaye, T. (1990) Building the foundations for an electronic campus. In A. W. Bates (Ed), Media and technology in European distance education. (pp 227 - 234). Milton Keynes, The Open University for the European Associations of Distance Teaching Universities.

Maor, D. (1998) How does one evaluate students’ participation and interaction in an internet based unit? In B. Black and M Stanley (eds.) Teaching and Learning in Changing Times 7th Annual Teaching and Learning Forum, (pp 176 - 182) Perth, University of Western Australia.

Schwan, S. (1997) Media characteristics and knowledge acquisition in computer conferencing. Monograph, Tbingen, Deutsches Institut fr Fernstudienforschung.

[top]

Last modified on: Monday, 23-Jul-2007 10:55:56 EST