The systematic evaluation methodology adapted is considered to be a very important parameter of the e-Hermes project since only a few ODL projects have been systematically evaluated (Savas, 1998). The monitoring strategy includes on field observation and two questionnaires for the students, before (pre-test) and after (post-test) the implementation in the classroom. A two-group design (experimental and control group) had been selected. The data from a control group (reference group) from each country were included in the analysis in order to rule out effects of normal development and ageing, which would be interpreted as project outcome misleading the conclusions. The members of the two groups were selected from the same population. 183 students filled out questionnaires. 108 of them participated to the project (experimental group). The average age of the students was about 14 years. The research process adapted is presented schematically in figure 1.

Research problem
Need to familiarize high-school students of co-operation with students abroad. Introduction of new subjects of international interest to the curriculum
Exploratory question
Could ODL offer a qualitative upgrade to everyday teaching, acting as a supplement to conventional teaching?
Procedure
Development of a pedagogical framework suitable for ODL's implementation in secondary education
Sample
High school students from 3 European schools
Data analysis
Test design Use of statistical techniques (Mann-Whitney U-test, x2 test, Friedman -test)

Figure 1: The research process in the e-Hermes project.

Quantitative results of the project are obtained by numerical methods. Many statistical tests were carried out in order the development of understanding, knowledge and attitude towards the use of new tools as communication channels, to be observed. Many comparisons between the students' groups from the three countries (Austria, Germany, Greece) were also performed. Differences between pre-test and post-test were analysed on the basis of regression-transformed* differences between pre- and post-test (Hays, 1973). For the standardised residuals of the linear regression between pre-test and post-test over all subjects comparison between experimental and control group has been performed with Mann-Whitney's U-test (Hays, 1973).

 

Figure 2: Relative changes of experimental group and control group between pre-test and post-test for the students' answers to questions about the "usage of Internet in the classroom" and their "interest in foreign subjects".

In figure 2 the relative changes of the experimental and the control group between the pre- and the post-test for the students' answers to questions about the "usage of Internet in the classroom" and their "interest in foreign subjects" are presented. A detailed presentation of the results of the quantitative analysis of the project is presented in (Apostolakis et al., 1999). In parallel qualitative results have been drawn. The quantitative analysis of the project shows to which extend the students succeeded in playing the roles chosen for them during the project's implementation. The main results of this analysis are based on the teachers' assessment of the students during the implementation of the project. In order to evaluate teachers' conclusions a series of "quality control" factors were used. A very interesting example is presented in figure 3. The "quality control" factor is the number of contributions to the two Bulletin Boards of the project. The first one is related to discussions of general interest. The second one was devoted to discussions on the measured quantities. It was a general filling during the first months of project's implementation in the classroom that students were not so interested in participating in discussions through the Bulletin Boards. In order to activate students the idea of Videoconference was introduced the last week of November 1998. Students first had to discuss their ideas during a videoconference and then make their remarks and suggestions for the possible answers to the questions presented in the Bulletin Board. As it can be seen from the histogram there is an important rise in the number of contributions to the scientific Bulletin Board after the application of videoconference, while the number of contributions to the other Bulletin Board is more or less constant during the whole running period of the project.

Figure 3: Determination of students' behavior towards the project's implementation. Columns S1 represent the number of students' contributions to the scientific Bulletin Board (answers to the question of the month) while columns S2 represent the contributions to the Bulletin Board where general topics were discussed. Please note the significant increase in the number of contributions after the introduction of videoconference in the beginning of December.

The implementation of the project thus far shows that it offers significant advantages in the following aspects of everyday teaching:

  1. Positive attitude of the students to the didactical intervention: the enthusiastic participation of the students to the project gives clear evidence that they find the idea of communicating and collaborating with students from various countries very challenging. Since it is commonly accepted that positive attitude is a key element to the cognitive progress (Nachtigall, 1991; Todt, 1993) this aspect is considered as a very significant advantage of the implemented approach. Furthermore it has to be mentioned here that to students having a PC at home a clear reorientation towards a didactically meaningful application of their PC (sending notes and comments to the Bulletin Board, observing plots etc) has been noticed.

  2. On the cognitive level students working in the project significantly gained more insight into measurements procedures, starting from simple aspects such as how to reduce reading errors at analog instruments or how often to measure in order to ensure accuracy, to more complexes such as elimination of systematical errors. Students also showed an increased ability to read, interpretate and analyze plots, identify quantities that correlate with each other, approach elements of error theory and understand scales, averages, etc. It has to be mentioned here that although all students had worked with plots in the class, they seemed to be familiar only to specific plot forms such as linear, exponential etc. Most of the students found the idea of working with non-linear "real" plots, challenging.

  3. Students undoubtedly presented an increased familiarization with the methodology of scientific inquiry.

  4. Finally an important side effect is the fact that through the work at environmental issues students get to know each other and thus cultural elements are transmitted. European state friendship can only be forwarded in a person to person basis. This can best be achieved when people from different nations work together with common aims and objectives.

*Linear regression means the determination of the straight line that is assumed to represent best a linear correlation between two or more specific variables. A residual of this linear regression is the distance of one value to this straight line. Calculating the residuals of the linear regression between the pre-test and the post-test, leads to level-independent measurements for existing differences between the two tests. In contrast to simple subtractions (post-test minus pre-test) these values reveal differences that are independent from the initial value. If, for example, a person marks a high score in the pre-test, he will likely mark a lower score in the post-test (statistical regression effect). Especially comparisons between different groups (country, gender, etc.) could then lead to incorrect results.

 

Comments, remarks or questions to: admin@ellinogermaniki.gr