This report has been produced by the International Working Group set up by the Interacademies Panel (IAP) on International Issues. The Working Group was charged with developing a proposal for providing assistance with the evaluation of the implementation of Inquiry-Based Science Education (IBSE) programs for pre-secondary school students in different countries. At least 30 countries, both developing and developed, are known to be implementing some form of IBSE in some of their pre-secondary schools, creating a need for information about the impact of these programs on students and on teachers.

Aims and procedures

The aims of the Working Group are to provide guidance in relation to one or more of the following :

· the collection of evidence about the extent to which what is being implemented in classrooms matches the intentions of the IBSE curriculum, to feed back to the local program developers information as to how implementation could be improved;
· the development of instruments for assessing students’ learning in science that are suitable for assessing the outcome of IBSE;
· the design of evaluation projects that enable a comparison to be made between IBSE programs and conventional science education programs, or that enable assessment of students’ progress from year to year within an IBSE program;
· the design of research projects that aim to increase understanding of IBSE and which can lead to the refinement of teaching methods, material and professional development for IBSE.

The working group sought to achieve these aims through two meetings, held in Washington (March 2006) and Paris (June 2006), and through preparing and revising drafts of this report between meetings. Nine international experts, including scientists, science educators and program developers were asked to review an advanced draft of the report and their views have been taken into account in producing the final document.

The meaning of IBSE

The Working Group views IBSE as being not a single pedagogical method, but an approach having key features that can be implemented in various ways. IBSE shares some features with traditional science education but differs from it in many respects that go beyond the manipulation of materials to the key factor of engaging students in identifying relevant evidence, in critical and logical reasoning about it and in reflection on its interpretation. Some key distinguishing characteristics of IBSE are:

· Students are developing concepts that enable them to understand the scientific aspects of the world around them through their own thinking using critical and logical reasoning about evidence that they have gathered. This may involve them in first hand manipulation of objects and materials and observation of events; it may also involve them in using evidence gained from a range of information sources including books, the Internet, teachers and scientists.
· Teachers are leading students to develop the skills of inquiry and the understanding of science concepts through the students’ own activity and reasoning. This involves facilitating group work, argumentation, dialogue and debate, as well as providing for direct exploration of and experimentation with materials.

The Working Group has further spelled out these general statements in terms of the inquiry practices of the teachers, the inquiry experiences of the students and the outcomes of students’ learning of inquiry processes, science concepts, attitudes and dispositions.

Evaluation roles

Evaluation has several roles in the development and implementation of IBSE programs. The Working Group has focused its work on the roles that evaluation can take once a program has been developed, or has been adopted or adapted from a program developed elsewhere. The important roles of evaluation in the course of program development are not included, as this would extend the project beyond its capacity. It is recommended that a separate project be set up to serve these purposes.

Implementation of an already identified program passes through various stages along a dimension from early implementation, where only a small number of classes and schools may be involved and not all aspects intended may be in operation, to more advanced implementation, where the program has been well established in most respects in a larger number of schools. The processes and role of evaluation vary according to the point reached along this dimension of implementation. At early implementation stages the focus of evaluation is formative, to provide information about how the implementation can be improved. The evaluation is designed to address questions such as: Are the activities that the teacher and students are engaged in consistent with those intended in the program? Is the content which engages the teacher and students consistent with the intended content? Do the teacher-student interactions and the student-student interactions match those intended? Is the nature of the classroom discourse consistent with that envisaged in the program? At the early stages the focus needs to be on classroom practices rather than on student achievement; the assessment of student outcomes becomes worthwhile at the later stages, when there is evidence that the students have the experiences that can lead to intended learning. Thus, at these later stages, in addition to evidence about how teachers use the IBSE materials and students’ experiences, data are collected on students’ inquiry skills, knowledge and understanding of science concepts and attitudes.

Evaluation procedures

When the purpose of the evaluation is to provide and feed back information about the early implementation of a program, the data collected will come from close observation of the teacher and students, from interviews with and questionnaires for teachers, administrators and students, and from review of teachers’ journals and students’ notebooks.

At later stages of implementation the evaluation will provide summative information on students’ learning of inquiry skills, science concepts and attitudes, in addition to information about classroom processes. The design of the evaluation will depend on the questions to be addressed. In some cases these will be about the extent to which the desired outcomes as specified in the program are realized. In other cases the questions will focus on whether student outcomes are the same or different (better, worse) when the IBSE program is compared with other science education programs. Addressing questions of IBSE compared with other programs requires careful evaluation design, choice of IBSE and of comparison classes and selection and creation of instruments. Where possible, existing assessment instruments ought to be used, supplemented by assessment items that require the specific skills and understanding that are outcomes of IBSE. Examples of evaluation instruments are given in appendix E to this report.

Reporting and feedback procedures

Countries participating in this project will be assisted in producing reports at different levels of detail for different audiences. Where appropriate, local and international evaluators will hold seminars to provide face-to-face feedback to different users of the results. However, the purpose of the international context is to add value to each individual evaluation study by sharing procedures and findings across countries. Thus, in addition to its unique value to the country in which the evaluation was conducted, the findings will contribute to a wider understanding of the effects of IBSE in different circumstances.

It is clearly understood that collection of reports from different countries will not be used to rank or compare countries in terms of the quality of IBSE programs nor of the performance of their students. The development of synoptic reports, bringing together findings from different countries will only be used to extend the vision of each participating country as science education continues to be developed internationally as well as permitting the accumulation of evidence on the effects of IBSE across contexts and countries.

Operation of the international evaluation project

It is proposed that the operation of the international project will be overseen by an International Oversight Committee (IOC) of about four or five members. The IOC will initially draw up a list of international experts in IBSE evaluation and research from which members are recruited for International Evaluation Teams (IET) to support particular evaluation projects. Countries wishing to participate in the international project will approach the IOC and, with its help, produce a proposal indicating the purpose of and the potential design of the evaluation study. The IOC will then identify, from the list of experts, an IET whose members will support the local evaluation team in planning, carrying out and reporting the evaluation. Reports will be published by each country and, in addition, will be available to the IOC for the purpose of producing a synthesis of reports from different countries which will then be available to all.

Recommendations

In relation to the international evaluation project, the Working Group recommends that:

· Countries implementing IBSE programs should evaluate the extent and effect of implementation and set up local evaluation teams for this purpose.
· An International Oversight Committee be set up to manage the provision of support for implementation evaluation to countries who request such support.
· International Evaluation Teams work with local evaluation teams to support the evaluation and enhance local capacity to conduct it.
· The reports of procedures and findings of evaluations conducted in various countries are made available to others and are synthesised by the IOC to add to general understanding of IBSE and its implementation in various contexts.

It is further recommended that the IOC should:

· Develop a handbook to guide evaluation design and procedures and a compendium of pertinent research literature for the evaluation IBSE.
· Hold an international workshop to discuss the outcomes of the synthesis of reports from different countries at a time when sufficient data have been accrued.
· Establish a program of post graduate fellowships that will support capacity building in countries where there may be a lack of evaluation and research expertise.

Finally, the Working Group identifies the following work outside this evaluation project that it recommends the IAP to consider in relation to IBSE:

· A project to provide help for countries in the design and development or adaptation of IBSE programs.
· The creation of guidelines aimed at enabling countries to generate for inclusion in their IBSE programs, materials that provide help for teachers in assessing their students and evaluating their own effectiveness in implementing IBSE.
· The promotion of research projects aimed at increasing knowledge and understanding of the nature of inquiry learning in science and how it is best implemented in various contexts.


To read the complete version in PDF, click here