Critical friend approach: policy evaluation between methodological soundness, practical relevance, and transparency of the evaluation process.
| Verfasser | Balthasar, Andreas |
1 Introduction
Evaluation is marked by a great diversity of approaches and methods. In Evaluation Theory, Models, and Applications, Stufflebeam and Shinkfield (2007) distinguished no fewer than 26 often-used evaluation models (see also House, 1987). In this paper, the "critical friend" approach is presented as a new and promising approach. Despite the great variety of evaluation approaches in practice, there are elements that characterize the current use of the term "evaluation": First, evaluation is regularly understood as assessment (Stockmann, 2000, p. 12; Clarke & Dawson, 1999; Scriven, 1991). Rossi and Freeman (1993, p. 5) therefore defined evaluation as "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs." A second, central element of evaluation underlines its basis in social sciences methodology. This is apparent in Rossi and Freeman's definition of the term as well. The third constitutive element is its practical relevance: Vedung (1999, p. 12) stated that the general practical orientation is so central that it must be a part of the definition of evaluation.
In this paper, we will first present the core aspects of three typical approaches representing methodological soundness, practical relevance, or transparency of the assessment process. We will then illustrate our "critical friend" approach based on the example of the evaluation of the child care voucher project in the city of Lucerne. The critical friend approach tries to find an optimal balance between methodological soundness, practical relevance, and transparency of the assessment process. In addition, it accounts for empirical findings showing that the use of evaluations in Switzerland depends decisively on the interest of the decision makers in the results of the study (Balthasar, 2007). At the end of the paper we will discuss possibilities and limitations of the critical friend approach in policy evaluation.
2 Diversity of evaluation approaches
As described above methodological soundness, practical relevance, and transparency of the assessment process can be considered as the three corners of the triangle in which most evaluation approaches can be positioned. The following sections sketch out core aspects of each of the three elements by presenting one typical approach for each element.
2.1 Methodological soundness: Donald T. Campbell
In the 1960s, the criticism that common evaluation practice was not scientific led Donald T. Campbell to demand scientific soundness when planning the evaluation design. Without a doubt, Campbell and Stanley's (1963) Experimental and Quasi-Experimental Designs for Research and Campbell's (1969) paper, "Reforms as Experiments," were important foundations for the development of evaluation methodology. Campbell's ideal of experimentally designed program evaluation continues to shape evaluation science in part up to today (Stufflebeam & Shinkfield, 2007). Similar to Campbell, also Chen and Rossi (1980; Chen, 1990) worked towards scientifically based evaluation research and drew up the central ideas of theory-driven evaluation. As stated in Rossi, Freeman, and Lipsey (1999):
Every program embodies a conception of the structure, functions, and procedures appropriate to attain its goals. This conception constitutes the 'logic' or plan of the program, which we have called program theory. The program theory explains why the program does what it does and provides the rationale for expecting that doing things that way will achieve the desired results. (p. 156) Methodologically oriented evaluation theories view evaluators foremost as independent specialists in methodological questions. Using clever and scientifically sound design, they provide firm answers to the question as to whether certain effects can be casually attributed to evaluated measures. In this tradition, evaluators are meant to mainly show the effects of programs. But they are supposed to abstain from assessing the results, so as not to jeopardize their independence.
2.2 Practical relevance: Michael Patton
There are excellent grounds for viewing Michael Patton as the father of evaluation approaches which focus on the practical relevance of evaluation results (Stockmann & Meyer, 2010, p. 120). Patton's "utilization-focused evaluation approach" aims to conduct evaluations in such a way that their results are used for policy practice: "The focus in utilization-focused evaluation is on intended use by intended users" (Patton, 1997, p. 20; see also Patton, 2000). For this reason, Patton places importance on having the users of an evaluation participate in important decision making from the very start of the evaluation process. The actual conducting of an evaluation is the responsibility of the evaluators, however, whose credibility and integrity depend on the quality of their handling of this communication task. In this regard, Patton assigns evaluators five functions (Patton, 1988, p. 15 ff.): First, they must help those involved overcome their fears concerning the evaluation. Second, they must make sure that the right questions are asked. Third, their task is to further develop the evaluation process taking into account and adapted to situational factors. Fourth, they must constantly keep in mind what has been done previously and draw conclusions as to how the evaluated program can be improved. Finally, evaluators must be advocates of the evaluation even when the process is at a critical point.
2.3 Transparent evaluation process: Michael Scriven
Evaluations always represent evaluative judgments (Stockmann & Meyer, 2010, p. 17). The assessments made in the context of an evaluation become comprehensible only when the criteria being used are disclosed. Transparent evaluation criteria and methods are therefore an essential determining feature of evaluations. It was especially Michael Scriven who studied this topic: "Evaluation is the process of determining the merit, worth and value of things, and evaluations are the products of that process" (Scriven 1991, p. 1). Scriven was unequivocal in his position that society requires valuing and that it is the role of the evaluator to do that job: "Bad is bad and good is good and it is the job of evaluators to decide which is which" (Scriven, 1986, p. 19). Scriven maintained that there is a science of valuing, and that it is evaluation. He acknowledged that "no single scientist ever observes 'reality' completely and undistorted, but rather observes through biased perceptual filters Using multiple perspectives helps construct a more complete picture of reality" (Shadish, Cook, & Leviton, 1991, p. 76). Using the perspectivism, or multiple accounts, that Scriven recommends for program evaluation corresponds to what in methodology is called triangulation (see section 3.4.1 below). (1)
3 The "critical friend" approach: Theory and practice
Methodological soundness, practical relevance, and a transparent evaluation process can thus be viewed as the three main elements of today's common understanding of evaluation. The problem is to find the optimal balance between these three challenges. Our own empirical findings showed that when scientific soundness is not questioned, the use of evaluations in Switzerland depends decisively on the interest of the program managers and decision makers in the results of the study (Balthasar, 2007). Taking into consideration this empirical evidence, we developed the "critical friend" approach, which tries to find an optimal balance between the challenges of evaluation theory and practice and the fact that the utilization of evaluations depends decisively on the decision makers' interest in the results.
The evaluation process is designed differently depending on the evaluation approach. This means that every evaluation approach yields different responses to the central questions that arise during the conducting of an evaluation. This is also true for the critical friend approach. For this reason, in this section the evaluation process is described along these questions. In the following, the critical friend approach will be outlined in general and then illustrated by taking the example of the evaluation of the child care voucher pilot project in the city of Lucerne. This evaluation was undertaken in the fall of 2008 and is ongoing until 2012. It began shortly prior to the launching of the pilot project in Lucerne and will be completed only after the pilot project ends.
3.1 First question: What questions should the evaluation answer?
3.1.1 The idea....
Every evaluation starts with politically relevant questions. In a first step, these questions must be stated more precisely as scientifically addressable evaluation questions. To do that, it is helpful to develop a "logic model" as proposed by Chen and Rossi (Chen & Rossi, 1987; Rossi et al., 1999). A logic model links impacts with program inputs and processes in a more or less linear way. The logic model facilitates thinking, planning, and communications about the objectives and actual accomplishments of a program. The term logic model is frequently used interchangeably with the term "program theory" in the evaluation field. Logic models can alternatively be referred to as theory, because they describe how a program works and to what end. Basically, a logic model is "a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve" (W.K. Kellogg Foundation, 2004, p. 1). A logic model is used to state the evaluation questions more precisely, to define the evaluation objects, and to identify the factors that may foster or hinder implementation and the effects. The logic model may be developed by those involved, by target groups, or by the evaluators of a program, depending on...
Um weiterzulesen
FORDERN SIE IHR PROBEABO ANCOPYRIGHT GALE, Cengage Learning. All rights reserved.
