Pilot projects for evidence-based policy-making: three pilot projects in the Rhine Basin.

Author:Vreugdenhil, Heleen

1 Introduction

Pilot projects are commonly applied instruments in diverse policy domains, including water management. They provide a space to introduce and test innovations without large risks. Risks are reduced by confining the geographical scale or the duration of the intervention. Additionally, 'failure' of the innovation is somewhat more tolerated. These conditions enable actors to participate more easily (Vreugdenhil et al. 2010).

In Evidence Based Policy Making (EBPM) also, pilot projects have been recognized as important tools to develop 'evidence' of policy innovations (Cabinet Office 1999). Although there is no unanimous definition of Evidence Based Policy Making (EBPM) it is claimed to (Cabinet office 1999; HM Treasury GSRU 2007):

* Improve understanding of an issue and its associated actors

* Develop solution in collecting evidences and shared work

* Test solutions and communication practices

* Enrich understanding of contexts (local, political, social, technological, economical ...).

Furthermore, EBPM is a dynamic concept that applies at different stages of policy development: initiation, implementation, defense and justification, evaluation and readjustment

In our understanding, the role of pilot projects in realization EBPM consists of eliciting grounded knowledge, identifying what works, and testing policies. The evidence developed in pilot projects on the functioning of the policy innovation in a practical setting could be used to inform policy-making, which can be improved based on the evidence (Pawson and Tilley 1997; Cabinet Office 2003). In other words, pilot projects can serve realizing EBPM by guiding policy-making with knowledge of what is appropriate for specific problems, which is what EBPM is about (Sanderson 2003).

The importance placed on pilot projects is increasing due to the increasing technological and societal complexity of contemporary public policy making (Cabinet Office 1999; Ker Rault 2008), the loss of confidence of the public in many professions and public bodies, and the requirement of national governments to make use of pilot projects to find out whether policies do or do not work as intended (Solesbury 2001; Cabinet Office 2003; HM Treasury GSRU 2007). For researchers it provides a tool to improve innovations and to cooperate with societal actors and to collect financial resources. Practitioners can find out whether certain innovations make management more efficient and are thus worth to be applied at a larger scale. The number of pilot projects initiated is difficult to estimate, because both policy and nonpolicy actors in any policy domain can initiate a pilot project. However, to give an idea of pilot projects initiated by practitioners, the Dutch water management authority has conducted about 40 pilot projects in river and coastal management over the last 5 years.

Despite the prosperous expectations and enthusiasm with which pilots are initiated, results are often considered disappointing (Sanderson 2002; De Groen et al. 2004). Policy-makers seem to 'abuse' pilot projects as alibi to show supremacy of their personal ideas, rather than using them to learn from. Particularly in a narrow view of EBPM addressing the issue of 'what works under which circumstances', pilot projects are therefore not uncontested. Additionally, despite their wide use, pilot projects have received little research attention (Huitema et al. 2009), especially in a trans-disciplinary domain such as water management.

In this paper, we theoretically and empirically study pilot projects to deepen understanding of how they can contribute to EBPM and which limitations and problems may arise when realizing EBPM. Let us note that pilot projects can also be used for many other purposes, but for an analysis thereof we refer to Vreugdenhil et al. (2010). The research questions addressed in this paper are the following:

* Which types of pilot projects contribute to the realization of EBPM?

* Which factors and mechanisms influence the realization of EBPM through pilot projects?

In attempting to address these research questions and following the presentation of the methods in section 2, we develop a framework of analysis (section 3). We have identified three types of pilot projects, all of which function differently for EPBM. We discuss their characteristics and effects to policy and management. Next, we study three pilot projects in the Rhine Basin on their use, characteristics and effects (section 4). We identify to what extent those pilot projects contributed to EBPM and we discuss factors and mechanisms influencing the realization of EBPM (section 5). Last we provide some recommendations for enhancing their impact on policy-making (section 6).

2 Methods

The analysis of pilot projects in relation to the realization of EBPM for this paper consists of three major steps.

First, elaborating on a desk-based review of 16 out of 40 pilot projects conducted by the Dutch water management authority in river and coastal management over the last five years, we develop an analytical framework of pilot projects (Vreugdenhil et al. 2010). We identify three types of pilot projects: research, managerial and political-entrepreneurial pilot projects. We conceptualize the types of effects a pilot project can have in terms of biophysical and actor-network responses, knowledge development and diffusion. Bearing in mind the importance of 'what works when' objective of EBPM (Solesbury 2001), we focus mainly on diffusion of knowledge gained in pilot projects. By understanding diffusion one might gain insights in mechanisms associated to the use of developed knowledge as evidence for policy-making.

Second, we apply the framework to three case studies in the Rhine Basin. The generic policy and economic contexts of the case studies is comparable. Indeed, they are all part of a long history in formal international cooperation of nearly two centuries along the Rhine. Recently, the international cooperation has been reinforced by the European Water Framework Directive, requiring countries sharing a river basin to further enhance cooperation and achieving comparable quality standards. The case studies are all pilot projects dealing with floodplain revitalization, aiming to combine ecological enhancement with the dominant purpose of that floodplain i.e. flood defense or drinking water production. The pilot projects have all been conducted in the public sphere in multi-actor contexts. Actors include universities, research institutes, public agencies, NGOs and citizens, representing diverse disciplines and stakes ranging from hydraulics and ecology to social geography. Important to the case study selection was also that the pilot projects achieved a broad range of effects. Lastly, accessibility to information provided an additional practical reason to select the three pilot projects.

The first case study, 'Altenheim' in Germany, was chosen to provide an example of a finalized pilot project to enable the evaluation of longer-term effects of a pilot project. The pilot project was initiated from the 'classical' EBPM point of view, meaning that this was the first application of a policy program and results were expected to provide the evidence of whether the innovation worked or not (Pawson and Tilley 1997, Cabinet Office 2003). The case study illustrates that despite the confirmation of the policy program on the evidence from the pilot project, implementation remains difficult. The second case study, 'Beuningen' in the Netherlands, illustrates that evidence can find its way into policymaking even when a pilot is not fully finalized yet. The third case study, 'Basel' in Switzerland, provided an example of how a pilot project can harm relationships and so limit the realization of EBPM (Vreugdenhil et al. 2009, Vreugdenhil and Slinger 2009, Vreugdenhil et al. 2010).

Data for the case studies are collected through (i) participation in the Dutch case study in 2004 by the first author, (ii) document analysis and (iii) 24 interviews between 2006 and 2008 with river managers, scientists, citizens' representatives, floodplain managers and drinking water producers. The participation consisted of contributing to designing interventions and modeling their hydraulic impact. Additionally, related studies on the influence of scale perceptions on the innovation were performed (Vreugdenhil et al. 2010b). Attendance to project meetings, discussions and workshops was very valuable to gain inside understanding of pilot projects' dynamics. Interviews focused on the history of the project, on actors' perceptions on river management, on knowledge development and on the diffusion of the project into policy-making.

Third, we elicit factors and mechanisms of importance in the realization of EBPM from the pilot projects. We discuss which characteristics of pilot project may influence EBPM. Of particular interest is the role of perceptions. Based on the insights in influential factors, we propose some strategies for pilot project initiators to increase the effectiveness of their pilot project for EBPM.

3 A framework to analyze pilot projects

We first present three types of pilot project based on the intention of their initiator. Then, we refine the analytical framework with six characteristics and potential effects on management and policy-making.

3.1 Three pilot project types

Head (2008) argues that three lenses of evidence-based policy can be distinguished. These include scientific research, political judgment and professional practices. Similarly, we identify that pilot projects can be used for three different purposes in EBPM. Researchers, practitioners and politicians can be all involved in initiating a pilot project. The three pilot project types include:

* Research Pilot Project

* Managerial Pilot Project

* Political-Entrepreneurial Pilot Project

Research pilot projects are primarily focused on the development of knowledge about the innovation...

To continue reading