Providing evidence to improve practice

What is What Works in Conservation?

What Works in Conservation

What Works in Conservation provides expert assessment of the effectiveness of interventions based on the summarised evidence in synopses. These assessments are available in the searchable database, and the full publication can be downloaded as a pdf or purchased from OpenBook Publishers.

Panels of experts have assessed the collated evidence for each intervention to determine its effectiveness, the certainty of the evidence and, in most cases, whether there are negative side-effects on the group of species or habitat of concern (harms). Using these assessments, interventions have been categorised, based on a combination of effectiveness (the size of benefit or harm) and certainty (the strength of the evidence). The following categories are used: Beneficial, Likely to be beneficial, Trade-off between benefit and harms, Unknown effectiveness, Unlikely to be beneficial, Likely to be ineffective or harmful.

What Works in Conservation 2017 provides assessments of the evidence in the following synopses: Amphibian Conservation, Bat Conservation, Bird Conservation, Farmland Conservation, Forest Conservation, Natural Pest Control, Soil Fertility and Control of Freshwater Invasive Species. It will be updated annually to include new evidence and cover additional taxa and habitats.

 Using the assessments

It is important to remember that Conservation Evidence provides a summary of the evidence available for different conservation interventions, which should act as a starting point in deciding whether or not to carry out an intervention. The assessments provided by What Works in Conservation are based on the available evidence for the target group of species or habitat for each intervention. The assessment may, therefore, refer to different species or habitat to the one(s) you are considering. Before making any decisions about implementing interventions it is vital that you read the more detailed accounts of the evidence (both the action pages and the individual studies) in order to assess their relevance for your study species or system.

There may also be significant negative side-effects on the target groups or other species or communities that have not been identified in the assessments.

A lack of evidence means that we have been unable to assess whether or not an intervention is effective or has any harmful impacts.

How is the evidence assessed?

The average of several experts’ opinions is likely to be a more reliable and accurate assessment than the opinion of a single expert. We therefore ask a panel of experts to use their judgement to assess whether evidence within a synopsis indicates that an intervention is effective or not. They are also asked to assess how certain they are of the effectiveness given the quality of evidence available for that intervention (certainty of the evidence). Negative side-effects described in the collated evidence are also assessed for the species group or habitat of concern (harms). Experts base their assessment solely on the evidence in the synopsis.

We use a modified Delphi method to quantify the effectiveness and certainty of evidence of each intervention, based on the summarized evidence. The Delphi method is a structured process that involves asking a panel of experts to state their individual opinion on a subject by scoring anonymously. They can then revise their own scores after seeing a summary of scores and comments from the rest of the panel. Final scores are then collated. Scores and comments are kept anonymous throughout the process so that participants are not overly influenced by any single member of the panel.

For each intervention, experts are asked to read the summarized evidence in the synopsis and then score to indicate their assessment of the following:

Effectiveness: 0 = no effect, 100% = always effective.

Certainty of the evidence: 0 = no evidence, 100% = high quality evidence; complete certainty. This is certainty of effectiveness of intervention, not of harms.

Harms: 0 = none, 100% = major negative side-effects to the group of species/habitat of concern.

The median score from all the experts’ assessments is calculated for the effectiveness, certainty and harms for each intervention. Effectiveness categorization is based on these median values, i.e. on a combination of the size of the benefit and harm and the strength of the evidence.

How are actions categorized by effectiveness?

After one or two rounds of initial scoring, interventions are categorized by their effectiveness, as assessed by the expert panel. The median score from all the experts’ assessments is calculated for the effectiveness, certainty and harms for each intervention. Categorization is based on these median values, i.e. on a combination of the size of the benefit and harm and the strength of the evidence. The categories and their associated scores are listed in the table below. There is an important distinction between lack of benefit and lack of evidence of benefit.

 

Categories

Effectiveness   score

Certainty   score

Harms score

Beneficial

>60

>60

<20

Likely to be beneficial

OR

>60

40 - 60

40 – 60

≥40

<20

<20

Trade-offs between benefits & harms

≥40

≥40

≥20

Unknown effectiveness

Any score

<40

Any score

Unlikely to be beneficial

<40

40-60

<20

Likely to be ineffective or harmful

OR

<40

<40

>60

≥40

Any score

≥20

 

Once interventions are categorized, experts are given the chance to object if they believe an intervention has been categorized incorrectly. Interventions that receive a specified number (depending on the size of the panel) of strong objections from experts are re-scored by the expert panel and re-categorized accordingly.

Why do some interventions not have an assessment of effectiveness or harms?

Although most interventions and synopses have been assessed to produce What Works in Conservation, some sections (notably the Bee Conservation synopsis) have not yet been assessed by an expert panel. These sections will be assessed soon.

Some interventions in the Farmland Conservation and Bird Conservation synopses have assessments of effectiveness and certainty, but not harms. This is because these assessments were carried out as part of different projects. Details of the assessment for interventions to deal with Invasive alien and other problematic species in the Bird Conservation synopsis are given in Walsh J.C., Dicks L.V. & Sutherland W.J. (2015) The effect of scientific evidence on conservation practitioners’ management decisions. Conservation Biology, 29, 88-98. The assessment methodology used on interventions in the Farmland Conservation synopsis is provided in Dicks L.V., Hodge I., Randall N.P., Scharlemann J.P.W., Siriwardena G.M., Smith H.G., Smith R.K. & Sutherland W.J. (2014) A Transparent Process for "Evidence-Informed" Policy Making. Conservation Letters, 7, 119-125.