Guidelines for conducting and reporting reviews of reviews: Dealing with topic relevance and double counting
Poster presentation at the 2011 Cochrane Collaboration Colloquium.
Suggested Citation:
O’Mara AJ, Jamal F, Parry W, Lorenc T, Cooper C. (2011, Oct). Guidelines for conducting and reporting reviews of reviews: Dealing with topic relevance and double-counting. Presented at the 19th Cochrane Colloquium, 19-22 October 2011, Madrid, Spain.
Background:
Reviews of review-level evidence (tertiary reviews) are desirable when a research question is... more
Background:
Reviews of review-level evidence (tertiary reviews) are desirable when a research question is time-sensitive and/or the scope is broad. However, reviews included in a tertiary review often have only partial overlap with the tertiary review’s research question and, consequently, not all included studies are relevant to the tertiary review. Additionally, the reviews can include some of the same primary studies (known as double-counting). These concerns might lead to biases in the evidence base.
Objectives:
To explore the issues of relevance and double-counting in a tertiary review and to present guidelines for identifying and addressing potential related problems.
Methods:
We examined data from a completed systematic tertiary review on a public health effectiveness topic. We established the relevance of the included reviews by determining how many of the primary studies included in each review met our inclusion criteria and by analysing the included reviews’ synthesis sections using the PICO elements (population, intervention, comparison, outcome) in relation to the tertiary review research question. We demonstrate graphical (plot-based) approaches and a matrix-based approach to establishing the extent of double-counting.
Results:
Of twenty reviews that met our inclusion criteria, ten reviews had less than 50% of the primary studies included that were relevant to our research question. Exploring the synthesis sections of included reviews using a PICO framework was useful in establishing the degree of relevance of the findings to the tertiary review research question and yielded a review ‘utility’ rating. Our graphical and matrix-based approaches allowed us to evaluate the extent of double-counting across reviews; 14 primary studies were included in more than one review, with some studies appearing in four reviews.
Conclusions:
Issues of relevance and double-counting need to be assessed in tertiary reviews, but are often overlooked. The guidelines proposed can help identify potential biases and attempt to address them.
22 views
Seen by:Review of review-level evidence to inform the development of NICE public health guidance for the prevention of pre-diabetes among adults in high-risk groups
Suggested citation:
O’Mara, A. J., Marrero-Guillamón, I., Parry, W., Cooper, C., & Lorenc, T. (2010, Aug). Review of review-level evidence to inform the development of NICE public health guidance for the prevention of pre-diabetes among adults in high-risk groups. London: National Institute for Health and Clinical Excellence (NICE).
[Excerpt from the executive summary]
1.1 Introduction
This report presents the findings of a... more
[Excerpt from the executive summary]
1.1 Introduction
This report presents the findings of a systematic review of review-level evidence concerning the prevention of pre-diabetes (raised and impaired glucose levels) in populations/groups at higher risk (black and minority ethnic [BME] or low socioeconomic status [SES]) using community- and population-level interventions.
The primary research question for the review was:
What is known from review-level evidence about the effectiveness and cost-effectiveness of population- and community-level interventions to improve modifiable risk factors associated with pre-diabetes and type 2 diabetes among BME and low-income / low-SES groups?
The secondary research questions were:
What is known about promising ways to tailor interventions for diabetes risk factors to BME or low-income groups, for outcomes including improved BMI, physical activity levels, and blood pressure?
What are the barriers/facilitators to the effectiveness of interventions?
1.2 Methods
To locate evidence, a range of databases and websites indexing relevant literature were searched. Review reports were included if they:
1. were related to diabetes or pre-diabetes, obesity, healthy eating or dietary behaviour relevant to diabetes, or physical activity. Studies containing populations that already had diabetes were excluded;
2. were reviews of literature;
3. were published in 1999 or later;
4. were published in English;
5. included studies of adults (18-74 years);
6. included evaluations of interventions using any design (e.g. RCT, comparative trial, one-group);
7. had a focus on population- or community-level interventions; and
8. had a focus on either (i) low-SES or disadvantaged groups or (ii) any BME group relevant to the UK.
The quality of included reviews was assessed and data were extracted using adaptations of the standard tools for NICE public health evidence reviews of reviews.

