Impact Evaluation

Evaluating the Effects of Vocational Training in Africa

by Christian Kingombe

An attempt to quantify the quality of student bibliographies

by Anne Middleton

The Impact of a Feeder Road Project on Cash Crop Production in Zambia‟s Eastern Province between 1997 and 2002

by Christian Kingombe

Co-authored with Salvatore di Falco
IHEID Working Paper
CSAE Conference Paper

This paper investigates the dynamic impacts of rural road improvements on farm productivity and crop choices in... more This paper investigates the dynamic impacts of rural road improvements on farm productivity and crop choices in Zambia’s Eastern Province. There are several channels through which the feeder road improvements impact on farmers. Our aim is to estimate whether the differential outcomes in the five treatment districts and three control districts generated by the expansion of market agricultural activities among small to medium scale farmers could be explained by rural road improvements that took place after the new Chiluba MMD government in 1995 had completed an IMF rights accumulation programme bringing the principal marketing agent system to an end. Our district-level empirical analysis is an extension to the Brambilla and Porto(2005, 2007) cross-provincial level approach which proposes a dynamic approach accounting for entry and exit into the agricultural cotton sector to avoid biases in the estimates of aggregate productivity, when measuring productivity in agriculture applied to a repeated cross-sections of farm-level data from the Zambian post-harvest survey (PHS). Despite the limitations of the PHS data covering the period from 1996/1997 to 2001/2002 when the Eastern Province Feeder Road Project (EPFRP) was being implemented. The identification strategy relies on differences-in-differences of outcomes (i.e., cotton productivity) approach across two phases (pre-treatment and post-treatment). We use maize productivity to difference out unobserved household and aggregate agricultural year effects. Through our descriptive analysis we do find that changes in land allocation and in yields to Eastern Province’s most important cash crop – cotton did occur at the district level. However, it is difficult to conclude that these changes are linked directly to the improved accessibility obtained from the implementation of the EPFRP based on our differences-in-differences estimator or our Tobit model.

How to use programme theory to evaluate the effectiveness of schemes designed to improve the working environment in small businesses

by Kirsten Olsen

Abstract. Due to the many constraints that small businesses (SBs) face in meeting legislative requirements,... more Abstract. Due to the many constraints that small businesses (SBs) face in meeting legislative requirements, occu-pational health and safety (OSH) regulatory authorities and other OSH actors have developed programmes which can reach out to SBs and motivate and assist them in improving the working environment. A number of conceptual models help to enhance our understanding of OSH interventions in SBs and their effectiveness. However, they have mainly been evaluated on output rather than the process relating to the change theory underlying the intervention, and hence have seldom been rigorously evaluated. Thus little is known about how particular features of SBs can be transformed into national programmes. This paper shows how realist analysis and programme theory may be used as a framework for evaluating, developing and improving national intervention programmes for the improvement of the working environment and reducing injuries in SBs. It illustrates this for a specific New Zealand intervention: the Workplace Safety Discount scheme and its implementation in the agriculture sector. In practice, realist analysis should be performed during the planning, implementation and management stages so that ongoing findings can be fed back to the participant social actors to help them make appropriate changes to enhance the likelihood of success

Evaluación del impacto de la mina a cielo abierto de Cerredo (Degaña, Asturias) sobre los osos pardos

by César J. Pollo

Co-authored with J. Naves & A. Fernandez-Gil
Resumenes V Jornadas de la SECEM, Vitoria (España). SECEM, 2001

Report on the Social Audit Pilot 1998/99

by Garth Nowland-Foreman

COMMACT Aotearoa, Auckland (2000)

Causal models in program theory evaluation

by Patricia Rogers

Four key tasks in impact assessment of complex interventions

by Patricia Rogers

Keynote address. Workshop on Rethinking Impact. Understanding the Complexity of Poverty and Change, Consultative Group on International Agricultural Research (CGIAR) Cali-Colombia. Cali, Colombia. Retrieved 14 May 2009 from http://www.prgaprogram.org/riw/files/papers/Rogers%20material%20for%20workshop.ppt

Population and Building Factors That Impact Residential Fire Rates in Large U.S. Cities

by Texas State PA Applied Research Projects

Huang, Kai, "Population and Building Factors That Impact Residential Fire Rates in Large U.S. Cities" (2009). Applied Research Projects, Texas State University-San Marcos. Paper 287.
http://ecommons.txstate.edu/arp/287

WINNER OF THE 2009 MCGREW AWARD (student paper category)

The purpose of this explanatory research is to evaluate the impacts of population and building characteristics on the... more The purpose of this explanatory research is to evaluate the impacts of population and building characteristics on the residential fire rates among large cities in the continental U.S. This study used existing aggregated data from 69 U.S cities in the continental U.S to explain the effect of population and building factors on cities' residential fire rates. Overall findings indicate that two building characteristics - vacancy rate and building age - significantly influence residential fire rates. None of the population and natural environment variables examined found to significantly impact residential fire rates. Therefore, during the economic downturn, fire prevention efforts should focus more on areas with high vacancy rates and old building structures.

An Impact Evaluation of the Texas Department of Mental Health and Mental Retardation Performance Contract-Incentive Recoupment Program

by Texas State PA Applied Research Projects

Wilson, Sam C., "An Impact Evaluation of the Texas Department of Mental Health and Mental Retardation Performance Contract-Incentive Recoupment Program" (1999). Applied Research Projects, Texas State University-San Marcos. Paper 95.
http://ecommons.txstate.edu/arp/95

This paper evaluates the effectiveness of a government performance contract which utilizes incentives and penalties to... more This paper evaluates the effectiveness of a government performance contract which utilizes incentives and penalties to improve performance. In addition, it traces the evolution of privatization and contracting out of government services. It also discusses government contracting with other governmental entities and non profit organizations. The evaluation of the Texas Department of Mental Health and Mental Retardation Performance Contract is divided up into three hypotheses, all of which argue that a system of incentives and penalties improves contractor performance. The results generally support all three hypotheses.

An Impact Evaluation of the Texas Department of Mental Health and Mental Retardation Assertive Community Treatment (ACT) Program on Participant Use of State Hospitals

by Texas State PA Applied Research Projects

Kuhns, Melody Crain, "An Impact Evaluation of the Texas Department of Mental Health and Mental Retardation Assertive Community Treatment (ACT) Program on Participant Use of State Hospitals" (1998). Applied Research Projects, Texas State University-San Marcos. Paper 84.
http://ecommons.txstate.edu/arp/84

An Analysis of the Impact of the Texas Department of Public Safety Motorcycle Operator Training Program upon Motorcycle Related Traffic Accidents

by Texas State PA Applied Research Projects

Waller, Frankie Leon, "An Analysis of the Impact of the Texas Department of Public Safety Motorcycle Operator Training Program upon Motorcycle Related Traffic Accidents" (1992). Applied Research Projects, Texas State University-San Marcos. Paper 166.
http://ecommons.txstate.edu/arp/166

An Evaluation of the Economic Impact of the Sematech Research Consortium on the Economy of Austin, Texas

by Texas State PA Applied Research Projects

Garcia, Armando, "An Evaluation of the Economic Impact of the Sematech Research Consortium on the Economy of Austin, Texas" (1991). Applied Research Projects, Texas State University-San Marcos. Paper 234.
http://ecommons.txstate.edu/arp/234

An Evaluation of the Snack Tax on the Obesity Rate of Maine

by Texas State PA Applied Research Projects

Oaks, Brion, "An Evaluation of the Snack Tax on the Obesity Rate of Maine" (2005). Applied Research Projects, Texas State University-San Marcos. Paper 29.
http://ecommons.txstate.edu/arp/29

Obesity has reached epidemic proportions in the United States and is becoming one of our nation’s most discussed... more Obesity has reached epidemic proportions in the United States and is becoming one of our nation’s most discussed public health concerns. Americans are bombarded with advertisements for quick weight loss medications, diet plans, and novelty fitness equipment, all of which guarantee results overnight. But what impact has this had on the weight of Americans?

Traditionally viewed as an individual’s disease, research now indicates that the environment is a main causative agent of overweight/obesity. Manipulation of food pricing to encourage consumers to make healthy purchases is a new area of public policy that can help address this epidemic, and has received reasonable academic interest.

As a pricing strategy, healthy foods can be subsidized to increase consumption, or unhealthy foods can be taxed to increase prices in order to discourage consumption. The state of Maine had a snack tax from 1991 to 2001 and provides a setting to evaluate the impact of taxing snacks, pastries, and soft drinks on a state’s obesity rate.

According to the “law of demand”, individuals will reduce their consumption of an item if its price rises. A snack tax increases the price of snack foods and soft drinks. Because consumption of these foods is associated with increased weight gain, it is hypothesized that a statewide tax on snacks and soft drinks in Maine would have a negative impact on the state’s obesity rate.

This study utilizes an interrupted time series comparison group design to conduct a regression analysis with obesity data obtained from the Behavioral Risk Factor Surveillance Survey. This study seeks to examine the relationship between a snack tax and obesity rates.

Although the regression model for the interrupted time series design was significant, the findings did not provide any significant results for independent variables that could help identify and interpret a relationship between the snack tax and obesity rates for Maine.

Will the lessons be learned?: Reflections on local authority evaluations and the use of research evidence

by Jill Clark

This paper was published in the Journal: Evidence and Policy (2008) volume 4, Issue 2, Pages 255-268 and is co-authored by Jill Clark and Elaine Hall

Sure Start programmes are complex, community-based initiatives – fore-runners of the Children’s Centres Initiative -... more Sure Start programmes are complex, community-based initiatives – fore-runners of the Children’s Centres Initiative - which have been evaluated nationally and locally. Using an in-depth, retrospective case study of an evaluation of one local programme, the authors raise key issues pertinent to both practice and evaluation in the field, highlighting conflicts and dilemmas both within evaluation generally and, specifically relating to the evaluation of this programme. We illustrate the difficulties placed upon local evaluators by the lack of clear structures within which to work, and provide useful lessons as we move forward into the development and evaluations of new services for children and families.

Ghosts at the Feast: The role of research centres in supporting innovative practice in local authorities

by Jill Clark

Co-authored with Elaine Hall in the Journal: Studies in research: Evaluation, Impact and Training (2007) volume 2, p. 1-9.

This paper reports on the tensions of evaluation for project workers, managers and researchers. In the UK, a great... more This paper reports on the tensions of evaluation for project workers, managers and researchers. In the UK, a great deal of the innovative practice dedicated to improving the life chances of children and their families is developed by Local Authorities under the umbrella of project funding which includes an independent evaluation: apparently an ideal opportunity for researchers and practitioners to work collaboratively but, in our experience, there are key structural problems. We explore why it is that researchers fail to give formative feedback to local authorities and why they can feel that they are unwanted guests – ‘ghosts at the feast’.

Ghosts at the feast? The role of research centres in supporting innovative practice in local authorities

by Jill Clark

Conference Proceedings (inc. Abstract) at the BERA Annual conference
Conference, University of Warwick, 6-9 September, 2006

Co-authored with Hall, E, and McCaughey, C

A great deal of the innovative practice dedicated to improving the life chances of children and their families which... more A great deal of the innovative practice dedicated to improving the life chances of children and their families which is developed in Local Authorities is done so under the umbrella of project funding: Single Regeneration Budget programmes, New Deal for Communities, Sure Start. Researchers interested in issues of access and inclusion, the cultures of home and schools and the interactions between communities and services need to work closely with local authorities in what could hopefully be a mutually beneficial research-practice dialogue. The structure of project funding for local authorities demands that a certain proportion of the budget is always allocated to evaluation, which is to be conducted by independent researchers, usually affiliated to a HEI local to the area. This would appear to be the ideal opportunity for researchers and practitioners to work collaboratively but, in the experience of the authors, there are key structural problems which mean that this relationship is fraught with tensions. In this paper we offer examples drawn from over ten years experience of evaluating projects, programmes and initiatives across several local authorities, funded through variety of sources, such as LEAs, council, SRB (rounds 1-4) and Sure Start to explore why it is that researchers fail to give the formative feedback to local authorities and why they can feel that they are unwanted guests – ‘ghosts at the feast’. Successful locally driven projects, for example the Nuffield-funded speech and language development research (Mroz et al, 2002, Mroz and Hall 2003, Hall and Letts, 2003, Hall, 2005) manage to ground the research agenda in local needs, producing valuable outcomes for all parties, so what are the tensions particular to evaluation work? Firstly, there is a tension between the researchers’ desire to objectively evaluate what has gone on during the project, offset against the needs of the local authority (and the project) to have evidence for positive publicity and PR. This is exacerbated by high profiles and generous funding: for some projects it is not just a case of ‘cannot be seen to fail’ but of being a political ‘flagship’ and as such not even to damned with faint praise. Secondly, linked strongly to very short timescales and limited research funding, there is the tension between the desire of the researchers to give formative feedback and the danger of such evaluation becoming more summative – which is far from ideal for both the funders and the researchers. Our experience suggests that a key factor is the breadth of focus of the initiative: only when the goals of the intervention are clear and achievable can some of these tensions be ameliorated. The final tension, and perhaps most the most important one, is the desire of the researchers and practitioners to ‘shape’ funding agendas rather than just react to them. Currently, all parties involved are not able to set or shape this agenda, as many areas and projects react to particular funding sources. Consequently, all parties are working from a similar default position, and through a continuing default mechanism, which rarely creates a situation which genuinely promotes better and smarter ways of working with families and children.

Evaluation: From Public Programmes to Research’

by Philip McDermott

Co-authored with Zahbia Yousef in Duggan and Bush (eds.) "Evaluating Research in Violently Divided Societies" (IDRC, 2012 ‐ Forthcoming)

See
http://www.incore.ulst.ac.uk/cgi-bin/projects.pl?task=fullout&id=62

Measuring the Performance of Community Organisations In Queensland and New Zealand

by Kerry Brown

The network approach to evaluation: uncovering patterns, possibilities and pitfalls

by Kerry Brown