Program Evaluation

ADB: Disabled People and Development

by Lorna Jean Edmonds

Asian Development Bank (2005)

The Disabled People and Development describes the evolution of the global response to disability as well as the... more The Disabled People and Development describes the evolution of the global response to disability as well as the concepts and tools for addressing the needs of disabled people

Devil in the detail: using a pupil questionnaire survey in an evaluation of out-of-school classes for gifted and talented children

by Mike Lambert

Education 3-13, 36(1) February 2008, 69-78.

The use of questionnaires to evaluate educational initiatives is widespread, but often problematic. This paper... more The use of questionnaires to evaluate educational initiatives is widespread, but often problematic. This paper examines four aspects of an evaluation survey carried out with very able pupils attending out-of-school classes: ethics, design, bias and interpretation. There is a particular focus on the interpretation and analysis of pupils’ answers to open questions. Conclusions are drawn from this analysis which will help teachers and others to take a careful and critical approach to their use of questionnaires in educational evaluation.

Author profile: http://www.wlv.ac.uk/default.aspx?page=13209

[A limited number of eprints of this article are available - send me a message if you require one].

Transitions of Truants: Community Truancy Board as a Turning Point in the Lives of Adolescents

by Kevin Wright

Johnson, Charles L, Kevin A. Wright, and Paul S. Strand (2012). "Transitions of Truants: Community Truancy Board as a Turning Point in the Lives of Adolescents." Journal of Juvenile Justice, 1(2), 34-51.

School dropout represents a major turning point in a person’s life that could be seen as an initial step on a... more School dropout represents a major turning point in a person’s life that could be seen as an initial step on a difficult pathway to reduced conventional opportunities. The challenge is to identify interventions that can successfully reintegrate students back into a school setting in a manner that encourages continued attendance and involvement. One such program is the West Valley Community Truancy Board in Spokane, Washington. In addition to the truancy board process, the program employs a court-appointed officer to mentor students and manage the overall process of identifying and attending to the risks and needs that promote truancy. Guided by Sampson and Laub’s (1993) age-graded theory of informal social control and Cullen’s (1994) application of social support to delinquency, the current analysis seeks to determine the overall effectiveness of the truancy board based on both quantitative analyses of outcomes and qualitative interviews with key actors. We discuss the implications for the ongoing theoretical, empirical, and policy debates surrounding truancy intervention.

The limits of the new public diplomacy: Strategic communication and evaluation at the US State Department, Foreign & Commonwealth Office, British …

by James Pamment

PhD thesis. Department of Journalism, Media & Communication (JMK), Stockholm University

The new public diplomacy is a major paradigm shift in international political communication.
Globalisation and a... more The new public diplomacy is a major paradigm shift in international political communication.
Globalisation and a new media landscape challenge traditional foreign ministry ‘gatekeeper’ structures, and foreign ministries can no longer lay claim to being sole or dominant actors in communicating foreign policy. This demands new ways of communicating foreign policy to a range of nongovernmental international actors, and new ways of evaluating the influence of these communicative efforts. But where do the lines between old and new public diplomacies actually meet? How much current PD policy and practice conforms to older styles of communication, and how much can truly be considered new? What are the practical constraints upon the adoption of an entirely ‘new’ PD?
This PhD thesis investigates the methods and strategies used by 5 foreign ministries and cultural institutes in 3 countries as they attempt to adapt their PD practices to the demands of the new public diplomacy environment. The question is not simply of how government actors have phased out their archaic old PD practices, but of how the continual need for short-term influence – for discernable impact, outcomes, value-for-money – complicates the paradigm shift. The case studies are based around an analysis of US, British, and Swedish strategies. Each chapter covers national policy, evaluation methods, and examples of individual campaigns. Material consists of 25 interviews with PD practitioners, detailed policy studies, reconstructions of 5 PD campaigns, and analysis of communication models and evaluation methodologies.

Foubert, J.D. & Masin, R. (in press). Effects of The Men’s Program on U.S. army soldiers’ intentions to commit and willingness to intervene to prevent rape: A pretest posttest study. Violence and Victims

by John Foubert

Non-commissioned male officers in the U.S. Army stationed in Germany were trained to present a one-hour rape... more Non-commissioned male officers in the U.S. Army stationed in Germany were trained to present a one-hour rape prevention workshop, The Men’s Program, to 237 enlisted male soldiers. A comparison group of 244 male soldiers received a briefing focused on reducing the individual’s risk for experiencing sexual assault, discussion of myths and facts about sexual assault, and how to avoid being accused of sexual assault. Participants in The Men’s Program experienced significant change in the predicted direction for bystander willingness to help, bystander efficacy, rape myth acceptance, likelihood of raping, and likelihood of committing sexual assault with low to medium effect sizes. Comparison group participants experienced no effect on these variables except for a significant decline in rape myth acceptance with a very low effect size. Between-group differences pointed to the efficacy of The Men’s Program. Implications of these results for rape prevention programming in the military are discussed.

Exploring formal information technology evaluation practices in African firms

by Rennie Naidoo

Co-authored with Palk, Walter

Despite its universal appeal, little has been done in the way of empirical research to probe into the role of formal... more Despite its universal appeal, little has been done in the way of empirical research to probe into the role of formal information technology (IT) investment evaluation practices and its influence on IT governance and performance in African firms. A preliminary survey of 74 firms from South Africa, Nigeria and Zimbabwe, reveal that while firms are generally aware of formal IT investment appraisal techniques and IT value realisation prescriptions, relatively less formality is actually applied to appraising, managing and realising IT benefits. This research unveils many issues in exploiting IT benefits, with a majority of firms acknowledging substantial targeted benefits losses. Addressing how IT investments can be managed more effectively using formal investment appraisal and benefits management practices, to predictably realise intended payoffs remains a top concern for African firms and the continent’s IT researchers.

Key words: Information technology (IT), IT governance, IT investment appraisal, IT benefits management, IT performance.

Determining Faculty and Student Views: Applications of Q Methodology in Higher Education

by Susan Ramlo

Ramlo, S. (2012).  Determining Faculty and Student Views: Applications of Q Methodology in Higher Education.  Journal of Research in Education, 22(1), pp 86-106. 

William Stephenson specifically developed Q methodology, or Q, as a means of measuring subjectivity. Q has been used... more William Stephenson specifically developed Q methodology, or Q, as a means of measuring subjectivity. Q has been used to determine perspectives / views in a wide variety of fields from marketing research to political science but less frequently in education. In higher education, the author has used Q methodology to determine views about a variety of situations, from students' views about a newly developed bioinformatics course to faculty members' views of reading circles as a professional development experience to improve teaching and learning in their classrooms. The purpose of this article is to introduce Q methodology and demonstrate its versatility in addressing research purposes in higher education, especially where the focus is on determining student or faculty perceptions about a topic. Such determinations can be helpful for program evaluation and improved teaching/learning in higher education.

Evaluation of Contingent Repayments in Microfinance: Evidence from a Natural Disaster in Bangladesh

by Masahiro Shoji

The Developing Economies, forthcoming

The frequent and strictly scheduled repayments and savings in Microfinance often deteriorate the liquidity of members... more The frequent and strictly scheduled repayments and savings in Microfinance often deteriorate the liquidity of members in the face of negative shocks. Previous articles suggest the introduction of contingent repayment system that allows such members to be rescheduled, but the unavailability of a suitable dataset makes it difficult to examine how it actually works. This study is one of the first to evaluate the impact of this repayment system on household livelihood. In employing a unique dataset of Bangladesh, I show that rescheduling reduces the possibility of bindings the credit constraints and borrowing from moneylenders, implying that it may reduce transitory poverty. However, short-term rescheduling has insignificant effects. Finally, members indebted and with less liquid assets were more likely to be rescheduled.

(2011) American State Litter Scorecard: New Rankings for An Increasingly Environmentally Concerned Population

by Steve Spacek

Cited by THE BOSTON GLOBE and numerous Newspapers; TRAVEL+LEISURE; "Reducing Litter on Roadsides" Academic Journal of (U.S) National Transportations Academies.

By popular demand--a NEW, up-to-date State Litter "Scorecard" is now released for the 2011 ASPA Baltimore... more By popular demand--a NEW, up-to-date State Litter "Scorecard" is now released for the 2011 ASPA Baltimore Conference--measuring each state’s overall environmental quality through public property/spaces debris removal efforts. The “Scorecard” uses tried-and-true, hard-to-publicly obtain objective and subjective measures, leading to a total overall score for each measured jurisdiction. Readers can thus gain a realistic "picture" of "what's going on" within one or all of the 50 states. Littering/dumping remains harmful, serious American environmental crimes, creating dangers to public health and safety, and contributing to the deaths of over 800 Americans in debris-attributed motor vehicle accidents. The Scorecard has been a valuable tool in enabling improved debris/litter abatement practices in states and regions.

Washington was deemed the overall top BEST state, followed by California, Iowa, Maine, Connecticut, Rhode Island, Vermont, Oregon, New Hampshire, North Carolina and New York. Kentucky led the bottom –the ultimate WORST and poorest performer of the 50 United States, followed by Louisiana, Mississippi, Nevada, Alabama, Indiana, Georgia, Illinois, Oklahoma, Montana, North Dakota and Texas.

Regretfully, some states still do not collect important comparative litter data measures, nor yet to have made REAL differences in improving environmental conditions to protect citizenry health and welfare. This paper is yet another call to action to help remedy these unattended-to issues—and, to save lives.

Wilson, H, Marais, A de K, Old, S. 2000. Evaluation of a Business Management 1 course presented via distance education at the University of South Africa: factors that affect student preparation for industry. In, Hugo, WMJ (Ed). 2000. Conference proceedings of the 12th Southern Africa Institute for Management Scientists (SAIMS), 'Africa's Century'. Johannesburg: SAIMS, Held at the School for Business Leadership, Unisa, 31 October – 2 November 2000.

by Hentie Wilson

Community based divorce education programmes: Short-term and longer-term impacts

by Sherrill Hayes

co-author Lori Pelletier

Evaluation of a community-based parenting education program for parents in conflict over child custody and visitation.... more Evaluation of a community-based parenting education program for parents in conflict over child custody and visitation. The evaluation shows the positive short and longer term impacts of this program on parenting attitudes and behaviors and situates it in similar programs across North America.

More than a conversation: using cogenerative dialogues in the professional development of high school chemistry teachers

by Sonya N. Martin

Martin, S., & Scantlebury, K. (2009). More than a conversation: Using cogenerative dialogues in the professional development of high school chemistry teachers. Educational Assessment, Evaluation and Accountability, 21(2), pp. 119-1136

This paper focuses on content-based and pedagogical instructors’ use of cogenerative dialogues to improve... more This paper focuses on content-based and pedagogical instructors’ use of cogenerative dialogues to improve instructional practice and to evaluate program effectiveness in a professional development program for high school chemistry teachers. We share our research findings from using cogenerative dialogues as an evaluative tool for general assessment of various program-related issues. We discuss how engaging students in cogenerative dialogues improved teaching and learning in chemistry and chemistry education courses. This research provides insights and direction for improving content-based professional development programs for science teachers and the learning experiences of high school science students. Cogenerative dialogue has the potential to expand evaluation methodologies that will position participants more centrally in not only the collection of data, but also the analysis of these data to catalyze transformative practices in educational programs.

Quality After School Time: An Evaluative Study of the Eastside Story After School Program in Austin, TX

by Texas State PA Applied Research Projects

Lindsey, Jennifer, "Quality After School Time: An Evaluative Study of the Eastside Story After School Program in Austin, TX" (2010). Applied Research Projects, Texas State University-San Marcos. Paper 322.
http://ecommons.txstate.edu/arp/322

Research Purpose: As the need for quality after school care for school-age children grows, so does the need for a tool... more Research Purpose: As the need for quality after school care for school-age children grows, so does the need for a tool to measure the quality of after school programs. This study uses existing literature to develop a practical ideal type for after school programs. The model is then used to asses The Eastside Story; a program in Austin, Texas. Recommendations to improve of The Eastside Story program are drawn from the assessment.

Methodology: A thorough examination of literature pertaining to after school programs reveals five primary components of quality programs: strategic planning, partnership elements, staff elements, effective program practices, and child-centered location. Utilizing the case study research method, these five components are used to assess the Eastside Story program. The data collection methodologies in this case study include document analysis, structured interviews, and direct observation.

Results: The results of the case study illustrate Eastside Story's strengths and weaknesses. While current level of strategic planning at The Eastside Story fails to meet the criteria set in the practical ideal type, the criteria was met or exceeded for effective program practices and staff elements. Room for improvement within the partnership elements component was also identified.

Attitudes Toward Sunset Review in Texas

by Patricia Shields

Presented at the Annual meeting of the Southwestern Political Science Association, Fort Worth, Texas 1990

Appreciating appreciative inquiry

by Patricia Rogers

Improving the effectiveness of evaluations: Making the link to organizational theory* 1

by Patricia Rogers

Grammatikopoulos, V. (2012). Integrating program theory and systems based procedures in program evaluation: a dynamic approach to evaluate educational programs. Educational Research & Evaluation, 18(1), 53-64.

by Vasilis Grammatikopoulos

An Analysis of the Impact of the Texas Department of Public Safety Motorcycle Operator Training Program upon Motorcycle Related Traffic Accidents

by Texas State PA Applied Research Projects

Waller, Frankie Leon, "An Analysis of the Impact of the Texas Department of Public Safety Motorcycle Operator Training Program upon Motorcycle Related Traffic Accidents" (1992). Applied Research Projects, Texas State University-San Marcos. Paper 166.
http://ecommons.txstate.edu/arp/166

An Evaluation of the Snack Tax on the Obesity Rate of Maine

by Texas State PA Applied Research Projects

Oaks, Brion, "An Evaluation of the Snack Tax on the Obesity Rate of Maine" (2005). Applied Research Projects, Texas State University-San Marcos. Paper 29.
http://ecommons.txstate.edu/arp/29

Obesity has reached epidemic proportions in the United States and is becoming one of our nation’s most discussed... more Obesity has reached epidemic proportions in the United States and is becoming one of our nation’s most discussed public health concerns. Americans are bombarded with advertisements for quick weight loss medications, diet plans, and novelty fitness equipment, all of which guarantee results overnight. But what impact has this had on the weight of Americans?

Traditionally viewed as an individual’s disease, research now indicates that the environment is a main causative agent of overweight/obesity. Manipulation of food pricing to encourage consumers to make healthy purchases is a new area of public policy that can help address this epidemic, and has received reasonable academic interest.

As a pricing strategy, healthy foods can be subsidized to increase consumption, or unhealthy foods can be taxed to increase prices in order to discourage consumption. The state of Maine had a snack tax from 1991 to 2001 and provides a setting to evaluate the impact of taxing snacks, pastries, and soft drinks on a state’s obesity rate.

According to the “law of demand”, individuals will reduce their consumption of an item if its price rises. A snack tax increases the price of snack foods and soft drinks. Because consumption of these foods is associated with increased weight gain, it is hypothesized that a statewide tax on snacks and soft drinks in Maine would have a negative impact on the state’s obesity rate.

This study utilizes an interrupted time series comparison group design to conduct a regression analysis with obesity data obtained from the Behavioral Risk Factor Surveillance Survey. This study seeks to examine the relationship between a snack tax and obesity rates.

Although the regression model for the interrupted time series design was significant, the findings did not provide any significant results for independent variables that could help identify and interpret a relationship between the snack tax and obesity rates for Maine.

An Evaluation of the Southeast Austin Community Youth Development Program

by Texas State PA Applied Research Projects

Darrington, Patti, "An Evaluation of the Southeast Austin Community Youth Development Program" (2001). Applied Research Projects, Texas State University-San Marcos. Paper 159.
http://ecommons.txstate.edu/arp/159

The purpose of this research is explore the effectiveness of the Community Youth Development (CYD) program to reduce... more The purpose of this research is explore the effectiveness of the Community Youth Development (CYD) program to reduce juvenile crime and have a positive impact on targeted communities. The CYD Program is the responsibility of the Texas Department of Protective and Regulatory Services (PRS). In 1995, the 74th Texas Legislature created the Community Youth Development (CYD) Program to address juvenile crime. The goal of the program is to decrease the incidence of juvenile crime in Texas.

Research methodologies to explore the working hypotheses consisted of comparative case study and field research. First, the comparative case study allowed the comparison of the two zip code areas (CYD local area and comparison local area), to determine the effect of the Community Youth Development (CYD) Program in the targeted community. Second, the field research tested the impact of CYD in the targeted community through open-ended interviews, direct observations, and a focus group.

Although the evidence reviewed does not clearly indicate the impact of the program on juvenile crime, the Southeast Austin Community Youth Development Program has community support. The program is definitely meeting the needs of the youth for extracurricular activities. Children and families are afforded opportunities and resources within the 78744 community that were not present prior to the implementation of the program. Although many of the individuals interviewed for this study admitted they do not know the impact the CYD Program has had on juvenile crime in the community, they believe the program has had a positive impact in the community. Although preliminary evidence did not show the clear impact of the program in meeting the goal of reducing juvenile crime, the Southeast Austin Community Youth Development Program is certainly meeting many of the needs of the targeted community.