Evaluation Research

Universidad e investigación: la financiacion competitiva de los proyectos de I+D a de Investigaciones Sociológicas, n. 109, pp. 181-218.

by Luis Sanz-Menendez

Luis Sanz Menéndez (2005)
“Universidad e investigación: la financiación competitiva de los proyectos de I+D, con especial referencia a las ciencias sociales y económicas”,

Published in "Revista Española de Investigaciones Sociológicas", n. 109, pp. 181-218.

El objetivo de este trabajo es medir la relación de las universidades con la investigación. Se trata de situar a cada... more El objetivo de este trabajo es medir la relación de las universidades con la investigación. Se trata de situar a cada universidad en escalas relativas de capacidad investigadora, de esfuerzo u orientación hacia la investigación y de excelencia investigadora. El método utilizado se basa en el análisis de los resultados agregados de la competencia que los investigadores desarrollan por la financiación pública de la I+D. El periodo utilizado para la construcción de los datos corresponde a los proyectos de I+D financiados por la Administración General del Estado entre 1996 y 2001, a través de los Programas Nacionales de I+D y del Programa de Promoción General del Conocimiento (PGC). De modo singular se analizarán las actividades de investigación en el ámbito de las Ciencias Sociales y Económicas.

Useful, Used, and Peer Approved: The Importance of Rigor and Accessibility in Postsecondary Research and Evaluation

by Bo McCready

Traditionally, researchers, policymakers, and practitioners have perceived a tension between rigor and accessibility... more Traditionally, researchers, policymakers, and practitioners have perceived a tension between rigor and accessibility in quantitative research and evaluation in postsecondary education. However, this study indicates that both producers and consumers of these studies value high-quality work and clear findings that can reach multiple audiences. The authors discuss the perceived importance of rigor and accessibility, barriers to accomplishing both in practice, and suggestions for achieving an appropriate balance between the two.

International handbook of labour market policy and evaluation

by Jacqueline O'Reilly

Devil in the detail: using a pupil questionnaire survey in an evaluation of out-of-school classes for gifted and talented children

by Mike Lambert

Education 3-13, 36(1) February 2008, 69-78.

The use of questionnaires to evaluate educational initiatives is widespread, but often problematic. This paper... more The use of questionnaires to evaluate educational initiatives is widespread, but often problematic. This paper examines four aspects of an evaluation survey carried out with very able pupils attending out-of-school classes: ethics, design, bias and interpretation. There is a particular focus on the interpretation and analysis of pupils’ answers to open questions. Conclusions are drawn from this analysis which will help teachers and others to take a careful and critical approach to their use of questionnaires in educational evaluation.

Author profile: http://www.wlv.ac.uk/default.aspx?page=13209

[A limited number of eprints of this article are available - send me a message if you require one].

‘Erm, that question... I think I probably would’ve just put something in the middle and sort of moved on to the next one, because I think it’s really unclear’: How art and design students understand and interpret the National Student Survey

by Susan Orr

This paper (co authored with Bernadette Blair and Mantz Yorke) explores the ways that art and design students interpret the NSS questions.

Since the introduction of the National Student Survey in 2005, it has been noted that the average satisfaction scores... more Since the introduction of the National Student Survey in 2005, it has been noted that the average satisfaction scores vary across different disciplines (see, for example, Vaughan and Yorke 2009, p.8). And for just as long, it has been noted that art and design graduates’ satisfaction is among the lowest scoring disciplines. As the influence and reach of the NSS has increased - most recently in its inclusion as a key component of the government’s Key Information Set initiative - the art and design community has become increasingly concerned to explain the satisfaction gap experienced by many students studying in this area.

The community’s response is well articulated in the title of a study that explores this issue: “’I can’t believe it’s not better’: The Paradox of NSS scores for Art & Design” (Vaughan and Yorke 2009). This study was commissioned by GLAD  to look at how art and design has experienced and responded to the NSS, especially in the two major areas of feedback and assessment and organisation and management. The report showed that institutions varied considerably in terms of the ratings students gave to NSS questions, implying that there had been differential success in addressing aspects of the student experience probed by the survey. What the report did not attempt to do was gain an understanding of the basis of the ratings given by students.

It was therefore a logical extension to ask how art and design students understand and interpret the questions in the National Student Survey, and GLAD and HEA commissioned the authors to carry out a small research project which takes a snapshot of how a small group of art and design students interpreted the NSS questions in 2011.

This report is an overview of this study, and it aims to help colleagues in art and design subjects to:

• understand why art and design subjects receive the NSS ratings that they do;
• illuminate and understand how art and design students interpret and understand this generic questionnaire; and
• provide further information to assist staff and students to prepare for the NSS.

The key aim of this study is not to explain the responses that students give in the NSS. Its focus is instead on seeking to understand the ways in which students understand the NSS questions.  As a result this is a qualitative interview-based study.

The NSS questionnaire comprises 22 questions, and before the interviews the researchers narrowed the focus of this study down to eight of these. The eight questions were selected because they were viewed as central to the survey (e.g. question 22), or because they were questions that are particularly interesting in relation to creative and studio-based pedagogy in art and design (e.g. question 1).

Where is Knowledge Generated? On the Productivity and Impact of Political Science Departments in Latin America

by David Altman

[2012] European Political Science 11 (1): 71-87.

Clear rules that encourage meritocracy, and that include the evaluation of scholarly productivity, are slowly and... more Clear rules that encourage meritocracy, and that include the evaluation of scholarly productivity, are slowly and unevenly taking hold in academic life in Latin America. While some countries have official rankings of political science departments, others rely only on informal assessments. In a third set of countries, we cannot even consider the competition because the market is dominated by a state monopoly. This article provides a first, systematic study of scientific productivity and concomitant impact in more than twenty departments of Political Science and International Relations in the region. I show that scholars’ productivity is intimately related to where they pursued graduate studies, what subfield of research they work on, and the explicit adoption of rules that encourage meritocracy and academic careerism.

A study of effective evaluation models and practices for technology supported physical learning spaces: JISC Final Report

by Sarah Lewthwaite

Pearshouse, I., Bligh, B., Brown, B., Lewthwaite, S., Graber, R., Hartnell-Young, E., (2009) A Study of Effective Evaluation Models and Practices for
Technology Supported Physical Learning Spaces. JELS project. Joint
Information Systems Committee (JISC) commissioned report.

The aim of this project was to identify and review the tools, methods and frameworks used to evaluate technology... more The aim of this project was to identify and review the tools, methods and frameworks used to evaluate technology supported or enhanced physical learning spaces. A key objective was to develop the sector knowledgebase on innovation and emerging practice in the evaluation of learning spaces, identifying innovative methods and approaches beyond traditional post-occupancy evaluations and surveys that have dominated this area to date. The intention was that the frameworks and guidelines discovered or developed from this study could inform all stages of the process of implementing a technology supported physical learning space. The study was primarily targeted at the UK higher education sector and the further education sector where appropriate, and ran from September 2008 to March 2009.

An evaluation of the Australian Research Council's journal ranking

by Jerome Vanclay

Journal of Informetrics 5:265-274 (2011)

As part of its program of 'Excellence in Research for Australia' (ERA), the Australian Research Council ranked... more As part of its program of 'Excellence in Research for Australia' (ERA), the Australian Research Council ranked journals into four categories (A*, A, B, C) in preparation for their performance evaluation of Australian universities. The ranking is important because it likely to have a major impact on publication choices and research dissemination in Australia. The ranking is problematic because it is evident that some disciplines have been treated very differently than others. This paper reveals weaknesses in the ERA journal ranking and highlights the poor correlation between ERA rankings and other acknowledged metrics of journal standing. It highlights the need for a reasonable representation of journals ranked as A* in each scientific discipline.

How does capacity building of health managers work? A realist evaluation study protocol

by Prashanth Nuggehalli Srinivas

The Development of Educational Evaluation

by Dr. Ahmed Gumaa Siddiek

http://www.lulu.com/product/hardcover/assessment-of-the-sudan-school-certificate-english-examinations/5559687 - UK

A biblical story: "Gilieadites who took the passage of Jordon before the Ephraimiates said, let me go over; that... more A biblical story: "Gilieadites who took the passage of Jordon before the Ephraimiates said, let me go over; that the men of Gilead said unto him , art you an Ephraimate. If he said, nay; then said they unto him, Say now Shibboleth: and he said Sibbloeth: for he could not frame to pronounce it right. Then they took him, and slew him at the passage of Jordon: and there fell at the time of Ephaimites forty and two thousand (Judges).

Standardization of the Saudi Secondary School Certificate Examinations and the anticipated impact on Foreign Language Education

by Dr. Ahmed Gumaa Siddiek

Published in "International Journal of Humanities and Social Science Vol. 1 No. 3; March 2011" - USA

Scholastic Achievement tests are standardized in form and content to fairly serve all the audience - who take them -... more Scholastic Achievement tests are standardized in form and content to fairly serve all the audience - who take them - under similar circumstances. Therefore, they are reliable tools for measuring the performance of the examinees. On the other hand, proficiency tests are intended to measure the overall abilities of the learners. So they cannot serve as yardstick to tell us about the degree of the attainment of the national educational objectives embedded in the textbooks. The Education authority is advised to recall the national certificate examinations which qualify learners for higher education, under fair national competition. The role of other proficiency tests should be integrative to the major role of the Ministry of Education, who alone should have the right to interpret the usefulness and credibility of secondary school certificate, and how to use this national document in the assessment of students' performance at the personal, national or international level.

Evaluation of the Sudan School Certificate English Examiantions

by Dr. Ahmed Gumaa Siddiek

Published in "English Language Teaching Vol. 3, No. 2; June 2010" CANADA

Examinations - among other things - are tools of quality control by which we can measure the attainment of the... more Examinations - among other things - are tools of quality control by which we can measure the attainment of the national educational goals. High-quality examinations are means of evaluation that can help teachers modify their teaching techniques as well as helping learners adjust their learning strategies. Examinations are also benchmarks that can help decision makers to judge the success of their educational, social and political aims. This paper explores the features of the Sudan School Certificate English Examinations (SSC) from the perspective of content validly and comprehensiveness. The data was collected through a questionnaire and analytically processed by machine to come out with some findings that: the (SSC) English Examinations in their recent forms are incomprehensive and lack content validity. They are proficiency tests rather than scholastic standardized achievement examinations. Therefore, they have negative backwash on language education development in our country.

Evaluation of the Sudan School Certificate English Examiantions

by Dr. Ahmed Gumaa Siddiek

Published in "English Language Teaching Vol. 3, No. 2; June 2010" CANADA

Examinations - among other things - are tools of quality control by which we can measure the attainment of the... more Examinations - among other things - are tools of quality control by which we can measure the attainment of the national educational goals. High-quality examinations are means of evaluation that can help teachers modify their teaching techniques as well as helping learners adjust their learning strategies. Examinations are also benchmarks that can help decision makers to judge the success of their educational, social and political aims. This paper explores the features of the Sudan School Certificate English Examinations (SSC) from the perspective of content validly and comprehensiveness. The data was collected through a questionnaire and analytically processed by machine to come out with some findings that: the (SSC) English Examinations in their recent forms are incomprehensive and lack content validity. They are proficiency tests rather than scholastic standardized achievement examinations. Therefore, they have negative backwash on language education development in our country.

Towards the Development of an Evaluation Questionnaire for Academic Conferences

by Clifford Lewis

Lewis, C & Kerr, G (2012), ‘Towards the Development of an Evaluation Questionnaire for Academic Conferences’, Event Management, 16(1), 11-23

Each year, academic conferences are held at destinations throughout the world. These conferences provide benefits to... more Each year, academic conferences are held at destinations throughout the world. These conferences provide benefits to the host destination's economy as well as to the conference participants. Involving travel and accommodation, academic conferences can be classified as business tourism. Academics often have a range of conferences from which to choose. The conference experience therefore may be important in the decision to reattend or recommend a conference to other potential attendees. While many conference organizers distribute a “conference evaluation sheet“ at the end of a conference, there is no evidence of a standardized questionnaire that evaluates the entire conference experience. The objective of this work is to make such a contribution by identifying the attributes that are deemed to be important to the academic conference attendee and assigning a measurement scale for each attribute. The attributes are identified by way of a review of the services and tourism literature, and through semistructured interviews with academics. In addition to evaluating the entire conference experience, the questionnaire can be used to make longitudinal comparisons of a conference, and comparisons between conferences.

Resultados de investigación científica con visibilidad internacional: México (ISI-WoS, 1990-2004)

by Zaida Chinchilla-Rodríguez

Moya-Anegón, F., Chinchilla-Rodríguez, Z., Vargas-Quesada, B., Corera-Álvarez, E., Muñoz-Fernández, F. J. y González-Molina, A. Resultados de investigación científica con visibilidad internacional: México (ISI-WoS, 1990-2004), 2006. In Diagnóstico de la Política Científica, Tecnológica y de Fomento a la Innovación en México (2000-2006). Foro Consultivo Científico y Tecnológico (México). pp.89-157. (Published) [Book Chapter].

The main goal of this study is to describe and analyze the scientific output of Mexico, both at institutional level as... more The main goal of this study is to describe and analyze the scientific output of Mexico, both at institutional level as thematic distribution, by the use of a set of scientometric indicators for the period 1990-2004 using Web of Science as data source.

El objetivo principal de este estudio es describir y analizar la producción científica de México, tanto a nivel institucional como la distribución temática, por el uso de un conjunto de indicadores cienciométricos para el período 1990-2004 utilizando la Web of Science como fuente de datos.

Producción científica e impacto: ranking general y por áreas de las instituciones universitarias españolas

by Zaida Chinchilla-Rodríguez

Corera-Álvarez, E., Chinchilla-Rodríguez, Z., Moya-Anegón, F., y Sanz-Menéndez, L.  Producción científica e impacto: ranking general y por áreas de las instituciones universitarias españolas, 2010. In La contribución de las universidades españolas al desarrollo. Informe CyD 2009. Fundación Conocimiento y Desarrollo (Spain). pp.254-262. (Published) [Book Chapter].

The objective is to present the scientific outputs, in terms of scientific publications of the High Education... more The objective is to present the scientific outputs, in terms of scientific publications of the High Education Institutions (HEI) to locate the universities in a ranking, taking into account: the volume of publications and its relative quality measures by impact indicators using Scopus as source data.

El objetivo de este recuadro es presentar los resultados de investigación, representados éstos por la publicaciones científicas, de las instituciones españolas de educación superior y utilizar los mismos para posicionar a las universidades en un ránking, atendiendo a dos dimensiones esenciales: por un lado, el volumen total de producción científica y, por otro, la calidad relativa, medida a través de indicadores de impacto o citación de esos trabajos. Los datos se han generado a partir de los registros bibliométricos incluidos en la base de datos Scopus.

Producción y colaboración científica en agroalimentación

by Zaida Chinchilla-Rodríguez

Chinchilla-Rodríguez, Z., Olmeda-Gómez, C. Producción y colaboración científica en agroalimentación, 2010. In Análisis sobre Ciencia e Innovación en España: Anuario ICONO. Fundación Española para la Ciencia y la Tecnología. pp.366-399. (Published) [Book Chapter].

This chapter presents a multilevel analysis that aims to characterize the generation of knowledge visible... more This chapter presents a multilevel analysis that aims to characterize the generation of knowledge visible internationally in the field of food and agriculture in Spain during the last fifteen years from the analysis of research results in the terms of scientific publications. From visual representations of scientific information collected in international databases, analyze the patterns of publication and collaboration. It is expected to provide information relevant to decision making in science and technology.

Este capítulo presenta un análisis multinivel que pretende caracterizar la generación de conocimiento visible internacionalmente en el campo de la agroalimentación en España durante los últimos quince años a partir del análisis de los resultados de investigación en forma de publicaciones científicas. A partir de representaciones visuales de la información científica recogida en bases de datos internacionales, se analizan los patrones de publicación y colaboración. Se espera aportar información de interés para la toma de decisiones en materia de ciencia y tecnología.

La producción científica española en el contexto internacional y la posición de sus instituciones de investigación en el ránking mundial

by Zaida Chinchilla-Rodríguez

Chinchilla-Rodríguez, Z., Corera-Álvarez, E., Moya-Anegón, F., y Sanz-Menéndez, L.  La producción científica española en el contexto internacional y la posición de sus instituciones de investigación en el ránking mundial, 2011. In La contribución de las universidades españolas al desarrollo.. Fundación Conocimiento y Desarrollo (Spain). pp.292-301. (Published) [Book Chapter].

El objetivo de este apartado es presentar los principales indicadores de producción científica españoles, respecto a... more El objetivo de este apartado es presentar los principales indicadores de producción científica españoles, respecto a la cantidad, calidad y visibilidad, del total de la producción española y del conjunto de instituciones del país. De esta manera, se contextualiza en una perspectiva comparada internacional la posición española, lo que permite un análisis más completo de la situación y su evolución.

The purpose of this section is to present the main Spanish scientific production indicators, regarding the quantity, quality and visibility of the total Spanish production and the whole country's institutions. Thus, it is contextualised in an international comparative perspective the Spanish position, allowing a more complete analysis of the situation and its evolution.

Producción científica e impacto: ránking general y por áreas de las instituciones universitarias españolas (2004-2008)

by Zaida Chinchilla-Rodríguez

Corera-Álvarez, E., Chinchilla-Rodríguez, Z., Moya-Anegón, F. y Sanz-Menéndez, L. Producción científica e impacto: ránking general y por áreas de las instituciones universitarias españolas (2004-2008), 2011. In La contribución de las universidades españolas al desarrollo. Fundación Conocimiento y Desarrollo (Spain). pp.283-291. (Published) [Book Chapter].

This chapter refers to the presentation of the results of the scientific production of Spanish institutions of higher... more This chapter refers to the presentation of the results of the scientific production of Spanish institutions of higher education, from which it is created a ranking, according to three key dimensions: the total volume of scientific production, the relative quality, measured through indicators of impact or citation of this work, and this year, the percentage of papers that have been published in journals ranked by their level of impact in the first quartile of its category (the most prestigious journals)

Este capítulo hace referencia a la presentación de los resultados de la producción científica de las instituciones españolas de educación superior, a partir de los cuales se crea un ránking, atendiendo a tres dimensiones esenciales: el volumen total de producción científica; la calidad relativa, medida a través de indicadores de impacto o citación de esos trabajos, y, este año, el porcentaje de trabajos que se han publicado en revistas clasificadas por su nivel de impacto en el primer cuartil de su categoría (las revistas más prestigiosas).

Redesigning Computer-Based Learning Environments: Evaluation as Communication

by Ivan Ricarte

Co-authored with Matthias Brust and Christian Adriano. Published in International Conference on Education and Information Systems, 2007.

In the field of evaluation research, computer scientists live constantly upon dilemmas and conflicting theories. As... more In the field of evaluation research, computer scientists live constantly upon dilemmas and conflicting theories. As evaluation is differently perceived and modeled among educational areas, it is not difficult to become trapped in dilemmas, which reflects an epistemological weakness. Additionally, designing and developing a computer-based learning scenario is not an easy task. Advancing further, with end-users probing the system in realistic settings, is even harder. Computer science research in evaluation faces an immense challenge, having to cope with contributions from several conflicting and controversial research fields. We believe that deep changes must be made in our field if we are to advance beyond the CBT (computer-based training) learning model and to build an adequate epistemology for this challenge. The first task is to relocate our field by building upon recent results from philosophy, psychology, social sciences, and engineering. In this article we locate evaluation in respect to communication studies. Evaluation presupposes a definition of goals to be reached, and we suggest that it is, by many means, a silent communication between teacher and student, peers, and institutional entities. If we accept that evaluation can be viewed as set of invisible rules known by nobody, but somehow understood by everybody, we should add anthropological inquiries to our research toolkit. The paper is organized around some elements of the social communication and how they convey new insights to evaluation research for computer and related scientists. We found some technical limitations and offer discussions on how we relate to technology at same time we establish expectancies and perceive others work.