Delphi Methodology

A Delphi Study of New Technology

by Rebeca Pop

Authors:
Michael L. Kent, Ph.D.
Adam Saffer, MA
Rebeca Pop, BA

In an effort to learn what other technology professionals know about new technologyand social media, this study used a... more In an effort to learn what other technology professionals know about new technologyand social media, this study used a Delphi methodology and solicited the participation of more than fifty technology professionals from computer science, professional writing, communication, art, business, music, and other areas. Keeping up with technological innovations in any area takes a lot of time. Keeping up with innovations in every area is probably impossible. Thus, this study sought to learn about broad issues of technologies and forecast trends.

Elragal, A. and Haddara, M., "The Use of Experts Panels in ERP Cost Estimation Research" in Varajao, J , Cruz-Cunha, M. and. António Trigo (Eds.), Organizational Integration of Enterprise Systems and Resources: Advancements and Applications, Information Science Reference, IGI Global, Hershey, PA, 2012, pp.117-130.

by Moutaz Haddara

Book chapter

A Vision for the Information Technology Experience in a University-based Business School

by Peter Mellalieu

Mellalieu, P. J., & Hooper, J. (1994). A Vision for the Information Technology Experience in a University-Based Business School. Department of Management Systems Discussion Paper. Palmerston North, NZ: Massey University.

Also presented at:
Mellalieu, P. J., & Hooper, J. (1995). Australia and New Zealand Academy of Management Conference (ANZAME), Wellington, New Zealand.

A Delphi survey of proficient information technology staff and students at a mixed-mode university identified features... more A Delphi survey of proficient information technology staff and students at a mixed-mode university identified features for a vision for the 'information technology' expereince of business studies students. The study, conducted in 1990, identified the need for: university-established computer network facilities that would facilitate both student-student and staff-student computer-mediated communication; 'Connection Centres' (hubs) located throughout the university campuses that would enable students to connect their portable computers to the university network; and a students' information technology users' club.

The improving performance/price ratio of personal computers implied the prospect of students purchasing or hiring their own computer systems. The Delphi respondents recommended that standards for hardware and software should enable students to select their systems from a wide range of options. This latter recommendation has important implications for the design of student assignments and network connection standards.

Reviewing progress four years after the initial study (conducted in 1990), the university has established the first element neccessary for the implementation of the vision, the establishment of an email system for extramural students.

Identifying outcome-based indicators and developing a curriculum for a continuing medical education programme on rational prescribing using a modified …

by Carl Savage

Esmaily HM, Savage C, Vahidi R, Amini A, Zarrintan MH, Wahlstrom R.
BMC Med Educ. 2008 May 30;8:33.
PMID: 18510774 [PubMed - indexed for MEDLINE] Free PMC Article
http://www.ncbi.nlm.nih.gov/pubmed/18510774

Abstract:
Continuing medical education (CME) is compulsory for physicians in Iran. Recent studies in Iran show... more Abstract:
Continuing medical education (CME) is compulsory for physicians in Iran. Recent studies in Iran show that modifications of CME elements are necessary to improve the effectiveness of the educational programmes. Other studies point to an inappropriate, even irrational drug prescribing. Based on a needs assessment study regarding CME for general physicians in the East Azerbaijan province in Iran, rational prescribing practice was recognized as a high priority issue. Considering different educational methods, outcome-based education has been proposed as a suitable approach for CME. The purpose of the study was to obtain experts' consensus about appropriate educational outcomes of rational prescribing for general physicians in CME and developing curricular contents for this education.

METHODS:
The study consisted of two phases: The first phase was conducted using a two-round Delphi consensus process to identify the outcome-based educational indicators regarding rational prescribing for general physicians in primary care (GPs). In the second phase the agreed indicators were submitted to panels of experts for assessment and determination of content for a CME program in the field.

RESULTS:
Twenty one learning outcomes were identified through a modified Delphi process. The indicators were used by the panels of experts and six educational topics were determined for the CME programme and the curricular content of each was defined. The topics were 1) Principles of prescription writing, 2) Adverse drug reactions, 3) Drug interactions, 4) Injections, 5) Antibiotic therapy, and 6) Anti-inflammatory agents therapy. One of the topics was not directly related to any outcome, raising a question about the need for a discussion on constructive alignment.

CONCLUSIONS:
Consensus on learning outcomes was achieved and an educational guideline was designed. Before suggesting widespread use in the country the educational package should be tested in the CME context.

Determining the most important factors for sustainable energy technology selection in Africa: Application of the Delphi Technique

by Herman Steyn

Development of a decision-framework for establishing a Health Register following a Major Incident

by Giovanni Leonardi

Article accepted for publication in "Prehospital and Disaster Medicine"

Karthikeyan Paranthaman 1, Mike Catchpole 2, John Simpson 3, Jill Morris 4, Colin R Muirhead 5, Giovanni S Leonardi 1

1 Health Protection Agency, Centre for Radiation, Chemical and Environmental hazards, Chilton, England
2 Health Protection Agency, Centre for Infections, London, England
3 Health Protection Agency, Centre for Emergency Preparedness and Response, Porton Down, England
4 Health Protection Agency, Local and Regional Services, Chilton, England
5 Institute of Health and Society,  Newcastle University

Objectives: Health Registers have been established in the UK and elsewhere following mass exposure to novel agents or... more Objectives: Health Registers have been established in the UK and elsewhere following mass exposure to novel agents or known agents in novel or exceptional circumstances but there is no consensus on the criteria for establishing such registers.  This study aimed to develop a decision-framework to assess the need for establishing a Health Register for major incidents.

Methods: The study comprised of three stages. In the first stage, the study team prepared a list of potential criteria that may be used to assess the need for setting up a health register based on literature review and personal experiences in previous incidents. In the second stage, the potential criteria were evaluated in two Delphi rounds involving experts and key decision-makers from the HPA and academic organisations. In the final stage, the criteria were converted in to a decision-framework and its utility was tested using four fictional scenarios. 

Results: A total of 11 criteria were proposed by the study group.  These criteria were suitably revised following feedback from 16 experts in the first Delphi round. All eleven statements achieved consensus at the end of the second Delphi round. The pilot testing of the agreed criteria on four fictional scenarios confirmed the validity and reliability for use in the decision process.

Conclusions: A decision-framework to assess the need for setting up a Health Register after a major incident was agreed and tested using four fictional incidents. Further areas of work for practical implementation of the criteria and related planning for systems and protocols have been identified.

Forecasting for Marketing

by J Armstrong

Co-authored with Roderick J. Brodie. Published in Graham J. Hooley and Michael K. Hussey (Eds.),

Research on forecasting is extensive and includes many studies that have tested alternative methods in order to... more Research on forecasting is extensive and includes many studies that have tested alternative methods in order to determine which ones are most effective. We review this evidence in order to provide guidelines for forecasting for marketing. The coverage includes intentions, Delphi, role playing, conjoint analysis, judgmental bootstrapping, analogies, extrapolation, rule-based forecasting, expert systems, and econometric methods. We discuss research about which methods are most appropriate to forecast market size, actions of decision makers, market share, sales, and financial outcomes. In general, there is a need for statistical methods that incorporate the manager's domain knowledge. This includes rule-based forecasting, expert systems, and econometric methods. We describe how to choose a forecasting method and provide guidelines for the effective use of forecasts including such procedures as scenarios.

Research Methodologies Study in Instructional Design (Working Title)

by Tim Stafford

Draft for future publication - not fully completed. Only working Title

Research methods, the research questions, and the context are always intertwined, but the researcher "must decide... more Research methods, the research questions, and the context are always intertwined, but the researcher "must decide what information will be needed to address the problem and how best to obtain that information" (Merriam, 1988, p. 71).While there are many techniques used to collect data in qualitative studies, three stand out due to their being easily implemented for small sample size studies, scalable if larger studies are prescribed and also easily combined to create greater accuracy through triangulation. This article identifies, discusses and compares the purposes of these three techniques, interviews, observations and the Delphi Methodology with specific attention given to their applicability for building consensus among established experts as is pertains to best practices for designing in new technology platforms like m-learning. Finally, the article recommends and defends an appropriate strategy for the use of m-learning based on these evaluations.

Introduction to paper and commentaries on the Delphi technique

by J Armstrong

Introductory remarks about the success of the Delphi technique in forecasting and other relevant disciplines. Touches... more Introductory remarks about the success of the Delphi technique in forecasting and other relevant disciplines. Touches upon the applications of the method.

“Findings from Evidence-based Forecasting: Methods for Reducing Forecast Error

by J Armstrong

Forthcoming (after revisions) in the International Journal of Forecasting.

Empirical comparisons of reasonable approaches provide evidence on the best forecasting procedures to use under given... more Empirical comparisons of reasonable approaches provide evidence on the best forecasting procedures to use under given conditions. Base on this evidence, I summarize the progress made over the past quarter century with respect to methods for reducing forecasting error. Seven well-established methods have been shown to improve accuracy: combining forecasts and Delphi help for all types of data; causal modeling, judgmental bootstrapping and structured judgment help with cross-sectional data; and causal models and trend-damping help with time-series data. Promising methods for cross-sectional data include damped causality, simulated interaction, structured analogies, and judgmental decomposition; for time-series data, they include segmentation, rule-based forecasting, damped seasonality, decomposition by causal forces, damped trend with analogous data, and damped seasonality. The testing of multiple hypotheses has also revealed methods where gains are limited: these include data mining, neural nets, and Box-Jenkins methods. Multiple hypotheses testing should be conducted on widely used but relatively untested methods such as prediction markets, conjoint analysis, diffusion models, and game theory.

Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared

by J Armstrong

Co-authored with Kesten C. Green and Andreas Graefe. Forthcoming in Foresight: The International Journal of Applied Forecasting. Working paper.

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare... more Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. Prediction markets can be run continuously, and they motivate participation and participants to reveal their true beliefs. On the other hand, they need many participants and clear outcomes in order to determine pay-offs. Moreover, translating knowledge into a price is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists’ knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts but may also lead to conformity due to group pressure. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the “black box” nature of prediction markets. We consider that, half a century after its original development, Delphi is under-utilized.

The Ecological System of Innovation: A New Architectural Framework for a Functional Evidence-Based Platform for Science and Innovation Policy

by Robert M. Yawson

In: The Future of Innovation - Proceedings of the XX ISPIM 2009 Conference, Vienna, Austria, June 21-24, 2009

Models on innovation, for the most part, do not include a comprehensive and end-to-end view. Most innovation policy... more Models on innovation, for the most part, do not include a comprehensive and end-to-end view. Most innovation policy attention seems to be focused on the capacity to innovate and on input factors such as R&D investment, scientific institutions, human resources and capital. Such inputs frequently serve as proxies for innovativeness and are correlated with intermediate outputs such as patent counts and outcomes such as GDP per capita. While this kind of analysis is generally indicative of innovative behaviour, it is less useful in terms of discriminating causality and what drives successful strategy or public policy interventions. This situation has led to the developing of new frameworks for the innovation system led by National Science and Technology Policy Centres across the globe. These new models of innovation are variously referred to as the National Innovation Ecosystem. There is, however, a fundamental question that needs to be answered: what elements should an innovation policy include, and how should such policies be implemented? This paper attempts to answer this question.

Expert attitudes to management contracts in the Dutch hotel industry: a DELPHI approach

by Rob van Ginneken

This is the author's version of the work. It is posted here by permission of
Australian Academic Press for personal use, not for redistribution. The definitive version was published in Journal of Hospitality and Tourism Management Volume 18, Issue 1, October 2011. doi:10.1375/jhtm.18.1.140 http://dx.doi.org/10.1375/jhtm.18.1.140

This paper reports an experimental Delphi study of expert attitudes to management contracts in the Dutch hotel... more This paper reports an experimental Delphi study of expert attitudes to management contracts in the Dutch hotel industry. The study was prompted by appreciation of the apparent trend towards the divesting of property by international hotel companies which is also explored here. The purpose of the research reported was to ascertain expert views of the implications of such a trend for the Dutch hotel industry which has a history of risk aversion and aversion to management contracts as a model for hotel operations. The Delphi approach allowed access to a variety of experts regarding different groups and stakeholder perceptions.
The findings of the study, which is the first of its kind to address this question in the Dutch hotel industry context, suggest that there is an expectation of greater diversity in the financial arrangements for managing hotel operations, but no inclination to radically change the business practices that have hitherto characterised the Dutch hotel sector.

E-Learning: Education for Everyone? Special Requirements on Learners in Internet-based Learning Environments

by Thomas Richter

Richter, T. & Adelsberger, H.H. (2011). In: Proceedings of the World Conference on Educational Multimedia, Hypermedia & Telecommunication, pp. 1598-1604, AACE, Chesapeake.

This paper addresses special skills, learners in E-Learning scenarios (Internet-based learning scenarios) need. In... more This paper addresses special skills, learners in E-Learning scenarios (Internet-based learning scenarios) need. In self directed learning scenarios, as most E-Learning scenarios are designed, learners bear the responsibility for their own learning progress. To ease this task, institutions could prime the learners for the situation, which may be quite different to the learning experiences, they previously made in face-to-face learning scenarios. We conducted a Delphi-study with experts from the Higher Education E-Learning sector in Germany, Austria, and Switzerland, and determined general requirements on learners in E-Learning scenarios.

The Delphi Method As a Research Tool: An Example, Design Considerations and Applications

by Chitu Okoli

The Delphi technique as a forecasting tool: Issues and analysis

by Gene Rowe

Elragal, A., & Haddara, M. (2010). The Use of Experts Panels in ERP Cost Estimation Research. In J. E. Quintela Varajão, M. M. Cruz-Cunha, G. D. Putnik & A. Trigo (Eds.), ENTERprise Information Systems (Vol. 110, pp. 97-108). Berlin Heidelberg: Springer.

by Moutaz Haddara

This paper is an effort towards illustrating the use of expert panel (EP) as a mean of eliciting knowledge from a... more This paper is an effort towards illustrating the use of expert panel (EP) as a mean of eliciting knowledge from a group of enterprise resource planning (ERP) experts as an exploratory research. The development of a cost estimation model (CEM) for ERP adoptions is very crucial for research and practice, and that was the main reason behind the willingness of experts to participate in this research. The use of EP was very beneficial as it involved various data collection and visualisation techniques, as well as data validation and confirmation. Beside its advantages, one of the main motives for using a group technique is that it is difficult to find a representative sample for a casual survey method, as ERP experts and consultants are rare to find, especially in the scope of SMEs’ ERP implementations. It is worth noting that the panel reached consensus regarding the results of the EP. The experts modified and enhanced the initial cost drivers (CD) list largely, as they added, modified, merged and split different costs drivers. In addition, the experts added CF (sub-factors) that could influence or affect each cost driver. Moreover, they ranked the CD according to their weight on total costs. All of this helped the authors to better understand relationships among various CF.

MAPAS DELPHI: INSTRUMENTO DE APOYO PARA LA PREPARACIÓN, INCUBACIÓN E INTUICIÓN DE IDEAS INNOVADORAS

by Javier Rivera

Rivera, J., Vidal, R., & Lloveras, M. (2007). Mapas Delphi: Instrumento de apoyo para la preparación, incubación e intuición de ideas innovadoras. XI Congreso Internacional de Ingeniería de Proyectos (págs. 700-711). Lugo: Universidad de Santiago de Compostela

Abstract
Diverse methodologies are focused to the advantage of the information and intelligence, the Delphi... more Abstract
Diverse methodologies are focused to the advantage of the information and intelligence, the Delphi method is one of them. Method used for the structuring of processes of communication between a group of individuals to treat a complex problem, aspect that locates it in a tool with great capacity to explore and to stimulate intelligence and/or the creativity.

This work tries to valorize the used and interaction of the graphical structures of communication in the Delphi reports and is reflected on the advantage of other graphical structures (Mind Maps and Road Maps), with the intensity to obtain the greater one I stimulate (amount, quality and impact) in the phases of the creative process (preparation, incubation, intuition and evaluation).

Resumen
Diversas metodologías están enfocadas al aprovechamiento de la información e inteligencia, el método Delphi es una de ellas. Método utilizado para la estructuración de procesos de comunicación entre un grupo de individuos para tratar un problema complejo, aspecto que lo sitúa en una herramienta con gran capacidad para explorar y estimular la inteligencia y/o la creatividad.

Este trabajo pretende valorizar el uso e interacción de las estructuras gráficas de comunicación de los reportes Delphi y se reflexiona sobre el aprovechamiento de otras estructuras gráficas (Mapas Mentales y Mapas de Rutas Tecnológicas), con la intensión de obtener un mayor estimulo (cantidad, calidad e impacto) en las fases del proceso creativo (preparación, incubación, intuición y evaluación).