Principles and processes in coaching evaluation
D E Gray
School of Management, University of Surrey
International Journal of Mentoring and Coaching 2(2).
Online at: http://www.emccouncil.org/uk/journal.htm
1
Principles and practice of coaching evaluation
Principles and processes in coaching evaluation
D E Gray
School of Management, University of Surrey
Abstract
Coaching, as a management development and change management tool is becoming
increasingly prominent. However, many coaching programmes are either evaluated
superficially (if at all) or at Kirkpatrick’s (1959) Level 1 – the reaction of participants
to the programme. For coaching to gain sustainable credibility, evaluation needs to be
more thorough, planned, systematic and inclusive. This article considers why
coaching should be evaluated, who should take part in the evaluation and how
evaluation can be conducted. These approaches to evaluation are influenced by the
philosophical stance of the evaluator and range from experimental to the more
illuminative evaluation of unanticipated outcomes. Whatever the results of evaluation
it is essential that the outcomes of coaching are disseminated to key stakeholders –
particularly those who sponsor and finance coaching interventions. But since
coaching is such a complex phenomenon, the deep psychological and social processes
of coaching also merit evaluation. Some empirical data on how a small sample of
coaches evaluate their own coaching processes is presented, and some
recommendations made for professional practice.
INTRODUCTION
Evaluation involves the systematic collecting of data, often so that essential
modifications and improvements to a coaching programme can be implemented. This
evaluation can take place at many levels from ad hoc and informal to highly
structured and formal. The difference between these approaches is largely around
how the evaluation is conducted. Informal evaluation, which is probably by far the
most common approach, may merely comprise verbal feedback from a client or
coachee on the merits, or otherwise, of a programme. In contrast, Lincoln and Gubba
(1986:550) describe formal evaluation as a form of ‘disciplined inquiry’ where
scientific principles are applied to the collection and analysis of information about the
content, structure and outcomes of programmes, projects and planned interventions.
Interest in evaluating coaching programmes is on the increase, stimulated, in large
part, by the growth of organisations that are using coaches, the growing expenditure
on coaching and the number of people offering coaching services. Coaches, at
present, constitute an unregulated profession where there is, as yet, no widely
recognised professional standard or qualification. Despite this, formal evaluation of
coaching initiatives is currently lacking (CIPD, 2004a). Furthermore, the evaluation
of most training and development initiatives (of which coaching is one) suffers from a
lack of accurate and complete information, poor information or a lack of information
when it is needed (Campbell, 1997).
The evaluation of coaching, then, offers the potential for understanding more about
the coaching process, what kinds of interventions are effective (and within what kinds
of organisational or cultural contexts), and the potential role for coaching within the
wider context of management development. To date, however, much of the data on
coaching evaluation is anecdotal or simply non-existent. This paper sets out to
examine why the evaluation of coaching is important, what features of coaching can
be evaluated, how, when and by whom.
2
Principles and practice of coaching evaluation
WHY EVALUATE COACHING?
Coaching, like any other professional development intervention, should be evaluated,
because evaluation allows us a critical window into our professional practice. It also
allows us to gain a degree of self-knowledge, and the opportunity to identify how
personal interactions and coaching processes can be improved.
But there are additional factors to suggest why, specifically, coaching programmes
deserve evaluation. Any new development initiative (and certainly the wide-scale use
of coaching in management development is relatively new) requires evaluation to
identify methodological and other weaknesses. Coaching, in particular, is worthy of
evaluation because it is practiced by people with a diverse range of skills, experience
and knowledge. Unlike, say, medicine, where entry to the profession requires many
years of study, formal examination, adherence to a specific code of conduct, and
statutory regulation by a professional body, coaching, as yet, has no, single, unified
professional structure.
This is not to call into question the competence of many practicing coaches. But a
number of stakeholders have much to gain from the evaluation process. Evaluation
offers coaches an opportunity to analyse and receive feedback on their professional
practice. Evaluation is of value to those organisations offering coaching services
because it allows them an opportunity to identify the strengths (and weaknesses) of
individual coaches and perhaps to ascertain the organisational or cultural contexts
most suited to their coaching style. For those organisations commissioning coaching
services, evaluation can provide data on the organisational and financial benefits of
coaching in comparison to other more traditional training and development initiatives.
It can also help organisations achieve a ‘best fit’ between the organisation’s coaching
needs and the coaching services on offer.
WHO SHOULD EVALUATE?
Given that there are numerous stakeholders in the coaching relationship, in principle,
several, or all of them could play a part in the evaluation process. Hay (1995), for
example, identifies coaching as a four-cornered contract involving the coachee, the
coach, the organisation (especially the HR department) and the coachee’s line
manager. All, potentially, have a contribution to make, and each can add a different
perspective. The coachee will want to reflect on what they may have gained from the
coaching process. Was it worthwhile? Were the objectives of the coaching
relationship achieved? Has the coaching left any lasting legacy of knowledge, or
understanding? Especially if the coaching intervention has been financed centrally,
the HR department will want to gain some insight into what the organisation may
have gained particularly in terms of hard data such as Return on Investment (if this is
possible to measure). Line managers may have been responsible for initiating the
coaching process or may have been persuaded by the HR department that it is needed
and will want to know if it has improved the performance of coachees.
As professional practitioners, coaches will want to evaluate their own practice. This
may involve seeking feedback from coachees, and perhaps their line managers but it
should also involve engaging in a process of self-evaluation. If coaches have also
engaged the services of a supervisor, then the self-reflective evaluation process can
take place through this supportive dialogue. Hence, coaches can discuss what they
felt was successful in a particular coaching relationship, but also, confidentially, raise
areas that went less well and even focus on failures or possible mistakes.
3
Principles and practice of coaching evaluation
The above analysis may give an impression that evaluation is undertaken on a silo
basis, with each stakeholder pursuing their own evaluation goals. But the evaluation
process may yield more illuminating data if conducted on a collaborative basis.
Hence, coach and coachee may wish to jointly review both the outputs and processes
of the coaching relationship. Especially if they have sponsored the coaching
programme, HR managers may want to meet with coaches to evaluate the planning,
conduct and outputs of coaching. This process might be particularly illuminating if
the organisation sets up a focus group of coaches where it may be possible to
aggregate themes and issues emerging at an individual level into more macro-level of
strategic concerns.
Another key issue is whether the evaluation process should be conducted by internal
or external sources. Table 1 outlines some of the potential benefits and drawbacks of
each. While internal evaluators will be more familiar with the culture and needs of
the organisation, they may not always possess the degree of independence required by
a thorough and systematic evaluation programme. External evaluators, however,
bring an independence to the evaluation process and can use their experience of
evaluating in other organisations as a comparative benchmark.
Table 1 The advantages and disadvantages of using internal or external
evaluators
Advantages
Internal evaluators will be:
• Familiar with the history, policies, issues and culture of the organisation
• Likely to focus on the central concerns of management
External evaluators have:
• An independent stance and offer a fresh perspective
• An overview of numerous organisations to serve as a comparison
• Knowledge and experience of a wide range of evaluation techniques
Disadvantages
Internal evaluators may:
• Have a vested interest in a particular outcome
• Sometimes be over influenced by the views of management
• Find it difficult to persuade stakeholders to participate in the evaluation process
External evaluators may be:
• Unaware as to who are the key players in a particular setting and hence more easily misled by
interested parties
• Influenced by the need to secure future contracts
• Insensitive to internal norms and internal relationships
(Source: Adapted from Clarke, 1999)
4
Principles and practice of coaching evaluation
WHAT SHOULD BE EVALUATED – AND WHEN?
In his highly influential work, Kirkpatrick (1959) makes recommendations for
evaluation that have provided the basis for the process to this day. He argues,
essentially, that the evaluation of any training programme should occur at four levels.
• Level 1, Reaction: evaluating the reactions of participants on the programme
(usually by the use of a questionnaire).
• Level 2, Learning: measuring the knowledge, skills and attitudes that result from
the programme and which were specified as learning or developmental objectives.
• Level 3, Behaviour: measuring aspects of improved job performance that are
related to the Level 2 objectives.
• Level, 4 Results: relating the results of the programme to organisational objectives
and other criteria of effectiveness.
Sadly, Bramley and Kitson (1994) suggest that in the UK and USA over 80 per cent
of training is only evaluated at Level 1, with participants commenting on how much
they enjoyed or thought they benefited from the programme. This information is
gathered through the issue of evaluation forms or, in modern jargon, ‘happiness
sheets’. While there is a lack of empirical evidence on this subject, anecdotal
evidence suggests that the evaluation of coaching largely follows similar principles.
However, as Bramley and Kitson (1994) point out, the problems of evaluating at high
levels such as Levels 3 and 4 are not well understood. Measuring changes, for
example, in job performance is difficult, in part due to the amount of work involved in
designing accurate and reliable measurement criteria. So, if the focus of the
evaluation is cost-benefit analysis (see Table 2), exploring the relationship between
the cost of the coaching and the financial benefits, this aim might be worthy but
difficult to calculate. In contrast, other types of evaluation such as effectiveness focus
or impact focus may be easier to measure, provided the measurement criteria are non-
monetary. Nor should it be assumed that Level 4 evaluation is necessarily ‘better’
than the others. A process focus with its emphasis on participants’ experience of the
coaching programme, for example, might be deemed to be at Level 1, but is no less
important for that. Table 2 shows that it is possible to evaluate coaching from a wide
range of perspectives. What is important is that the key questions or approach are
determined in advance of both the coaching programme and its evaluation, so that
evaluators know what they are looking for (as well as being open to surprises and
unanticipated results).
Table 2 Types of coaching evaluation and their defining question or approach
Focus or type of evaluation Key questions or approach
Comparative focus How does the coaching programme compare on specific indicators
with an alternative training or development initiative?
Cost-benefit analysis What is the relationship between the costs of coaching and the
benefits expressed in monetary terms?
Descriptive focus What occurs in the coaching programme? What can be observed?
Effectiveness focus To what extent is the coaching effective in attaining its goals?
5
Principles and practice of coaching evaluation
How can the programme be more effective?
Formative evaluation How can the coaching programme be improved (during its
planning and delivery phase)?
Goal-based focus To what extent have the goals of the coaching programme been
attained?
Impact focus What are the direct and indirect impacts of the coaching
programme on participants, the organisation, the community?
Process focus What do participants experience on the coaching programme?
How can these processes by improved?
Quality assurance Are minimum standards (as, for example, laid down by a
professional coaching association) being provided? How can
quality be monitored and demonstrated?
Summative evaluation What is the overall merit or worth of the coaching programme?
Should it be modified? Should it continue?
(Source: Adapted from Patton, 1984)
Another important focus of evaluation is the administration of the coaching
programme. According to the CIPD (2004a) this could comprise a range of areas,
including:
• The choice of coach. Did the coach have the right range of knowledge and skills?
Were the coaching models used by the coach (for example, psychotherapeutic,
cognitive/behavioural, business) effective and appropriate?
• Choice of coachees. Were the right people chosen as recipients of the coaching?
Are there people who were not chosen but who could benefit from the process?
• Matching process. Did the matching process succeed in matching coachees with a
coach who suited their needs and also the needs of the organisation? How could
the matching process be improved in the future?
• Was coaching the right kind of development intervention? Are other interventions
(such as conventional training courses) more effective or better value for money?
• The coach briefing process. Did coaches understand the needs of the
organisation? How was the briefing process conducted? How could it be done
better?
• How disruptive was the coaching process? Was the coaching fitted into the
normal working day and, if so, did this interfere with work patterns?
• The attitude and support of line managers. Did line managers support the
coaching intervention? Did they allow the coachees time and space in which to
try new strategies? Did they support their learning after the intervention?
In addition to understanding what needs to be evaluated, the timing of evaluation is
also important. It was Scrivner (1967) who first formulated the concepts of formative
and summative evaluation. Formative evaluation (see Table 2) takes place during a
programme so that corrective action can be taken where problems are identified. In
6
Principles and practice of coaching evaluation
contrast, summative evaluation is used to determine the overall effectiveness of a
programme and is conduced when the programme is completed. It is clear from Table
2 that some evaluation approaches, such as descriptive and process focus could be
conducted either during or at the completion of a programme. Most others, however,
appear to be more appropriate at the end.
HOW SHOULD EVALUATION BE CONDUCTED?
The way in which an evaluation of coaching will be conducted will depend, in large
part, on the philosophical approach of the evaluator and the objectives of the
evaluation. Easterby-Smith (1994) classifies evaluation into four school of thought:
experimental, systems, illuminative and goal-free. To these we can add: decision-
making, goal-based, professional review, and interventionist. The role of the
evaluator will be quite different depending on which evaluation process is being
adopted. With more formal ‘scientific’ approaches, the evaluator may often be an
independent external consultant providing feedback to a commissioning organisation.
In constructivist approaches, the evaluator may seek to work in collaboration with,
say, HR professionals or coachees in order to facilitate change. Let us look at each
approach in turn, keeping in mind that some of these are not necessarily mutually
exclusive and may overlap.
RESEARCH
Experimental Illuminative
Goal free
SCIENTIFIC CONSTRUCTIVIST
Systems
Decision-making Interventionist
Goal-based
Professional review
PRAGMATIC
Figure 1 Model of schools of thought in evaluation (adapted from Easterby-
Smith, 1994)
Experimental evaluation
Experimental evaluation seeks to demonstrate that any observed change in behaviour
or outcomes can be attributed to the intervention (that is the coaching programme).
There is an emphasis on research design, and quantitative measurement, the use of
valid and reliable data gathering tools, and sometimes the use of control groups and
7
Principles and practice of coaching evaluation
treatment groups – just as one would expect in any, typical, experimental approach.
So, for example, within a team of 50 company middle managers, 25 could receive the
coaching with the other 25 assigned to a control group who receive no coaching.
Both groups would be measured against a set of skills or attitudes both before and
after the coaching intervention. If coaching is successful, then we would expect those
in the coached group to achieve a greater increase in scores overall than those in the
control. Care would be taken to randomly assign the managers to each group in case
one group did better than another merely because it contained generally more able
managers or better learners.
There are, however, many difficulties and challenges to the experimental evaluation
of coaching. Firstly, there may be practical problems in maintaining experimental and
control groups that are evenly matched both in terms of size and composition. It may
be the case, for example, that, as a matter of company policy, it is only senior
managers who receive coaching, so middle managers have to comprise the control
group. These are hardly comparable groups. Secondly, it may be difficult to find
measurement variables (such as output or even measures of attitude) that are not
affected by factors other than coaching. If a group of coached managers show an
increase on a measurement scale after coaching, how can we say with absolute
confidence that this was due to the coaching alone and not to some other extraneous
factor? There could also be the Hawthorn effect at work – managers show an increase
in scores because of the attention the coaching allots them, not necessarily the
intrinsic benefits of the coaching itself.
Systems evaluation
In a systems approach to coaching evaluation there is an emphasis on identifying
outcomes and on providing feedback on these outcomes to coaches and the sponsors
of the coaching programme. In the systems approach the evaluation process is
planned in advance (specifying the objectives of the evaluation and the sources of
information). Data gathering instruments are piloted and amended and the data then
gathered. Finally, a report is drawn up containing descriptions, analysis and
recommendations. One of the benefits of systems evaluation is its rigour. However,
by assuming that outcomes can be pre-planned, it may miss some of the more subtle
and unanticipated results of the coaching process. Furthermore, as Easterby-Smith
(1994) warns, an evaluation approach that places an emphasis on outcomes, may
represent a mechanistic view of learning. This may be adequate when learning to
interpret a balance sheet, but makes little sense when applied to some of the more
experiential forms of learning, of which coaching is one.
Goal-based evaluation
The main focus of goal-based evaluation is identifying gaps between planned and
achieved goals. In this approach, objectives are usually expressed in behavioural
terms (for example, ‘after the coaching intervention, coachees should be able to
demonstrate the ability to lead a team’). Of course, what is observed in terms of
behaviours, does not necessarily represent what people believe or think. People may
even hold beliefs that they are not consciously aware of or willing to express.
Nevertheless, goal-based evaluation is useful in providing a focus on the measurable
gains from coaching. One of the challenges evaluators face in using decision-making
evaluation as an approach is in reporting unexpected results, particularly when they
might be unpopular or politically embarrassing with senior decision-makers.
8
Principles and practice of coaching evaluation
Decision-making evaluation
As House (1980) points out, one of the dilemmas of evaluation is that it is not always
known who the evaluation results are aimed at. This problem is solved by decision-
making evaluation, since evaluation reports are usually commissioned by, and written
for, top decision-makers and managers. The many large companies, for example, that
are now making use of executive coaching will tend to adopt a decision-making
evaluation stance to help inform them whether future coaching programmes should be
commissioned. As an approach, decision-making evaluation often makes use of
survey methodology, using data-gathering tools such as questionnaires and interviews.
Professional review: validation and accreditation
Professional development programmes leading to accredited qualifications have to go
through a formal evaluation process known as validation. For coaching, a modest
number of such qualifications currently exist. If such qualifications are delivered by a
UK institution of further or higher education, then it is a government requirement that
a rigorous and formal evaluation (validation) process takes place. A panel will be
drawn up which usually contains members from a range of disciplines within the
institution (to encourage objectivity and independent judgements) as well as senior
academics or practitioners from external institutions. The validation process will
consider (evaluate) many factors, including:
• The design of the programme
• The contents of the programme (in the future it is possible that some institutions
will choose to define these contents against the competency standards being drawn
up by the EMCC)
• The resources, such as tutors, needed to support the programme
• Assessment methods and criteria
Given that there are currently few formal, academic coaching qualifications, it is
likely that validation as a form of evaluation will be used increasingly in the future.
Illuminative evaluation
Illuminative evaluation takes a much more open-ended approach. Rather than focus
on measurement and outcomes, it seeks the views of participants, recognising that
there are ‘multiple perspectives’ on any matter under scrutiny. So the illuminative
evaluation of a coaching programme will seek to promote collective understanding
about a programme within an organisation across all stakeholders. In terms of
methodology, it is often associated with the case study approach, and will tend to use
qualitative methods such as in-depth interviews and direct observations of programme
activities. One of the drawbacks of using illuminative evaluation, is that clients and
sponsors may want more than just ‘illumination’ but results that can lead to action.
These do not always emerge from approaches of this kind. Illuminative evaluation,
however, recognises that it is not always possible (or desirable) to evaluate finite
outcomes. By qualitatively exploring participants’ perspectives, illumanative
evaluation may uncover unanticipated outcomes.
Goal free evaluation
In sharp contrast in seeking to evaluate whether the objectives of a coaching
programme have been achieve, goal free evaluation seeks to look of unanticipated
outcomes and, above all, processes. In terms of research paradigms, goal-free
9
Principles and practice of coaching evaluation
evaluation is typical of a constructivist approach, exploring how participants make
sense of their experiences. Ballantine et al. (2000) support this approach, arguing that
evaluation should look less at financial outcomes, and more at subjective views where
personal judgements are made explicit. Certainly, in face-to-face interventions such
as coaching, the complex processes through which learning and knowledge
acquisition takes place requires an understanding of processes as well as outcomes. It
is worth contemplating whether goal free evaluation is most applicable to
interventions such as life coaching where the intended outcomes (to the extent that
they can be planned) are more holistic and less constrained than in, say, more business
orientated coaching approaches.
Interventionist evaluation and action research
Interventionist evaluation concentrates on adapting a programme around the needs of
participants, often on an ongoing basis. The role of the evaluator is not independent
and detached, but involved and committed to initiating change. A widely used
interventionist approach includes action research where a practitioner sets out to solve
a problem through planning, implementing and evaluating a change process. This
change process, of course, could be the coaching intervention itself. An action
research approach to the evaluation of coaching could involve the coach, coachee,
sponsor and other stakeholders in a collective commitment to both the coaching
intervention and the measurement of its impact.
THE TOOLS OF COACHING EVALUATION
According to Clarke (1999), those who conduct evaluation rely heavily on existing
social science research methods and methodologies for planning and implementing
the gathering of data. Hence, a wide spectrum of evaluation processes, sources and
instruments are available. As Table 3 shows, feedback from participants themselves,
is the most common source of data on the effectiveness of coaching interventions
(CIPD, 2004a). However, it is also significant that organisations do not simply rely
on this obvious source, but try to gather data from other sources, some of which, at
least, appear to focus on ‘hard’ data. Most appraisal systems, for example, will seek
to establish a set of observable and measurable objectives to be achieved. However,
in practice even when a set of objectives is met, it will be difficult to determine the
extent to which coaching alone has contributed to this achievement.
Another common source of evaluation data comes from coaches themselves (see
Table 3). It is important, however, that coaches are genuinely objective and
reflective about their coaching intervention. Furthermore, organisations need to
consider what kinds of data gathering methods they can use to elicit feedback from
coaches. One approach is to organise a focus group where, for example, the session is
tape recorded, transcribed and the data analysed. The views of coaches can also be
gathered through a tele-conference, possibly facilitated by an independent consultant,
in the role of researcher-evaluator.
It is interesting, and perhaps of some concern, that Table 3 also suggests a relatively
low priority given to assessment against objectives set at the start of the coaching
programme. The CIPD (2004a) report also shows that only one in four organisations
make use of 360 degree assessment to evaluate the effectiveness of coaching. This
approach is particularly valuable if done on a pre-coaching and post-coaching basis, to
see what changes, if any, the coaching has produced.
10
Principles and practice of coaching evaluation
Table 3 Measures used to evaluate the effectiveness of coaching
Evaluation measure % of respondents reporting using
this measure
Feedback from participants 75
Appraisal system 61
Feedback from coaches 44
Employee attitude survey 41
Exit interviews 38
Assessment against objectives set at the start of a coaching 37
initiative
Business performance indicators 29
360 feedback 25
Staff turnover rates 21
(Source: adapted from CIPD, 2004a)
One potential source of data gathering not mentioned in Table 3 is the use of
reflective journals. Reflective journals provide a medium through which managers
can identify viable topics for concern, and reflect critically on their experiences (Uline
et al, 2004). Used in conjunction with coaching, reflective journals may help
managers to think about their attitudes, beliefs and assumptions in order to promote
self-evaluation. A reflective journal can contain personal anecdotes, stories, or
descriptions of work-related problems, with the contents shared with others (for
example, peers, a facilitator or coach) if the writer seeks feedback or support.
REPORTING EVALUATION RESULTS
Particularly where the coaching has comprised a major intervention, evaluation
reports may have a multiplicity of audiences (Gray, 2004). Especially if the report is
going to be seen by senior managers or directors, it is usually prudent to furnish the
report with an executive summary. The coaching programme’s clients such as the HR
department (if it sponsored the programme) will want to know about its impact and
results. Coachees may be interested in how the programme has affected knowledge
and skills across all recipients. Whoever the intended recipient, it is important that the
report is written with a specific audience or audiences in mind. It should normally
make use of figures and tables for clarity and provide accessible summaries of the
data. Planning the report should allow time for its review (evaluation!), by
independent persons who can identify errors, inconsistencies and misunderstandings.
The reporting of the results of coaching evaluation poses a particular set of
challenges. While the objectives of the coaching intervention may be set by a
sponsoring organisation, the HR department or line managers, the coach-coachee
conversation is (or should be) confidential. If this were not the case, the coach will be
unable to develop the depth of trust needed to make the partnership effective. But if
the conversation is confidential, what can or should be reported? There are no simple
11
Principles and practice of coaching evaluation
answers to this question. One solution is to keep the details of conversations
confidential, but to report themes, issues and developmental improvements at a
general level. This is particularly possible (and beneficial) when several or many
people are being coached and consistency of themes for organisational change
emerge.
A key factor in ensuring that the recommendations in an evaluation report are acted
upon is to ensure that the findings do not come as a shock to the programme’s
sponsors. A number of steps can be taken to ensure that this is likely to occur,
including:
• Involve sponsors in formulating the objectives and the design of the evaluation to
gain their interest and commitment.
• Get commitment from senior organisational stakeholders in advance of the
evaluation so that action will be taken on the basis of results.
• Consider establishing a steering group to monitor and help the evaluation project
and get senior stakeholders onto the group if possible to increase its credibility.
• Keep sponsors informed of outputs (particularly unexpected ones) as they occur.
COACHING EVALUATION IN PRACTICE: SOME INDICATIVE DATA
Some data on how coaches evaluate their practice were collected by the author when
recruiting coaches for a European Social Fund sponsored ‘LeadershipSkills’ project,
led by the University of Surrey. A total of 60 applicants (most from the Surrey area,
or neighbouring counties) completed a detailed proforma, giving details of their
background, knowledge, coaching qualifications, experience and evaluation practises.
Permission was sought for using these data for research purposes and granted in all
but two cases.
Results indicated that, not unexpectedly, most coaches discussed a clear set of goals
with coachees at the outset of the coaching process (recalling Figure 1, a goal-based
approach). In three cases, these goals were defined in terms of SMART objectives
(specific, measurable, achievable, relevant, and time-based), but even where this
framework was not specified, the need for measurement or quantification was a clear
focus. Measurables were seen as important for logging achievement ‘so we know it
when we see it!’ Sometimes goals were set just by the client or sponsor, but in other
cases a three way meeting (the coach, coachee and sponsor – often a line manager)
would determine objectives, usually before the coaching and sometimes afterwards as
well.
Yet, precisely how one measures these outcomes is a more complex process. One
respondent alluded to the need to measure pragmatic, tangible business benefits and
the ‘high return on investment’ although no guidance was provided on how these
would be calculated. Others, however, pointed to the collection of evidence about the
manager’s improved performance from peers and colleagues or the acquisition of a
new position or promotion as a tangible indicator. One coach asked the company for
feedback on the coachee’s performance pre and post coaching, whilst another
contacted the organisation six months after the coaching had been completed, to see if
improvements in performance had been sustained. For a number of coaches a simple
way of measuring whether their coaching was successful, was the number of referrals
from networks of clients.
12
Principles and practice of coaching evaluation
The timing of evaluation was a constant but variable theme, some coaches being
content on post-coaching evaluation, but others constructing a more iterative process,
one case comprising evaluation at the end of each session, a review session at the 3-6
session mark, and a review and closure session at the last meeting (incorporating two-
way feedback).
It was clear from responses that some coaches elicited feedback from coachees and
clients through informal dialogue. In some cases, however, it was not easy to
establish from the responses precisely how feedback was gained. The exception to
this were the nine responses that specifically mentioned the use of post-programme
questionnaires. In some cases, these were used for ‘checking in percentage terms
coachee’s satisfaction with the achievement of goals’, while another respondent
described how he produced an assessment sheet for every session, asked the coachee
to make a numeric assessment to decide which of the coaching objectives that session
contributed to, and to suggest any ways that the coaching might be improved. Four
respondents mentioned making use of 360 feedback between coachees and their line
managers or colleagues at the end of the coaching programme, or, in one case, both
pre and post programme. However, the fact that the majority of respondents made no
mention of formal evidence-gathering tools, suggests that their approach is less formal
than the above discussion suggests. This is in line with the CIPD (2004a) findings.
Three coaches talked about evaluating their coaching using the services of their
supervisor ‘to make sure the coaching relationship is healthy and productive and that I
have done my best to help my client’. For some coaches, this supervision session was
undertaken on a monthly basis and, in addition to other functions, was used to reflect
on a range of coach-coachee relationships, exploring what worked well and how
personal coaching skills could be strengthened. ‘From my perspective, there is
always more to learn …. I particularly enjoy the process of making things happen, re-
kindling motivation and energising renewal in others. In general, though, this
reflective, self-evaluation was the exception rather than the norm.
While no claims for the representative nature of this study are intended, the data
described here may be indicative of some general trends. If this is the case, then the
current approach to coaching evaluation seems somewhat traditional in scope and
limited to a narrow set of paradigms – largely around systems and goal-based
evaluation. While these approaches have the benefit of trying to measure what has
been achieved, as Easterby-Smith (1994) cautions, they may serve to miss or ignore
both the less tangible outcomes of coaching and its processes. Indeed, with some
exceptions, the data revealed quite significant silences on the interactive process
between coach and coachee. Furthermore, while many evaluation processes laid
emphasis on objective rigour, none made reference to any kind of formal evaluation
report or the use of independent evaluators. This, however, may have been partly due
to the narrow focus of the questions asked. Clearly, further research in this area
would be valuable.
CONCLUSIONS & RECOMMENDATIONS
Many claims are made for the effectiveness of coaching, but these often emanate from
interested parties such as individual coaches or coaching organisations. If coaching is
to achieve the level of credibility the coaching profession and the purchasers of
coaching need, better designed evaluation is needed to ‘prove the case’. As we have
seen, what is evaluated, why and by whom is problematic and will be influenced, in
13
Principles and practice of coaching evaluation
large part, by the requirements of sponsoring organisations. Currently, however, the
most popular approach to evaluating coaching is by eliciting feedback from
participants. This is very much at Kirkpatrick’s (1959) Level 1, measuring the
reaction to a programme. For coaching to move up the credibility scale, some of
Kirpatrick’s higher levels need to become coaching evaluation targets, particularly in
terms of changing behaviours and results. We can see from the small scale study
discussed above, some coaches clearly recognise this and are acting on it.
It may prove easier, however, to measure behaviour as the unit of analysis (the
behaviour under review) because it is more easily observed and measured. In
contrast, the ways in which coaching impacts on tangible organisational results (such
as managerial performance and other measures) is more problematic. This is largely
because of design difficulties in shielding out extraneous variables (events or
influences other than the coaching) that may also have an effect on results. As we
have seen, one way of controlling for these extraneous variables is through an
experimental design. To date, however, few if any of such studies have been
undertaken. The reasons for this are not hard to find. Assigning people to
experimental and control groups may be politically unpopular in organisations,
particularly amongst those in the non-coached control! Furthermore, given that the
coach-coachee relationship often involves the negotiation of desired outcomes, it may
be impossible to compare these diverse outcomes across groups.
We are probably left then, with evaluation approaches that focus on whether agreed
outcomes have been achieved. What is essential, however, is that the evaluation of
coaching is inclusive of the intentions and motivations of all stakeholders – coaches,
coachees, line managers and sponsors. The small scale study described above shows
that this is certainly happening in many cases. Furthermore, given the complexity of
the coach-coachee relationship, evaluation should not just focus on measuring
reactions to coaching (as in many evaluation approaches) but also on the deep,
psychological and social processes that underpin the coaching journey. This requires
a reflective stance on the part of the coach and coachees, an approach adopted by only
a small number in the small scale study reported above.
While any set of recommendations cannot be definitive, it is suggested that individual
coaches and the professional coaching associations and networks pay more attention
to the issue of evaluation. In particular, it is recommended that evaluation of
coaching should:
• Be done as a matter of course and be pre-planned and the processes agreed, before
the coaching takes place
• Include ethical considerations including who should have access to evaluation
data
• Be undertaken on a collaborative basis by all relevant stakeholders, including
coachees, coaches and sponsors, so that all perspectives are acknowledged
• Include goals or outcomes that are negotiated between the stakeholders,
recognising that these may change and be re-negotiated during the coaching
process
• Ensure that, as well as tangible outcomes, the focus includes interactive processes
and the coaching relationship
14
Principles and practice of coaching evaluation
• Include a focus that goes beyond the coaching relationship to include the selection
of coachees and the management of the matching process
Word count: 6108
References
Ballantine, J., Levy, M., Martin, A., Munro, I. and Powell, P. (2000) ‘An ethical
perspective on information systems evaluation’, International Journal of Agile
Management Systems, 2/3 233 – 241.
Bramley, P. and Kitson, B. (1994) ‘Evaluating training against business criteria’,
Journal of European Industrial Training, 18(1) 10-14.
Campbell, C.P. (1997) ‘Training course/program evaluation: Principles and practice
Journal of European Industrial Training 22(8) 323-44.
CIPD (2004a) Coaching and buying coaching services London: CIPD.
CIPD (2004b) Training and Development 2004. London: CIPD.
Clarke, A. (1999) Evaluation Research: An Introduction to Principles, Methods and
Practice. London: Sage.
Easterby-Smith M. (1994) Evaluations Management Development, Training and
Education. Aldershot: Gower Publishing.
Gray, D.E. (2004) Doing Research in the Real World London: Sage.
Hay, J. (1995) Transformational mentoring London: Sherwood Publishing.
House, E,R, (1980) Evaluating with Validity Beverley Hills, CA: Sage.
Kirkpatrick, D.L. (1959) ‘Techniques for evaluating training programmes’, Journal of
the American Society of Training Directors 13(3-9) 21-6.
Lincoln, Y.S. and Gubba, E.G. (1985) Naturalistic Inquiry. Newbury Park, CA:
Sage.
Patton, M.Q. (1984) ‘Data collection: options, strategies and cautions’, in L. Rutman
(ed.) Evaluation Research Methods: A Basic Guide, 2nd edn. Newbury Park, CA:
Sage.
Scriven, M. (1967) ‘The methodology of evaluation’ in R.W. Tyler, R.M. Gagne, and
M. Scriven (eds) Perspectives of Curriculum Evaluation Chicago, IL: Rand
McNally.
Uline, C., Wilson, J.D., D., Cordry, S. (2004) ‘Reflective Journals: A valuable tool for
teacher preparation’ Education 124 (3)
15
