Evaluating social work supervision

INTRODUCTION: The question of whether the practice of professional supervision is effective, and how its effectiveness can be measured, has been debated by both social work and other professions. This study explored how practitioners, supervisors and managers in Aotearoa New Zealand currently evaluate the supervision they receive, provide and/or resource. The study was interprofessional involving counsellors, mental health nurses, psychologists and social workers. This article focuses on the findings from the social work cohort. METHODS: Through an on-line Qualtrics survey participants were asked: 1) how they currently evaluated professional/clinical supervision; and 2) how they thought professional/clinical supervision could be evaluated. Data were extracted through the Qualtrics reporting functions and thematic analysis was used to identify themes. A total of 329 participants completed the survey of which 145 (44%) were social workers. FINDINGS: A majority of the social work participants reported that they evaluated supervision in some form. No culture or policy emerged regarding supervision evaluation, but social workers expressed interest in training and resources to assist evaluation and some saw a supportive and endorsement role for the professional or regulatory bodies. An unexpected finding was reports of unsatisfactory and harmful supervision. CONCLUSION: Evaluation of supervision is an activity with which social workers engage, but further research is needed to explore how evaluation can be embedded in supervision practice. More critically, a broader audit is required to reconsider the definition and model of social work supervision in Aotearoa New Zealand and the environments within which supervision occurs.


QUANTITATIVE RESEARCH
. Here scholarly debate wrestles with questions of what should be evaluated in supervision and how that evaluation should take place. The focus of evaluation to date has been largely on the benefits of supervision in three areas: benefits to the supervisees, benefits to the organisation and benefits to the clients. Examining research publications on the effectiveness of supervision of child welfare workers between 2000 and 2012, Carpenter et al. (2013) found benefits to supervisees of "job satisfaction, self-efficacy and [protection against] stress" whilst the organisations benefited through "workload management, case analysis and retention" (p. 1843). Likewise, Watkins (2011) in a review of 30 years of psychotherapy research found that supervisees gained through "enhanced selfawareness, enhanced treatment knowledge, skill acquisition and utilization, enhanced self-efficacy, and strengthening of the supervisee-patient relationship" (p. 236).
Whether supervision is of benefit to clients, however, is more difficult to determine. Carpenter et al. (2013) concluding that: "the evidence for its [supervision's] effects on workers' practice is weak" (p. 1851), whilst Watkins' (2011) earlier review reported that "the drawing of any conclusions about supervision's effects on patient outcome seems premature" (p. 236). Overall, the literature reports a lack of reliable measures by which supervision can be evaluated. The 49 scales and measures identified by Wheeler and Barkham (2014) as designed for this task, are testament to the energy focused on this area, however, the validity of these tools and measures and the research surrounding them has been questioned (Bernard & Goodyear, 2009;Carpenter et al., 2013;O'Donoghue & Tsui, 2013;Watkins, 2011;Wheeler & Barkham, 2014).
More pertinently, it has long been regretted (Grauel, 2002;Milne, 2007) that there is no agreed multi-professional definition of supervision and, as noted above, existing definitions and practice reflect differing emphases on factors such as risk, compliance, learning, development and support. Falender (2014), a champion of competency based supervision, argues that, before any outcome assessment can take place, preliminary steps need to be taken. "The entire process of supervision is acutely in need of understanding and developing empirical support for its components and impacts" (p. 143). Falender concludes that, "to study outcomes of supervision, the ingredients of effective supervision are essential" (p. 145).
What constitutes effective, or more specifically inadequate and harmful supervision, was explored by Ellis et al. (2014). With reference to the required standards for accreditation and licensure and to the "guidelines and standards for clinical supervision" of a number of different professions, Ellis et al. (2014, p. 439) developed a list of "criteria for minimally adequate clinical supervision" across disciplines.
Harmful supervision was considered to include those situations where action, or inaction, on the part of the supervisor was known to cause harm.
In subsequent research, Ellis, Creaner, Hutman, and Timulak (2015) conducted a study of supervisees from a range of professions who worked in either the Republic of Ireland (RI) or the United States (US). In this cross-national study, the professional affiliations of both cohorts, Irish and American, were similar. Three types of supervision were explored inadequate supervision (IS), harmful supervision (HS) and exceptional supervision (ES). These categories were rated by two scores: selfidentified (SI) and de facto (DF). SI scores were those reached by the supervisee when considering supervision activity in the light of a definition of IS, HS or ES. DF scores involved a third party, matching aspects of the supervision described against external criteria, some of which derived from professional or legal requirements. An interesting discovery from this research was QUANTITATIVE RESEARCH

ORIGINAL ARTICLE
that, despite the national differences and the fact that the US group were trainees and the RI group were predominantly post-qualified practitioners, "no differences emerged in the high occurrence rates of IS, HS, and ES between countries" (Ellis et al., 2015, p. 628).
Closer examination of the scores for ES however, revealed a disquieting finding which highlights the subjective and personal elements of evaluation and the complexity of the exercise. Ellis et al. (2015) noted that "more than half of the Republic of Ireland and U.S. supervisees reported receiving [self-identified exceptional supervision] SIES from their current supervisors." They continue however, observing "that the findings that supervisees reporting SIES were also categorised as currently receiving [de facto inadequate supervision] DFIS (Republic of Ireland: 79%, United States: 70%) and [de facto harmful supervision] DFHS (Republic of Ireland: 40%, United States: 25%) somewhat contradicts this conclusion" (p. 629). These findings, Ellis et al. describe as "substantial discrepancies between supervisees' perceptions versus more behavioral-based, objective criteria of the inadequate or harmful supervision they received" (2015, p. 629).

The Aotearoa New Zealand study
Whilst there have been studies evaluating supervision in localised settings, for example O'Donoghue (2016) and Rains (2007), to date in Aotearoa New Zealand there have been no comprehensive studies evaluating supervision in any profession. The focus of this present study however, was not to evaluate supervision per se, but rather to explore the ways in which supervision is currently evaluated by those most closely involved: supervisees, supervisors and managers.
The research reported here is an interprofessional study involving four professions: counselling, mental health nursing, psychology and social work. The study was designed to explore and document the current status of supervision evaluation in Aotearoa New Zealand, to identify issues, concerns and possible gaps and to make appropriate recommendations. Participants were also asked to comment on what they considered to be ideal or best practice, for the evaluation of supervision.
This article reports and discusses the responses of the social work participants to these questions and considers important issues which were raised.

Methodology
The study employed a sequential design which used a range of methods within a qualitative research methodology. Stage one comprised semi-structured interviews, conducted with 24 experienced practitioners from the four professions, which explored how evaluation of professional supervision was understood and actioned in practice. Following the analysis of the data from these interviews, the findings of which have been reported elsewhere (Davys, O'Connell, May, & Burns, 2017), an online Qualtrics survey was developed (stage two). The design of the survey reflected and incorporated the content and conversations of the stage one interviews. The study has the approval of the Waikato Institute of Technology Human Ethics Committee.

Sample
In November 2015, participants were invited to respond to an online Qualtrics survey regarding their experiences of evaluation of professional/clinical supervision in Aotearoa New Zealand. Invitations were sent electronically through the respective professional network communications and publications. Social workers were alerted to the research through the Aotearoa New Zealand Association of Social Workers (ANZASW) website and e-notices. A total of 329 (N) participants (see Table 1) provided 344 (n) responses, thus indicating that 15 participants were affiliated to more than one of the identified professions. Of this group social workers formed the largest group, comprising 44%.

Data collection and analysis
Data were collected via the Qualtrics online survey where a total of 45 questions were asked. The survey compromised three parts: part one Demographics, part two Current Practice, part three Best Practice (future ideals). Parts one and three were completed by all participants while in part two, managers, supervisees and supervisors answered separate sections according to their role(s).
The results function of the Qualtrics software was used to prepare a report of the responses to all questions in the survey. The data contained in the reports were reviewed independently by the researchers and emergent themes identified. In this thematic analysis (Braun & Clarke, 2006) the themes were compared and agreed by all researchers. Responses were then coded and cross-checked to ensure consistency. Subsequent filters were applied to the data to select the responses specific to each profession. The 145 responses, specific to social work participants, form the basis of this article.

Demographics
In order to understand a range of perspectives, participants were asked to group themselves according to role: supervisee, supervisor and manager. The experience of interviewing the experienced practitioners in phase one had highlighted the fact that many practitioners held more than one role. Participants were accordingly invited to respond to as many roles as were applicable. A total of 145 (N) social work participants provided 206 (n) responses to this question thus demonstrating that a large number of dual roles were held by the participants. The profile of the social work participants in this research is presented in Table 2.

Findings
The tables and data presented in this section report four sets of social work responses from the survey: supervisors, supervisees, managers (part two) and best practice (part three). Best practice responses required participants to indicate what they thought was the ideal practice in relation to the questions asked in part two of the survey. These responses were not separated according to role, and thus reflect an overall social work perspective. Many questions invited participants to select as many responses as applied. With one exception, the organisation of the data in the following tables reflects the order in which the participants were asked to respond in the survey. Table 5, however, which identifies What is evaluated, records the responses in descending order according to best practice scores.

Type of evaluation
Participants were provided with two definitions of evaluation, outcome and process, and asked what type of evaluation was employed in the supervision with which they were engaged. The following definitions were provided: Outcome evaluation is concerned with understanding the overall effectiveness or impact of a programme or service.
Process evaluation is concerned with understanding the means or process, by which the programme is being implemented. (Fox, Martin, & Green, 2007, p. 67) The question allowed five choices of response (see Table 3). The responses of supervisors and managers to this question indicate that evaluation of some sort is occurring in social work supervision, with the supervisors clearly favouring process evaluation, closely followed by a combination of process and outcome. This combination was also reported by 45% of managers and 33% of supervisees. It is interesting that 37.8% of supervisees reported that no evaluation was taking place. The best practice score indicated a clear preference, 92.8%, for combined process and outcome evaluation.
Participants who did not evaluate supervision were asked to comment on why this did not occur. Two managers responded to this question. Neither knew why there was no evaluation, one adding "have not been asked myself." The 12 responses to this question from the supervisors fell into two categories. A majority of the supervisors, nine, reported that the reason that no evaluation occurred was because there was no expectation or requirement for this from the employer. Some, like the managers, were unsure why this was: Can't say I know -this has never been discussed with me by my employer and I haven't raised this with my manager.

QUANTITATIVE RESEARCH
Others saw it as a reflection of the way in which supervision was understood and valued by their organisation: Because there's no form of measurement or protocols requested by management to monitor effectiveness. Attending supervision is a requirement, whether it works or not doesn't seem to matter.
One supervisor reflected on the difficulty of maintaining confidentiality and managing the power dynamic: I expect that's because it's considered confidential, and to evaluate my process might require an evaluator to know the content. … I ask for verbal feedback from supervisees, but because of the inherent power dynamic, it could be difficult for most to say if there's anything that they don't like.
Forty supervisees provided reasons why their supervision was not evaluated. As with the supervisor group, over 50% of supervisees reported that they had never been asked to evaluate supervision; it was not a requirement and/or that they did not know why it was not evaluated. The supervisees also commented on the lack of value placed by some organisations on supervision and a focus on performance indicators: The organisation appears not to know what clinical supervision is, and to hold little value for [it]. There is a focus on administrative supervision to ensure KPI achieved, supervisors mostly untrained, do not understand or provide clinical supervision, therefore appear to see no reason to evaluate what they do provide, or its impact on practice.
For others there was a belief that evaluation was pointless as no change would occur: Sometimes I give verbal feedback about how the process is for me, but most supervisors are fixed in their own patterns, so you just make the most of it really. And: I just get told what to do and how to do it and questioned why something hasn't been done. What I think isn't granted any importance. And: There is no evaluation because of the culture within our agency. Social workers' reflections about anything in-house are stifled. If shared, the social worker is unpopular and usually doesn't stay long.

Frequency
Evaluation was reported as most commonly occurring annually, and 41% of participants saw this as best practice. A number of supervisors (36.7%) and supervisees (27%) reported evaluating on a session-by-session basis and 27% supervisees also evaluated

Method of evaluation
The responses to the question of how supervision was evaluated suggest that more than one method is used. By far the most common current method of evaluation (see Table 4) was an informal discussion between supervisor and supervisee (74.6% supervisors; 75.6% supervisees; 30.0% managers), followed by evaluation at time of the review of the supervision contract (67.8% supervisors; 43.6% supervisees; 50% managers) while 47.5% of supervisors, 39.7% of supervisees and 10% of managers reported that focused feedback occurred between supervisee and supervisor. Evaluation happening in threeway conversations between the supervisee, the supervisor and the manager were reported by 20% of managers and 18.6 % of supervisors but this was the experience of only 6.4% of the supervisees.
Best practice scores indicated preferences for focused feedback between supervisee and supervisor (75.8%), at the time of contract review (71%), informal discussion between supervisor and supervisee (59.4%) and documented review (57.8%). There was some, but less clear, support for more formal types of evaluation: 46.9% indicating preference for a questionnaire; 39% for a rating scale; and finally 36.7% for a checklist to guide evaluation.

What is evaluated in supervision?
Participants were provided with a list of possible areas for evaluation in supervision and asked to identify what they currently evaluate (see Table 5). A similar list was used to indicate best practice. The top best practice score (90%), was in relation to evaluating the impact of supervision on the supervisee's practice. Over current evaluation, supervisors and supervisees were in agreement that whether reflection is occurring in supervision was the most frequent focus of evaluation.
Interestingly, cultural considerations were amongst the lowest scores for all groups, including best practice. When later asked what cultural considerations need to be embedded in any evaluation of supervision, participants however, had clear recommendations. A majority of the comments focused on the importance of evaluating whether cultural needs, in the broadest sense, were being met. Culture and difference, they noted, should be acknowledged, respected and part of the supervision conversation. Where necessary, it was also important for external resources to be available:

QUANTITATIVE RESEARCH
How is difference identified, discussed and addressed within the supervision relationship. Recognition that crosscultural supervisor relationships may need to be augmented with cultural support/supervision for the practitioner and supervisor.
The supervisees' safety and competence to practice were also important: The ability of supervision to assist in the development of a supervisee who can effectively work cross-culturally.

Who gets the information?
When asked who had access to the evaluation information in Table 6, 86.4% of supervisors, 83.6% of supervisees and 54% of managers groups said that it was kept within the supervision relationship. Somewhat confusingly however, and in contradiction, 30.5% of supervisors also said that the supervisee's manager had access to this information. The best practice score overwhelmingly supported the information being kept in the supervision relationship (92.9%) but again, confusingly,

ORIGINAL ARTICLE
the next highest score (44%) suggested the information should also be available to the supervisee's manager. It is, however, possible that an explanation for the confusion noted above is that it refers to situations where the supervisor holds a dual role and so is both the supervisee's supervisor and manager.

Reason for current evaluation
The opportunity to enhance the supervision relationship through mutual giving and receiving of feedback was the primary motivation given for the current practice of evaluation by both supervisees (64%) and managers (80%). Supervisors (88%), on the other hand, said they evaluated supervision because it was good practice to do so and because they wanted feedback on the supervision they provided (83%). Providing feedback to the supervisee was less important to all groups but nevertheless was rated in the top four reasons.

What would help?
When asked what might assist in the evaluation of supervision, 70 social workers responded and there was evident interest in accessing a process and/or structure for evaluation. Over half of the participants indicated that they would, or could, benefit from: training or a guide to an evaluation process; a checklist; rating scale; or a formalised outcome measure or tool. The need for evaluation to be a topic of discussion, embedded in the supervision process and/or addressed at an organisational policy level, was also identified. Several social workers saw a key role for the ANZASW and/or the Social Worker's Registration Board (SWRB).
It would be good if this were in some form of policy by ANZASW or SWRB with a variety of tools that could be used. This would ensure organisations have to support/enforce this process; highlight the value of clinical supervision as safe and ethical best practice; and ensure that supervision is a valuable process for those engaged in the process. It would help to provide a guideline to measure effectiveness of supervision rather than supervisees experiencing poor supervisory relationships/process and for supervisors having difficulty with engagement from supervisees.

Other comments
At the end of the survey, participants were invited to add any further comments which they wished to make. A range of themes were covered in the 53 responses received. Some expressed appreciation of the research which had prompted a rethinking of evaluation in practice: A very thought-provoking survey, thank you, I will reconsider my evaluation tools.
Participation in this research has made me aware of the importance of formal evaluation in supervision!!!!

QUANTITATIVE RESEARCH
Others affirmed the importance of evaluation as a means for motivating growth and development for supervisors and of ensuring that supervision was meeting supervisee needs.
I think regular evaluation would be a good idea, as it would inspire supervisors to do continuing professional development and make sure that they're meeting the needs of the supervisee, rather than going by rote and collecting a cheque. Also, it could help managers know if there was a mismatch between supervisor and supervisee and supervisees could be encouraged to change supervisors and get someone who better suits their needs. For supervisors, it could be a [challenge] to continually grow and improve.
Suggestions were offered with regard to evaluation: I wonder if there would be value in having practitioners' supervisors also listed [on publicly available registration lists]. This would empower the public and also help ensure that practitioners maintained supervisory relationships as required via registration.
Of particular concern however were comments which reported bad supervision experiences. Supervisees commented that individual requests and initiatives to meet their needs had been blocked: Supervisees reported that they felt unsafe both within the relationship and within the work environment. In these situations, supervisees said that their fear of the consequences to themselves, and sometimes their supervisor, prevented them from providing honest feedback: Even if I had the opportunity to evaluate supervision, I would be concerned about how that information would be used by my team leader and/or manager … many of my colleagues also have similar feelings, however also fear repercussions if they speak out.
As I am required to attend supervision, I have no other choice than to attend once a month, and say as little as possible in order to keep myself safe.
I would like a more supportive work environment within management. I currently do not feel safe to disclose the poor supervision I am receiving.
….the supervisors get hauled over the coals by managers if cases go bad, or time frames are not met. This stressor/ pressure to work faster, work efficiently is passed on to the supervisee by their supervisor. When the supervisee is overwhelmed with cases, they may get behind in visits, recording and reporting. The more the supervisee "fails" the more pressure the supervisor places on them. It is a very top down approach.
Of greater concern were reports by supervisees that they had been bullied: My supervisor regularly bullies me, and I do not know where I stand with her. She is inconsistent in her supervision approach, and I often leave supervision feeling confused and vulnerable.
I attend supervision with my team leader out of requirement, not by choice. I actually dread it. I find it both patronising and sometimes punitive.

ORIGINAL ARTICLE Discussion
The findings presented in this article, collected from 145 social work participants, have provided a snapshot of how evaluation of supervision is experienced and practised within the social work profession in Aotearoa New Zealand. Although there was, at times, agreement between the three groups (supervisees, supervisors and managers) about the practice of evaluation, differences were also evident. It is important to note that it is not possible to determine if any of the participants were in supervision partnerships together. All responses have therefore been considered as relating to separate and independent supervision relationships and experiences.
Overall, the findings indicated that social workers do evaluate supervision to some degree but there was no evidence of a culture of evaluation of supervision nor of any organised approach. Only three social work participants named specific evaluation tools for supervision but did not name any developed specifically for social work. Interestingly, although over 80% of supervisors and managers described some form of evaluation, evaluation was reported by only 65% of supervisees. Whilst many social workers appeared content with their current method of evaluation, 70 social workers (48%) contributed suggestions regarding ways in which this could be assisted. These suggestions, which included requests for specific resources and training, also favoured a systematic approach and identified a co-ordinating role from an external body such as the ANZASW or SWRB.
Evaluation of supervision was not on the agenda for some participants and the common reasons given by these 63 social workers was that it was not required or had not been suggested. It is unclear whether these responses, which convey a degree of passivity, reflect personal views of the status of supervision or a lack of agency and autonomy experienced by the supervision participants. Participants in this survey not only provided detail about how supervision was evaluated, but also offered an account of their supervision experiences. This unexpected and unsolicited information comprised two types of response. The first recorded expressions of appreciation of existing supervision arrangements, supervision relationships and current modes of evaluation.
Of concern, however, was the second group of responses. Here both supervisors and supervisees described organisational cultures where supervision was not valued nor, at times, understood. Consistent with other reports (Beddoe, 2010;O'Donoghue, 2015) supervision was described as a process for control where compliance with management priorities and work targets shaped supervision agendas and relationships. Participants noted that professional, regulatory and other policy requirements ensured that supervision took place, but the actual quality of supervision was considered irrelevant and supervisees believed that their needs were regarded as unimportant. Sometimes the organisational culture itself was described as toxic and a failure to meet work targets was seen to have negative consequences for both supervisors and supervisees. Threads of cynicism, resignation and distrust were scattered throughout these responses and at least three accounts of bullying were reported. The importance of safety within supervision relationships has been emphasised in other studies (Beddoe, 2010;O'Donoghue, Munford, & Trlin, 2006) and lack of safety is a component of Ellis et al.'s (2014) inadequate and harmful supervision.
Whilst it is important to acknowledge that the participants' comments reflect only one side of the relationship, this does not minimise the distress expressed in these statements. In the workplaces described above, any evaluation can be risky. This is compounded when the supervisor is also the team leader or line manager.

QUANTITATIVE RESEARCH
Supervisees reported accessing and paying for external supervision to avoid toxic inhouse supervision while others said that they were blocked from this option and choice of supervisor was denied to others.
How to address situations such as those described is difficult. It is evident from these reports that feedback and discussion within the supervision relationship is not an option.
Nor does it appear that appeal to higher management would be either productive or safe for many of these supervisees. External independent evaluation is a possibility, but who would oversee it, where would the information go and what authority and status would such evaluation have? Social work's tradition of in-house, linemanagement supervision where social workers have limited choice of supervisor, further compounds evaluation, at times seeding confusion between evaluation of supervision with evaluation of the supervisee. For evaluation of supervision to be useful and effective, rather than another process of tick-box compliance, the social work profession needs to address some of the underlying attitudes, practice and organisational cultures which impact on supervision.
In 2005

ORIGINAL ARTICLE
the twenty-first century, also advocates for review. Whilst recommending a mapping of supervision practice and an evaluation of "the effectiveness of supervision in relation to client, worker, agency and professional outcomes," he proposes a revisiting of "the definition, theory, practice and research evidence pertaining to social work supervision" (p. 146).
Is it time for social work to confront the issue and finally separate the organisational from the professional in supervision? To uncouple managerial from educative and supportive supervision, as suggested by Payne (1994) over 20 years ago, and to explore the long-promoted portfolio model of supervision (Beddoe & Davys, 2016;Garrett & Barretta-Herman, 1995)? This model, as O'Donoghue (2015) notes: "marks a change from supervision occurring solely within an organization by a hierarchical line supervisor, to a mixed provision involving both organizational and professional supervisors" (p. 143).

Limitations
The limitation of this study is that it reports the views and practice of a small sample of social workers in Aotearoa New Zealand and, as such, is merely a glimpse of current supervision practice and evaluation. The views of 95% of social workers who are members of ANZASW have not been heard. The reasons for this lack of response are a matter for conjecture but could include such factors as disinterest in supervision, lack of knowledge about supervision and evaluation, and the all competing pressures from workload and work stress. Further, a majority of the participants, 75%, who completed the survey had some form of supervision training. This, in turn, raises the possibility of sample bias. By reflecting the views of social workers who already have an interest in, and knowledge of, supervision practice, the research may have gathered an informed critique of social work supervision practice but may not have recorded the views of those less engaged with supervision.
Also, in an attempt to capture a broad understanding of evaluation in supervision from a range of perspectives, managers, supervisors and supervisees were invited to participate. Through appreciation of the possible multiple roles which individuals held, they were encouraged to respond from whichever combination of roles was relevant. Similarly, many questions invited participants to check as many options as were applicable. While this provided rich data, it also possibly obscured clear trends and responses to some questions.

Conclusion
This research has provided a window into the practice of supervision for social workers in Aotearoa New Zealand. The shape of current evaluation of supervision was identified for this group of social workers and a profile of best evaluation practice was described. For many participants the survey raised awareness and provided ideas for change. Education, resourcing and guidelines were identified as useful means by which evaluation could be supported and enhanced. Other responses however, report inadequate and harmful supervision which fails to address the professional needs of those social workers and where social workers struggle within toxic organisational cultures and abusive relationships.
Existing social work models of supervision have been described as outdated (O'Donoghue, 2015) and this article not only calls for further research into the evaluation of social work supervision, but also supports the call for a comprehensive audit and review of supervision practice. To ensure a model which is responsive to the complexities of the social work task, the organisational contexts and the needs of social workers themselves, the boundaries of social work supervision in Aotearoa New Zealand need to be determined and defined.