Effectiveness of a pilot school-based intervention on improving scottish students’ mental health: a mixed methods evaluation

Mental health di ﬃ culties are prevalent among children and young people and there is strong evidence-base for the e ﬀ ectiveness of school-based programmes in promoting mental health and resilience; however, there is a need for further trial and evaluation of such programmes in di ﬀ erent contexts. The present study aims to evaluate a pilot mental health pro- gramme, SafeSpot, trialled in a secondary school, assessing the impact of the programme on students ’ mental health as well as teachers ’ opinions and experiences. A mixed-methods design was employed. Results revealed sig- ni ﬁ cantly lower self-reported emotional distress post-intervention compared to pre-intervention. Teachers ’ overall experiences were positive, supporting the delivery of mental health content in schools and sharing perceived limitations and recommended adaptations to programme content. Preliminary ﬁ ndings are promising and further research into the programme is recommended. Mixed methods research is a useful tool for the evaluation of school-based interventions, and should be considered by researchers undertaking such projects.


Introduction
Mental health disorders in children and adolescents are widespread, with an estimated prevalence of 20% in the developed world (Patel, Flisher, Hetrick, & McGorry, 2007). Longitudinal research suggests that emotional and behavioural difficulties in childhood and adolescence are predictive of mental health disorders in adulthood (Roza, Hofstra, van der Ende, & Verhulst, 2003). Mental health difficulties in childhood are also associated with other adverse outcomes, including academic failure, substance misuse and violence (Fazel, Hoagwood, Stephan, & Ford, 2014). Despite the growing evidence for the effectiveness of mental health interventions, 70-80% of children and young people experiencing mental health difficulties do not access support or treatment (Farrell & Barrett, 2007).
Several barriers to help-seeking have been identified in a systematic review, including difficulty accessing specialist services (due to time, transport issues and cost), lack of mental health awareness and fear of stigmatization (Gulliver, Griffiths, & Cristensen, 2010). Therefore, a key challenge for professionals working within child and adolescent mental health is to develop effective, accessible and innovative ways to provide mental healthcare that can overcome or reduce the barriers to care.
Schools are increasingly recognized as a key setting for the provision of mental health support, screening and intervention (Caan, Cassidy, Coverdale, Nicholson, & Rao, 2014;Greenberg, 2010). Research outlines a positive influence of school prevention programmes on a wide range of social and academic outcomes (Greenberg, 2010). Evidence from key randomized control trials and systematic reviews supports the adoption of a universal preventative approach, based on the demonstrated impact on a range of outcomes including academic achievement, depression, anxiety, behaviour, bullying and coping skills (Barrett, Farrell, Ollendick, & Dadds, 2006;Weare & Nind, 2011). A universal school-based approach involves provision of mental health interventions to all students, regardless of risk-status, therefore overcoming many barriers to help-seeking. This is in contrast with targeted or indicated approaches that target young people at higher risk or those who are exhibiting signs of mental illness. Universal approaches can have a significant impact on the trajectory of children and young people's mental health by reducing early symptoms and increasing their ability to deal with adverse events (Adi, Killoran, Janmohamed, & Stewart-Brown, 2007). Universal approaches are attractive to schools because they can be delivered to large numbers of children, are less intrusive and can be straightforward to implement when integrated into the curriculum (Stallard, 2010).
However, critics of universal approaches have questioned whether delivery of interventions to children who do not require them is the best use of resources. Additionally, well-conducted and properly controlled trials are rare and generally, the methodological quality of evaluations is poor. Many studies fail to take into account mediating variables that may affect outcomes and sample sizes are generally small (Stallard, 2010). Therefore, researchers have called for further trial and evaluation of such interventions in real-world settings in order to identify facilitators and barriers in different school contexts (Neil & Christensen, 2007).
A recent international study by Patalay and colleagues (2016) (Patalay et al., 2016) indicated that schools report they do not provide adequate mental health support for young people. More than a third of UK schools in the study reported that pupils' mental health was not a high priority. Schools cited a number of barriers for this lack of support, including the inadequate capacity of staff, lack of funding and poor links with local mental health services. Therefore, although schools are a key site for mental health promotion and mental illness prevention, it is evident that there are barriers to the provision of mental health support in schools that have yet to be overcome. Therefore, developing alternative means for the delivery of mental health interventions that complement schools' efforts is necessary.
Research into the risk and protective factors for mental illness suggests associations between several school-related factors (e.g. academic pressure and an unsupported learning environment) and poor mental health (Patel et al., 2007). Family factors (including family conflict, parental mental health difficulties) are also associated with mental health difficulties. These findings are in keeping with self-report accounts of adolescents, which indicate family issues and school pressures as the main reported sources of distress. However, protective factors, such a positive school environment can act as a buffer in the face of these risk factors (Jessor, 1998).
Peer support is another key protective factor for the mental wellbeing of children and young people that can buffer the impact of stressful circumstances (Costello, Pickens, & Fenton, 2001).
Many universal school-based approaches incorporate some form of peer support due to the recognized positive impact that this can have on emotional and psychological development. Peer support can enhance healthy coping skills through modelling (Visser, 2004). A meta-analysis found that peer support programmes in schools had a small but significant positive impact on emotional, behavioural and academic outcomes (DuBois, Holloway, Valentine, & Cooper, 2002). Estimated effect sizes across these studies were relatively small compared to effect sizes of other educational, behavioural or mental health interventions. However, most evaluations of peer support programmes have taken place in the USA and as yet there is a limited and generally poor evidence-base for application of peer support in the UK, due to evaluations of such programmes being of poor methodological quality (Hall, 2003).
In recent years, mobile technology has been hailed as an alternative avenue for the delivery of mental health interventions (Whittaker et al., 2012). Recent technological advances have led to wider availability of smartphones. Approximately 73% of adolescents have access to a smartphone, with more than half reporting that they go online several times per day (Lenhart & Teens, Social Media & Technology Overview, 2015), making mobile health applications (MH apps) a potentially accessible source of support and intervention. Furthermore, research suggests that individuals prefer self-help support that is delivered via a familiar platform, such as a smartphone (Martinez & Williams, 2010). At present, there is little research into the use of MH apps in adolescents. This is concerning as thousands of MH apps are currently available for consumers to download (Proudfoot, 2013). Available applications lack a scientific evidence-base and are not subject to regulation or evaluation, questioning their efficacy and safety.
There is also a lack of research into the needs of users of MH apps. Kenny and colleagues (2014) (Kenny, Dooley, & Fitzgerald, 2014) reported cost, safety, engagement, social interaction and confidentiality as key components of MH apps reported by adolescents. These findings should inform the development of future apps, although further research into the key components that would make such apps useful and user-friendly for adolescents is required.

SafeSpot
Key elements of the SafeSpot programme include a universal school-based programme, peer support component and the SafeSpot mobile application. The school-based programme aims to promote resilience and mental wellbeing by equipping pupils with healthy coping strategies and improved awareness of mental health through lessons integrated into the Personal, Social and Health Education (PSHE) curriculum (Mitchell and Punukollu, unpublished). Peer support is incorporated via the "SafeSpotter Scheme' in which older pupils are selected to become the first points of contact in school for pupils experiencing difficulties. Pupils can access support and advice from SafeSpotter's through lunchtime drop-in sessions. The programme also incorporates a mobile application that is available for free download. The app includes personalized coping strategies, audio relaxation, access to a safety plan and crisis numbers and a social media stream with mental health awareness content.

Rationale and aims
It is evident from the literature that preventative approaches to mental health have a wide range of benefits for young people. Researchers and governing bodies repeatedly recommend schools as potential sites through which such programmes can be implemented. However, despite the comprehensive evidence-base for school-based prevention programmes, schools do not routinely adopt similar interventions (Giesen, Searle, & Sawyer, 2007). There is also a need for innovative and accessible ways for young people experiencing ongoing mental health difficulties to receive support. Peer support programmes and mobile health applications have been suggested as possible avenues; however, there is a recognized need for further research in these areas.
The aim of the current study is to build on prior knowledge of school-based interventions and the components that make them effective, by evaluating the effectiveness of the SafeSpot programme on improving the emotional distress of young people. Interviews sought to explore teachers' experiences of the SafeSpot programme, including their perceived strengths and limitations of the programme, as well as recommended adaptations and improvements. The evaluation also aimed to explore the main causes of young people's distress perceived by teachers and reported by young people themselves via a self-report questionnaire.

Research questions
• What do children and young people report as the main causes of distress?
• What do teachers perceive as the main causes of distress in children and young people? • Is there a statistically and clinically significant change in emotional distress between pre-and post-implementation of the SafeSpot programme?
• In teachers' opinions, how helpful is the SafeSpot programme and what are the main strengths and limitations?

Hypotheses
Based on previous research outlining school and family issues as the main risk factors for mental illness, it is anticipated that pupils will rate school and family problems as their main causes of distress. Furthermore, based on evidence of the positive impact of school-based mental health interventions on a range of social and emotional outcomes, it is anticipated that emotional distress, as measured by the HADS, will be statistically and clinically significantly reduced between pre-and post-intervention.

Design
An explanatory sequential mixed methods design was employed for the purposes of the evaluation. This involved secondary quantitative analysis of questionnaire data previously collected by the SafeSpot creators, followed by collection and analysis of qualitative interview data.

Quantitative phase
Participants were secondary school pupils from a high school near to Glasgow, Scotland where the SafeSpot programme was piloted. Three hundred and sixty-seven students took part in the intervention. Participants were aged between 11 and 17 and three classes from each year group (S1-S6) were randomly selected to participate.

Qualitative phase
Participants were guidance teachers involved in the preparation and rollout of the SafeSpot programme in the same secondary school. Only three teachers involved in the programme were available to take part in the research. Due to a small sample being selected within one school no further demographic information has been included to protect participants' anonymity.

Hospital anxiety and depression scale (HADS) (ZIGMOND & SNAITH, 1983)
The HADS (appendix I) is a 14-item validated measure used in clinical practice to assess symptoms of anxiety, depression and emotional distress. Items are related to symptoms experienced in the past week and participants respond on a four point Likert scale. A score of 0-7 is within the normal range, 8-10 is suggestive of an anxious or depressive state (borderline) and a score of 11 or above indicates probable presence of an anxiety or depressive disorder (clinical) (Snaith, 2003). The HADS has shown good psychometric properties, including internal consistency and concurrent validity when compared to several other measures of depression and anxiety (Bjelland, Dahl, Haug, & Neckelmann, 2002). Internal consistency in the current evaluation was good (Cronbach's alpha = .88).

Mitchell-Punukollu causes of distress scale (Mitchell and Punukollu, unpublished)
The SafeSpot creators designed a questionnaire specifically for the purposes of the internal SafeSpot programme evaluation (appendix II). The Causes of Distress (COD) measure is an 18-item scale (scored 0-3), which asks participants to rate how often they are distressed by certain triggers (e.g. bullying, exams). This measure was piloted for the first time in the current study.

Interview topic guide
Interviews with teachers were semi-structured and therefore a topic guide was employed to guide the discussion. Development of the topic guide was steered by the qualitative research questions. Questions were designed to explore teachers' opinions about the SafeSpot intervention, as well as to address some of the wider issues around students' mental health, aiming to expand upon the quantitative findings (see appendix V).

Quantitative phase
The SafeSpot creators and school guidance staff were involved in the implementation process of the programme in the school. Pupils completed several questionnaires, including the HADS and COD, in class at two-time pointsimmediately pre-intervention and 3-months later. Each individual pupil was assigned a unique code consisting of letters and numbers, allowing baseline and follow-up data to be matched. The first author obtained access to the data as part of this external evaluation at a later date. Therefore, quantitative data collection and data input to an Excel spreadsheet was completed prior to the commencement of the current evaluation. The writer was responsible for input of the data to SPSS and data analysis, including missing values analysis.

Qualitative phase
The writer contacted the school directly to outline the proposed research and obtain permission. Approval was obtained from the head teacher and the Local Education Authority. Teachers were invited to take part in the study via letter, with 100% of those invited agreeing to take part. Consent for participation, for the interview to be recorded and for the inclusion of anonymous quotations in the text was obtained from all participants. The study was granted ethical approval from the University of Edinburgh ethics committee (appendix VI). Data were collected via audio-recorded, semi-structured interviews lasting approximately an hour. Interviews were conducted with individual teachers within the school. In order to ensure interview quality, the interviewer listened to recordings after each interview to reflect on the process and consider areas for improvement, taking Whyte's Directiveness Scale (1982) (Whyte, 1982) for analysing interview techniques into account.

Quantitative data analysis
Initial inspection of the data set indicated large quantities of missing data. Notably, there were large amounts of demographic information missing from the data set and so this could not be included in the analysis. Prior to analysis, participants were split into groups according to data available on COD and HADS measures. HADS data were split into four categories: 1) Non-completers (n = 59), those who had participated in the programme but for whom neither baseline or follow-up data were available, 2) Partial completersbaseline only (n = 217), those who only baseline measures were available for, 3) Partial completersfollow-up only (n = 19), those who only follow-up data were available for and 4) Completers (n = 72), those for whom baseline and follow-up data were available.
A significant proportion of follow-up data was missing. Only 20% of participants had completed baseline and follow-up HADS measures. Missing Values Analysis (MVA) was run in SPSS to determine whether there was any pattern to this missing data. Little's chi-square statistic tests the assumption that data is missing completely at random. This test was non-significant x 2 (376) = 405.194, p = .144, therefore the null hypothesis that data was missing completely at random was not rejected and it was concluded that cases with missing values were not systematically different from cases with no missing values. Mean imputation was then employed to maximize the available data for which there were single missing items on the HADS (n = 12).
Additionally, independent t-tests determined whether there was any difference in the HADS scores of those who completed the measures at both time-points and those who dropped out before follow-up. A t-test did not reveal a statistically significant difference between partial responders' HADS scores (M = 12.41, SD = 7.76) and completers' HADS scores (M = 11.16, SD = 6.48), t(167.6) = 1.398, p = .164. This is further evidence that cases with missing values were not systematically different from those with no missing values.
Statistical analysis of quantitative data was conducted using SPSS. Descriptive statistics were generated including mean, standard deviation and measures of normality. Visual inspection of the frequency distribution, stem and leaf plots, P-P plots and Q-Q plots suggested that both the baseline and follow-up data of the HADS were normally distributed. This was formally tested using the Kolmogorov-Smirnov test of normality. This test indicated that baseline and follow-up HADS data were normally distributed, D(72) = .200, D(72) = 0.53, p = > .05.
HADS data were analysed via repeated-measures t-test, to determine if HADS scores were statistically significantly different between baseline and follow-up. However, statistically significant change does not provide information regarding whether the outcome is clinically meaningful (Jacobson, Follette, & Revenstorf, 1984). Clinical significance is the extent to which the intervention induced the change that is of practical importance to individuals. In the current study, a clinically significant change was characterized by the movement of participants out of the clinical range on the HADS (i.e. moving below the clinical cut off of 11). Reliable change, on the other hand, allows researchers to be certain that the degree of change that has taken place is due to the intervention, rather than attributable to chance.
Reliable change in the current study was calculated based on the method outlined in Jacobsen and Truax (1991) (Jacobson & Truax, 1991) (see appendix VII for calculation).

Qualitative analysis
Thematic analysis was used to analyse interview data, following the framework outlined by Braun and Clarke (2006) (Braun & Clarke, 2006). Thematic analysis involves identifying, analysing and reporting patterns or themes within data. It is a theoretically flexible method meaning that it was compatible with the pragmatic approach adopted in the current study.
The first author transcribed audiotapes into Microsoft word documents with the assistance of two SafeSpot volunteers. Data transcribed by the SafeSpot volunteers were checked back against original recordings by the first author to ensure accuracy. This process helped with familiarization of data, outlined by Braun and Clarke as an important first step in the analysis. Patterns were identified in the data inductively (i.e. without trying to fit into a pre-existing coding framework) while holding the research questions in mind.

Research question 1: what do children and young people report as the main causes of distress?
Baseline COD data were available for 258 students (70%). The main causes of distress rated by young people are outlined in Figure 1 below. There was an 18 th item on this scale (emotional wellbeing), however 86% of responses on this item were missing from the data set and therefore this item was not included. The figure below shows the frequency of participants that reported feeling distressed by each item 'occasionally', 'a lot of the time' or 'most of the time'. This data indicates that young people rate the main causes of distress as schoolwork (n = 180, 63%), exams (n = 136, 47%), family problems (n = 135, 46%) and weight (n = 119, 41%). Those items least likely to cause distress were gender (n = 12, 4%), illegal drugs (n = 12, 4%) and smoking (n = 10, 3%).

Family environment
A key theme identified in the data was the perceived impact of the family environment on young people's wellbeing, with all teachers citing changes to family structure and relationships as potential causes of emotional distress. One teacher contrasted her own experience of family life with family dynamics that are more apparent today, citing a lack of warmth and nurture as a key issue. Teachers felt that changing the family structure and a lack of security has led young people to turn to their teachers for support. All teachers reported feeling that they were being increasingly relied upon to provide emotional support to pupils, with all of them attributing this to family factors.

Youth culture
Another theme identified in interviews with teachers was the negative impact of peer pressure to be involved in risky behaviours such as alcohol, smoking, drugs and sexual activity.

Social media
When discussing adolescents' use of social media, several interesting issues were identified. All teachers perceived social media as having a negative impact upon young people. One teacher felt that social media has changed the way that young people communicate. As well as the loss of communication skills, another teacher was particularly concerned with the amount of information that children are exposed to online and the impact that this has on wellbeing. Research question 3: is there a statistically and clinically significant change in emotional distress between pre-and post-intervention?
Participants' demonstrated a clinically significant change if HADS scores showed either a large improvement (moving from clinical range to low range) or a moderate improvement (moving from clinical range to borderline) following the intervention. As can be seen in Table 1, 8% of participants (n = 6) made a large improvement and 10% (n = 7) made a moderate improvement. The majority of participants remained within the same category at both baseline and follow-up: 28% remained in the low range while 33% remained in the clinical range. Only two participants moved from the borderline or low ranges into the clinical range.
As can be seen in Table 2 below, 18% of participants (n = 13) met the criterion for reliable improvement in the HADS, and of these seven (10%) improved in a way that was clinically significant. Three participants (4%) met the criteria for reliable deterioration in the HADS but none of these changes were clinically significant. Two participants met criteria for clinically significant deterioration, but this change was not reliable. Eight per cent of participants were reliably changed but this was not clinically significant. A further 8% of participants improved in a way that was clinically significant but these changes were not reliable. No participants were clinically and reliably deteriorated. It is important to note that the vast majority of participants did not change reliably.

Programme strengths
A primary aim of the study was to explore teachers' opinions about the effectiveness of the SafeSpot programme. Teachers described the delivery of the school-based content as a mainly positive experience, with all teachers expressing personal enjoyment as well as pupils' enjoyment. Teachers also reported feeling like they had benefitted from the programme, through learning a new language and concepts related to mental health:

Opening up discussion
When discussing the benefits of the programme, a pattern emerged in that all teachers felt pupils were better able to open up about emotional and mental health issues in the classroom. Some teachers felt comfortable opening up this discussion, however one teacher expressed some uncertainty about opening up discussion about mental health with pupils.

Pupil engagement
All teachers generally agreed that the majority of pupils bought into the programme and enjoyed it, particularly hearing their teachers openly discuss mental and emotional health.

Change in help-seeking behaviour and implications
Another pattern that was identified in the data was teachers' opinions that the SafeSpot programme had prompted help-seeking behaviour in pupils who were experiencing difficulties. Teachers felt that pupils approached them more frequently to discuss sensitive issues, such as emotional issues or self-harm.
However, although one teacher felt comfortable taking on this role, another teacher reported the difficulty faced by teachers trying to provide support for pupils with mental health difficulties while juggling a number of other responsibilities.
The emotional impact of pupil disclosure on teachers was raised at several points throughout interviews with the overall theme indicating that teachers felt they did not receive adequate training or supervision to deal with some of the emotional content brought by pupils. Teachers also felt that the focus on school delivery of mental health content was a big demand, and in the current circumstances, not entirely feasible. One teacher felt that too much pressure was being placed upon schools by governing bodies to deliver mental health support.

Limitations and adaptations
When discussing the perceived disadvantages of the programme, several interesting issues were highlighted regarding the content of the school-based programme. The main issue highlighted was that all teachers felt that the programme content was pitched at 'too high a level' for students. As a result, teachers spent time adapting the content, which most felt was an additional time pressure on their already demanding workload.
All teachers agreed that in order for the programme to be successful, the content should be delivered to pupils in a basic and accessible way. Teachers generally reported a lack of understanding of those outside of the school environment of what is likely to be appropriate for pupils. All teachers also reported their belief that content was overly focused on text-based activities. As the programme was integrated into the PSHE timetable, which is generally discussion-based, all teachers felt the content could be improved by making it more interactive for pupils. Teachers felt they had to be creative about the way that they delivered the content.

Mobile application
When discussing the mobile application, teachers' opinions varied. Some reported that they felt the app was a great addition to the programme and was well-received by pupils as it allowed them to access help through a medium they are very confident using, finding it very appealing to young people.
Other teachers felt that the app was less successful, citing a lack of sustained interest and disengagement as the main issue. Teachers felt that the app needed to do more to sustain pupils' attention. They suggested possible ways that this could be improved, including the involvement of young people in the development of the app and incorporating games.

SafeSpotter scheme
One teacher felt that inclusion of the SafeSpotter scheme was a positive addition to the programme. It was felt that recruiting older pupils as mentors had prompted them to take ownership over younger pupils' difficulties. The SafeSpotter's approached this teacher on several occasions to report when younger pupils were experiencing difficulties.

Discussion
A key aim of the present study was to evaluate the perceived causes of young people's distress reported in interviews with teachers and by young people on the Causes of Distress questionnaire. Consistent with the hypothesis, the main causes of distress rated by young people in the current study were related to school (specifically school work and exams) and family problems. These echoes findings from previous research outlining school factors (e.g. academic pressures) and family factors (e.g. family conflict) as significant causes of distress for young people (Byrne, Davenport, & Mazanov, 2007;Rees, Bradshaw, Goswami, & Keung, 2010). These findings also tie in with a large body of research outlining family discord and academic pressure as risk factors for mental illness in young people (Patel et al., 2007).
The qualitative findings suggest that teachers are in agreement that family problems are a key challenge for young people. Interestingly, the qualitative results revealed teachers' perceptions of the main causes of young people's distress otherwise differed from that of pupils, with teachers placing emphasis on the effects of social media and youth culture (e.g. pressure to be involved with alcohol, smoking or illegal drugs). Conversely, illegal drugs and smoking were the least rated causes of distress by young people. Almost two-thirds of pupils rated schoolwork as a significant cause of distress, however, teachers did not acknowledge this in interviews. This finding is not surprising given that agreement between young people, parents and teachers regarding young people's mental health is generally low in systematic studies (Collinshaw, Goodman, Ford, Rabe-Hesketh, & Pickles, 2009). It has been suggested that this lack of concordance can be explained by opposing frames of reference (i.e. young people and teachers' differing perceptions shaped by their prior beliefs, values and experiences) and differences in young people's functioning across settings (Fazel et al., 2014).
The primary aim of the present study was to evaluate the effectiveness of the SafeSpot programme in reducing the emotional distress of secondary school students. Consistent with the hypothesis, results indicated a statistically significant reduction in emotional distress between preand post-intervention. This finding is in keeping with the results of previous reviews, which have found that universal school-based interventions are effective in reducing symptoms of anxiety and depression (Adi et al., 2007;Weare & Nind, 2011).
In addition to statistical significance, a proportion of participants demonstrated clinically significant change (Jacobson et al., 1984), suggesting that improvement in these participants is clinically meaningful. Eighteen per cent of children in the clinical range at baseline displayed a large or moderate improvement in self-report emotional distress at follow-up, moving into the low or borderline ranges, respectively. Of these participants, 10% had improved reliably, indicating that improvements in these participants were due to the intervention, rather than attributable to chance (Jacobson & Truax, 1991). No participants displayed reliable and clinically significant deterioration between baseline and followup, indicating that the programme did not have any harmful effects on participants.
There are a number of possible reasons as to why the majority of children did not change clinically and reliably. It is not unusual for some participants to show no change following intervention (Lee, Tiley, & White, 2009). Research into the effectiveness of therapeutic interventions for children and young people report that 'up to a third of cases do not respond to the best available psychological treatments' (Carr, 2000). Furthermore, initial symptom levels tend to be lower in universal interventions, meaning that subsequent reductions may be more modest (Stallard, 2010). Indeed, in the current study, almost a third of the participants were in the 'low' range at baseline and follow-up, meaning there was less scope for improvement in these participants. Reviews tend to suggest larger effect sizes in targeted and indicated interventions, when initial symptom levels are higher.
In terms of the qualitative findings, overall feedback from teachers was largely positive. Teachers felt that both they and their pupils had benefited from, and enjoyed, the programme. The main reported benefit of the programme was that it had opened up a discussion about sensitive issues (e.g. self-harm) and had prompted help-seeking behaviour in pupils. It is possible that teachers' and SafeSpotters' modelling of open discussion and help-seeking as positive coping strategies facilitated changes in pupils' help-seeking behaviour (Barker, 2007).
However, a salient theme in interview data was the perceived barriers to implementation of the programme. Teachers felt pressurized to juggle several roles, including lesson delivery, supporting academic success and supporting pupil mental health. Therefore, although it was a benefit that the programme prompted help-seeking, teachers felt they did not have adequate time, supervision or training to adequately support pupils' with mental health difficulties. Similar concerns have been raised in previous research in which teachers have cited time constraints and lack of expertise as affecting their ability to help pupils with mental health difficulties (Cooper, Hough, & Loynd, 2005). This highlights the challenge of accommodating mental health programmes within busy school timetables. It may be beneficial for creators of school-based programmes to consider offering a medium through which teachers can access support and supervision.
Teachers also outlined concerns with the programme content. They felt the programme lacked in age-appropriate content and included too much emphasis on text-based activities rather than interactive activities (e.g. class discussion, role-play). A previous qualitative study of a school-based anxiety intervention by Skryabina and colleagues (2016) (Skyrabrina et al., 2016) also outlined teachers' and pupils' preference for practical activities. These findings highlight important considerations for those involved in the design of school-based interventions.
Teachers did not go in to detail about their opinions of the SafeSpotter peer support component in interviews. However, the general perception was that this component was a positive addition to the programme as it had encouraged older pupils to take ownership over the wellbeing of younger pupils. Although these are promising findings, further research into the benefits of incorporating peer support in mental health programmes is required.
Teacher's opinions on the SafeSpot mobile application varied. Some felt that it was a great addition to the programme as young people were confident using the app. Others felt it did not do enough to sustain young people's interest and so they lacked the motivation to use it. A key difficulty of a preventative approach is that individuals not currently experiencing difficulties are less motivated to change their behaviour and use a self-help medium (Stein, Bauman, & Ireys, 1991), which may have influenced motivation to use the app. Teachers recommended that involving pupils in the design process could make the app more appealing, a recommendation that has also been made by previous evaluators of other MH applications (Kenny et al., 2014).
These preliminary findings are interesting, however, further research into the efficacy of the SafeSpot app as a platform for mental health support is recommended.

Strengths and limitations
A key strength of the current study was in its methodology, as the adoption of multiple methods and triangulation of results can increase the validity of evaluation research (Stewart-Brown, 2002). In the present study the qualitative component added depth to the quantitative findings. Teachers' experiences of the programme gave valuable insight into how the programme was effective, as well as highlighting limitations and areas for improvement. Furthermore, the programme was conducted and evaluated in a real-world setting, ensuring high ecological validity, and therefore the findings are likely to be comparable in other school settings.
However, the study has a number of limitations, particularly due to large quantities of missing data. Although data were not missing systematically, removal of participants with missing values results in loss of statistical power (Cool, 2000). The large proportion of missing follow-up data meant that pre-and post-data was only available for 20% of participants in the intervention, which limits the strength of the conclusions that can be drawn. Moreover, large amounts of missing demographic data meant that further analysis according to age and gender was not possible, which could have further informed the evaluation. Furthermore, despite the sound psychometric properties of the HADS as a measure of emotional distress, the present study was limited in that no validated measure of mental wellbeing or coping skills was included, given this is what the SafeSpot programme aimed to improve. The use of a non-validated Causes of Distress measure also has limitations, as it is unclear whether the questionnaire captures what it intends to measure. Inclusion of additional validated coping and wellbeing measures should be considered for future implementation and evaluation.
Another limitation of the current study is that the programme was only trialled in one school, whereas other evaluations tend to include multiple schools. Inclusion of multiple schools across different contexts is likely to enhance the generalization of results and is recommended for future projects.

Conclusions
In conclusion, the present findings indicate that a universal school-based intervention targeting mental health awareness and coping skills can have a positive effect on the emotional distress of secondary school pupils. The present study provided further insight into the barriers to the implementation of school-based interventions perceived by teachers, particularly in terms of workload and competency in supporting mental health issues. These findings build upon previous research on school-based interventions and associated benefits and challenges (Fazel et al., 2014). Although the preliminary findings are promising, there is a need for follow-up research to further explore the efficacy of the various components of the SafeSpot programme. Mixed-methods research was found to be a useful tool for the evaluation of a school-based intervention, and should be considered by researchers undertaking such projects.

Ethical approval
The ethics committee of The University of Edinburgh approved this study [Appendix VI].

Contributorship
MP, CB and MM researched literature and conceived the study. MP gained ethical approval and completed the interviews. CB wrote the first draft of the manuscript. All the authors reviewed and edited the manuscript and approved the final version of the manuscript.

Guarantor
MP is the guarantor for this article.