Social-Emotional Learning in the Middle Grades: A Mixed-Methods Evaluation of theStrong KidsProgram

Abstract Adolescence can be a particularly difficult time for youth as they experience the physical, social, and academic changes that accompany maturation. One of the most critical challenges for schools is teaching adolescent students positive social and emotional skills, which are crucial for them to succeed academically and emotionally. Strong Kids is a social-emotional learning curriculum designed to reduce students’ internalizing symptoms. Strong Kids has shown promise in elementary schools; this was the first study to evaluate the newly updated version of the program in a middle level school setting. Two general education teachers implemented the curriculum with eight students at risk for emotional and behavioral disorders. Researchers used a mixed methods design to evaluate outcomes. Findings suggest that Strong Kids was effective at improving students’ social-emotional knowledge and internalizing symptoms; however, there were no significant changes in their externalizing symptoms.

Adolescence can be a particularly difficult time for youth as they experience the physical, social, and academic changes that accompany maturation (McGorry, Purcell, Goldstone, & Amminger, 2011). Specifically, the transition from an elementary school to a middle level school often marks a decline in students' motivation and their behavior may begin to decline as well (Eccles, Vida, & Barber, 2004). The behavioral decline may include acting aggressively, pestering teachers, or irritating peers. Students may also refuse to follow directions or may engage in other non-compliant behaviors (Hecker, Young, & Caldarella, 2014). With so many physical, social, and emotional changes occurring, adolescence is a time when mental health concerns and emotional disorders often begin to emerge (McGorry et al., 2011).
The most common disorders that affect adolescents are anxiety disorders, behavioral problems, mood disorders, and substance abuse (Merikangas, Nakamura, & Kessler, 2009). An estimated onefourth of youth between the ages of eight and 15 have a mental health disorder. Unfortunately, only half of them receive the treatment they need (Merikangas et al., 2009). Emotional and behavioral disorder (EBD) is the special education classification used for many students who have mental health issues. One primary identifying criterion for EBD is academic underachievement that cannot be explained by intellectual, sensory, or health factors (Reid, Gonzalez, Nordness, Trout, & Epstein, 2004).
EBD includes two categories of behavior: externalizing and internalizing (Kauffman, Simpson, & Mock, 2009). Students with externalizing disorders often exhibit behaviors that seem uncontrolled, putting them at a higher risk for aggressive and violent behaviors such as physical fighting or carrying a weapon to school (O'Connell, Boat, & Warner, 2009;Wolff & Ollendick, 2006). Externalizing behaviors can be highly demanding for teachers, as they interfere with everyday classroom activities (Weist et al., 2018). Thus, externalizing behaviors are likely to be noticed by teachers and other adults in the student's life. Conversely, many students exhibiting internalizing behaviors may not be identified; they may be overcontrolled (Wolff & Ollendick, 2006) as they inwardly or privately experience feelings of distress (Cosgrove et al., 2011). To cope with the feelings of pain, internalizing students may deal with their emotions in unhealthy ways, such as being quiet, shy, or self-isolating; having perfectionistic tendencies or negative perceptions of themselves; and suffering from somatic distress (e.g., body, stomach, and or headaches) without a medical explanation (Masten et al., 2005;Weist et al., 2018).
An estimated two-thirds of youth who have internalizing problems also meet the criteria for externalizing problems (Hastings, Zahn-Waxler, & Usher, 2007). Students who exhibit both externalizing and internalizing symptoms have led researchers to believe EBD's two categories are correlated in ways not fully understood (Masten et al., 2005). Students with internalizing problems have a lower positive affect but higher anxiety and cardiovascular arousal (Hastings et al., 2007). Those with externalizing problems have greater hostility and positive affect but less cardiovascular arousal. One reason for the high level of comorbidity is that some students use internalizing behaviors to cope with externalizing behaviors (Masten et al., 2005). For example, a student exhibiting externalizing symptoms may struggle academically and cope with the academic distress by exhibiting internalizing symptoms such as anxiety or depression. Consistent, safe, and positive school environments are critical for helping students learn tools to deal with externalizing and internalizing symptoms (Weist et al., 2018).

Social-Emotional Learning
One of the most critical challenges in schools is teaching students positive social, emotional, and behavioral skills that are crucial for them to succeed academically and emotionally (Carrizales-Engelmann, Feuerborn, Gueldner, & Tran, 2016). Interventions based on instruction regarding how to behave that include support and reinforcement for desired behaviors seems to be the best solution for helping students improve (Kauffman et al., 2009). Social-emotional learning (SEL) is an essential aspect of improving student outcomes. The Collaborative for Academic, Social, and Emotional Learning (CASEL, 2019) provided this definition for SEL: The process through which children and adults acquire and effectively apply the knowledge, attitudes, and skills necessary to understand and manage emotions, set and achieve positive goals, feel and show empathy for others, establish and maintain positive relationships, and make decisions. (What is SEL? ¶ 1) SEL is endorsed by schools across the country as highly effective in teaching students tools and strategies to deal with externalizing and internalizing symptoms by helping to strengthen their ability to plan and exert self-control (Greenberg et al., 2003;Van Loon, Van De Ven, Van Doesum, Hosman, & Witteman, 2015). SEL is also an identified area of focus in the American Educational Research Association (AERA) Middle Level Education Research Special Interest Group research agenda (Mertens & Caskey, 2018). Middle level schools can address SEL in a variety of ways including the use of free-standing lessons or programs, integration of SEL into academic course work, and the promotion of SEL as a primary tier (schoolwide) initiative (CASEL, 2019).

Strong Kids
Strong Kids is an evidence-based SEL program designed to prevent internalizing symptomatology by promoting social and emotional wellness and coping (Carrizales-Engelmann et al., 2016). The Strong Kids curriculum is useful for all three of the multi-tiered prevention levels but is ideally taught at a primary or secondary tier (Carrizales-Engelmann et al., 2016). The focus of Strong Kids is teaching self and social awareness, responsible decision making, self-management, and relationship management. The program creators aimed to provide an affordable, efficient SEL program; its manual was designed to be taken off the shelf and taught to students with little or no training required for the teacher. The program teaches students skills in five categories: (a) learning to create strong attachments early in life, (b) gaining ageappropriate skills, (c) having experiences that promote healthy well-being, (d) feeling they control their fate, and (e) learning to deal with stress in healthy ways (Merrell, Carrizales, Feuerborn, Gueldner, & Tran, 2007). The Strong Kids program consists of 12 lessons, each designed to last approximately 50 minutes (Carrizales-Engelmann et al., 2016).
Strong Kids is not a comprehensive program for preventing school violence or antisocial behavior, though it may assist in addressing such problems (Carrizales-Engelmann et al., 2016). It is an example of a standalone program that is not integrated into academics directly and has a limited focus on primarily internalizing symptoms. It is not designed to be a complete mental health treatment package for youth with severe mental health problems.
Previous research on Strong Kids found that 81% of students showed increased social-emotional knowledge after participating in the program (Gueldner et al., 2019). Studies also found that students who participated in the program selfreported lower levels of internalizing and other problem symptoms on their posttest compared to what they self-reported on their pretest (Merrell et al., 2007). In fact, 17 previous Strong Kids studies reported that 94% of students felt their internalizing symptoms had lessened after participating in the program (Gueldner et al., 2019).
Strong Kids research found that teachers considered teaching students coping skills to deal with life challenges to be important (Gueldner et al., 2019). Their attitudes about the Strong Kids program were generally positive but specifically mixed. Teachers observed increased student knowledge of socialemotional skills; however, not all reported significant changes in students' internalizing symptoms (Caldarella, Millet, Heath, Warren, & Williams, 2019). Research has noted that educators find it challenging to adequately and fully comprehend students' internalizing symptoms (Loeber, Green, & Lahey, 1990).

Study Purpose
Relatively few studies have been conducted using the Strong Kids program in middle level schools and none have examined the program's effects on students' externalizing symptoms. In 2016 a new version of Strong Kids was released, updating the curriculum and integrating mindfulness activities, but no studies have been conducted using the new version in middle school settings. The current study evaluated the new Strong Kids program with a middle grades population (Carrizales-Engelmann et al., 2016).
The purpose of the current study was to evaluate the extent of impact of the use of Strong Kids at a secondary level in a middle level school on students' externalizing and internalizing symptoms. The teachers' ability to teach the lessons with fidelity was monitored as well. Interviews and focus groups were conducted to evaluate social validity of the curriculum at the mid-point and conclusion of the study. The curriculum was implemented as a Tier 2 small group intervention. The following questions were addressed:

Setting
The Strong Kids program was implemented at a suburban middle school in a western state of the United States. The total population of the school was 1,054, consisting of White (68%), Hispanic/Latino (24%), Hawaiian Native/Pacific Islander (3%), Asian Pacific Islander (2%), multi-race (2%), African American/Black (1%) students. Of these students 6% were eligible for the reduced-price lunch program and 36% were eligible for free lunch.

Participants
Student Participants. This study began with 10 students (nine males, one female) in 7th-8th grade. Participants' ethnicity included Hispanic/Latino (50%), White (40%), and Asian Pacific-Hawaiian Native/Pacific Islander (10%). None of the participants was identified with special education needs or had an individualized educational program. Due to attrition (excessive absences) evaluation data were available on only eight students. Participants were enrolled in a class the school identified as the Academic Success Class. The goal of the class was to help at-risk students socially, emotionally, and academically. Students were enrolled in this class due to behavior problems and academic grades significantly below grade level in most or all core content areas. Problems with substance abuse and negative living situations were additional challenges that could lead to placement in the class. The students had their first and last period classes together every day and were integrated into general education classes during other periods.
The participating female teacher provided the following history about the students in the study. All students in the Academic Success Class came from challenging home situations with limited supervision outside of school. Most of the students reported a history of conflict with their parents. One student's mother did not have parental rights due to substance abuse. This student's father worked as a truck driver, traveling often to another state. Although a stepmother was involved, this student often stayed in the home alone during the week while the father was away working.
The students who had parents at home reported that their parents worked multiple jobs and were rarely home. One student's mother lived in the United States, but the student's father lived in another country. The father wanted to reestablish a relationship with the student, and the mother was supportive of this; however, the student did not want a relationship with the father because of past negative experiences. The tension with the parents caused the student major distress and anxiety, which possibly contributed to the student's anger and behavior problems.
Most of the students struggled with anxiety and depression. Learning how to interact with adults respectfully was difficult for many of these students, as they lacked responsible adults in their home lives. Students had difficulty discerning tones of voice, facial expressions, or appropriate conversation to use with adults. They would treat adults the same way they treated their peers.
Teacher Participants. The Academic Success Class was taught by two teachers who implemented the Strong Kids program in their classrooms. At the time of this study, both teachers were in their first year of teaching this class. A 29-year-old male teacher with four years of teaching experience taught the program during the students' first period of the day. He had a bachelor's degree in art education and was pursuing a master's degree in the same field. He taught two lessons. A 31-year-old female teacher with eight years of teaching experience implemented the intervention during the final class period of the day; she had a bachelor's degree in history education. She taught ten lessons. A 38-year-old female instructional aide, with a bachelor's degree in English and 13 years of experience, was present during all the lessons. She did not teach but would occasionally contribute comments.

Field Notes
A researcher attended and took field notes for all 12 of the Strong Kids lessons taught for this evaluation. These notes included the number of students attending each lesson, the start and finish times, comments made by the students, and components of the lessons taught. After each lesson the researcher and teacher discussed the lesson, including what went well and what could be done differently in future lessons.

Dependent Measures
Social Skills Improvement System (SSIS). The SSIS is a multi-rater measure allowing the teacher and students to rate the students on the frequency of various behaviors (Gresham & Elliott, 2008). The SSIS includes three domains of student functioning: Social Skills, Problem Behaviors, and Academic Competency. The Externalizing and Internalizing subscales within the domain of problem behaviors were used in this study.
The teacher version of the SSIS measures various externalizing and internalizing symptoms the teacher perceives in the student. The Externalizing subscale is composed of 12 items, while the Internalizing subscale is composed of 7 items. "Is aggressive toward people or objects" and "Withdraws from others" are sample items. The student version of the SSIS measures externalizing and internalizing symptoms students perceive in themselves. The Externalizing subscale consists of 12 items and the Internalizing subscale consists of 10. "I often do things without thinking" and "I think no one cares about me" and are among the items. All SSIS items are based on a four-point Likert scale rated as never, seldom, often, or always occurring.
As reported in the test manual (Gresham & Elliott, 2008 Researchers also completed interviews with the students individually midway through and at the completion of the study. The students were asked the following questions:  , 2016). As an SEL curriculum designed to decrease students' internalizing symptoms, the program consists of 12 lessons that cover topics such as identifying emotions, feeling empathy, managing anger, thinking clearly, managing stress, solving problems, and setting goals. The lessons include direct instruction from a teacher, role-play scenarios, group discussions, and worksheets for skill practice.
Intervention Fidelity. Fidelity checks were conducted on the integrity of implementation of the Strong Kids program. One researcher observed 100% of the lessons, and a second was present during 66% of the lessons to provide data for inter-observer agreement (IOA). These researchers completed a fidelity checklist included in the Strong Kids manual, recording whether the main objectives and activities for each lesson were completed fully, partially, or not at all. Interobserver agreement (IOA) was also calculated for treatment fidelity observations for occurrence and quality, by dividing the number of agreeing intervals by the total number of intervals. IOA averaged to 98.5%, with a range of 87.5-100%.

Procedures
The school purchased and implemented the Strong Kids curriculum. Teachers were not given any training. Initially researchers met with the teachers every other week for 30 minutes to plan for program evaluation. After the fourth Strong Kids lesson, researchers held a sixty-minute focus group during their meeting with the teachers to get feedback on how they felt the program was going. As the teachers became more comfortable with the program, these meetings quickly changed from bi-weekly to monthly.
A pretest and posttest which included the SSIS Externalizing and Internalizing subscales and the Strong Kids Knowledge test were administered to the students through a Qualtrics survey. Students completed the pretest one week before the intervention began and the posttest the week following the last Strong Kids lesson. The two teachers and the instructional assistant completed the SSIS Externalizing and Internalizing subscales on the same before-after timing.
Lessons were taught each Wednesday. The school chose to have the two teachers take turns teaching the Strong Kids lessons (one in the morning and one in the afternoon) alternating each week so that both teachers would be able to learn about and support the program with the class. Lessons lasted 45-55 minutes except for the ninth lesson, which was long and contained important terms the students did not know.
To be sure the students understood the terms, the teacher split the lesson onto the following day so she would be able to discuss ways the terms related to the students' lives. As an incentive for participation, researchers provided students with snacks each week during the Strong Kids lessons There was a three-week gap between the third and fourth lessons due to Christmas break; a review lesson restarted the program the Wednesday after students returned. The lead researcher conducted individual interviews with each student, lasting approximately 5-minutes, and one 60-minute focus group with the teachers following the review lesson.
Considering the results of the student interviews and teachers' focus group, the intervention delivery was modified. The female teacher taught all the remaining lessons in her class, and she modified some portions of the lessons by altering the stories or examples to be more relatable and ageappropriate for the students. The male teacher worked with the boys in groups of three throughout the week during his class to help them complete the homework from Strong Kids lessons, enabling him to review the information with the students as he did so.

Design and Analysis
Researchers used a mixed methods design, specifically a concurrent triangulation strategy that is often used to create social change or advocacy (Creswell, Plano Clark, Gutman, & Hanson, 2003). This strategy allowed researchers to more accurately examine relationships among the variables, with the goal of contributing to generalizable knowledge. To enable concurrent triangulation, both qualitative and quantitative data were gathered during the evaluation. Researchers obtained institutional review board approval to use a deidentified existing data set provided by the school to complete the evaluation.
Due to the small sample size, results from the SSIS Externalizing and Internalizing subscales and the Strong Kids Knowledge test were each analyzed quantitatively using Cohen's d to examine the effect size changes from pretest through posttest. Social validity and treatment fidelity were analyzed quantitatively using descriptive statistics and qualitatively to examine alignment with the Strong Kids goals, procedures, and outcomes.
The open-ended questions from the student interviews and the teacher focus groups were analyzed qualitatively. The first and second authors examined participants' responses and used interpretational analysis to code the data for common patterns or themes. Data collection and analysis were conducted sequentially. The first and second authors conducted the teacher interviews; student interviews were individually conducted by the first author. After the first interviews, data were transcribed, and a preliminary analysis was conducted to identify common themes organized according to participants' perceptions of Strong Kids. Thus, researchers could evaluate what had been learned and select topics to pursue in the second interviews. Following the second interviews, researchers compared the data from both interviews. Along with the interviews, researchers had communication with the teachers through e-mail and in-person conversations to further address questions surfacing from the data. This recursive process continued until saturation was reached and no new themes were uncovered.

Results
This study evaluated the effects of the Strong Kids program on the externalizing and internalizing symptoms of a classroom of middle grades students who were at-risk for EBD. In this section, a review of the research field notes will be reported, followed by answers to the six research questions.

Field Notes Summary
The students began calling the Strong Kids lessons their "AA Group" because the teachers put the students' desks in a circle facing one another for the lessons. Students ate snacks and talked about feelings during the lessons. One student said, "These lessons are way better than our regular classes." Students were often distracted and talkative during the lessons. During most of the lessons, students would listen and pay attention for approximately 10 minutes. From the beginning of the intervention, most of the students were open and willing to share their thoughts and experiences. At times the lesson would move away from the lesson plan, and the teacher would lead a short discussion when topics such as sluffing, drugs, and suicide emerged. Although these discussions were not part of the curriculum, they were opportunities for the teachers to have meaningful conversations with the students that they might not have been able to have otherwise.
In one of the first lessons the teacher asked the students if they had someone to talk to about their feelings and challenges. Some said they could talk to their parents; others said they did not have anyone to talk with. During the final lesson the teacher made sure that each student had a list of two or three people, written on a piece of paper, with whom to discuss feelings and challenges that would need to be shared.
When students were asked to write things down, they generally seemed engaged in doing so. However, a few of them had a hard time writing things down because they wanted to share their thinking aloud. During the individual interviews at the end of the intervention, one student said he would have felt more comfortable filling out the worksheets and writing thoughts down if he knew he would not have to give his paper to the teacher or let anyone else see it.
Throughout the program, the teacher would often share personal experiences related to the lesson with the students. As the program progressed, the students began sharing their personal experiences as well.
A student who was receiving one-on-one counseling outside of school would often pull out a notebook he kept during his counseling sessions and share ideas from his notebook. The Strong Kids lessons seemed to bring the students together and strengthen their relationships with one another and with the teacher. A student who had been kicked by one of his classmates walking in the hall told one of the teachers that he wanted to punch the kicker, but stopped, thought about it, and decided to kick a locker instead. Another student commented that he hoped he could stay in the class because of the relationships he had created.

Treatment Fidelity
The first research question asked whether teachers would be able to implement Strong Kids with fidelity.
In considering all the objectives as outlined in the manual, teachers either fully (81%) or partially (12%) implemented lesson components 93% of the time, indicating good treatment fidelity. On average, teachers did not implement 7% of the objectives indicated in the curriculum manual. The items most often omitted included review of the previous lesson, introduction to the new lesson, and conclusion of the lesson.

Social-Emotional Knowledge
The second research question examined whether Strong Kids implementation led to increased socialemotional knowledge. Table 1 contains descriptive data on students' scores on all pretest and posttest measures. The students had a mean score of 12.50 on the Strong Kids Knowledge pretest. At posttest, the students had a mean score of 13.63, yielding an overall d of .40, a medium effect size increase.

Externalizing Symptoms
The third research question examined whether implementing Strong Kids led to decreased externalizing symptoms. The students had a mean score of 19.80 on externalizing symptoms on the SSIS pre-self-rating, and at posttest their mean score was 19.25: scores were in the above average range at both pretest and posttest. The Cohen's d showed an effect size of .10, indicating very little change in students' self-rated externalizing scores. Table 2 contains the descriptive statistics for the two teachers' and the instructional aide's pretests and posttests describing student behavior. All three scored the students' externalizing symptoms on the SSIS in the above average range for the pretest and posttest. Results were mixed, as Teacher 1 indicated no change in externalizing symptoms over time, Teacher 2 indicated an increase in externalizing symptoms, and the instructional aide indicated a decrease in externalizing symptoms. Teachers commented at the beginning of the evaluation that the instructional aide spent the most time with the students and that her ratings might be the most accurate.

Internalizing Symptoms
The fourth research question asked whether Strong Kids implementation led to decreased internalizing symptoms.
At pretest, this sample of students had a mean score of 13.80 on internalizing symptoms on the SSIS self-rating, indicating that students as a group were initially slightly above average for levels of internalizing symptoms. Their mean score at posttest was 10.20, in the average range. The Cohen's d for students' self-rating on the SSIS Internalizing was .50, which indicates a medium effect size decrease between pretest and posttest. The ratings of Teacher 2 and the instructional aide were above average, with small to medium effect size decreases in students' internalizing symptoms. Teacher 1 rated the students' internalizing symptoms in the average range at both times, indicating a small increase.

Social Validity
Teachers. The fifth research question asked whether teachers perceived Strong Kids as socially valid. One Note. SSIS-E = Social Skills Improvement System-Externalizing; SSIS-I = Social Skills Improvement System-Internalizing Note. SSIS E = Social Skills Improvement System Externalizing; SSIS I = Social Skills Improvement System Internalizing; SKKT = Strong Kids Knowledge test teacher said, "I think the topics were incredibly beneficial." More than once the students said things like "I feel like this was written exactly for us." One teacher said each of the students responded especially well to something different. For example, a couple of students responded well to the lesson on stress, some seemed to need to discuss anxiety, and others related to the explanation of mind traps and dark glasses.
Both teachers felt that encouraging the students to think and talk about various experiences in their lives was the most beneficial part of the program. The teachers appreciated the resources the curriculum gave them that enabled conversations about topics they would not typically have initiated. These discussions led to the students talking about difficult experiences that helped the teachers understand the students better and have more empathy for them.
Teachers said that because of the lessons they were able to start more conversations with the students because they felt natural in talking through those experiences even outside of the lessons. Teachers said they would implement the program again, and they would be interested in implementing it in a general education classroom. Teachers also reported that teaching the program helped them reflect on their own life, making them aware of how the lessons applied to them as well as the students.
Regarding things that could be improved, teachers reported that they called the program Strong Teens because the students made fun of the title Strong Kids. Teachers also noted there is a drastic difference between sixth graders and eighth graders. In the Academic Success Class two of the students were seventh graders and the rest were eighth graders, which may have contributed to the students considering the material too juvenile for them.
Students. To answer the sixth question, the researchers examined whether students perceived Strong Kids as socially valid. When asked during the student interviews if they would participate in the program again, 83% said they would. When students were asked how the program helped them, the following responses were typical: Students said they thought the sessions could be improved by having more physical activities built into the lessons rather than sitting the entire class period.

Discussion
The purpose of this study was to examine the effects of the Strong Kids curriculum on middle grades students at risk for EBD. The study explored changes to the levels of students' externalizing and internalizing symptoms as well as their knowledge of social-emotional learning. Researchers also analyzed the treatment fidelity and social validity of the program. This was the first study to examine the second edition of the Strong Kids curriculum in a middle level school as well as the first to examine whether middle grades students' externalizing symptoms were impacted by participating in the program. The implications of this research are addressed according to the research questions. Specific limitations are discussed in the corresponding sections. Directions for future research and implications for this study are also included.
First, results showed that 89% of the Strong Kids curriculum was implemented with fidelity. These results can be compared to Feuerborn's (2004) study with a treatment fidelity of 70% and Skiba's (2017) study with a treatment fidelity of 93%, and further demonstrate that middle level school teachers are able to implement Strong Kids with fidelity without any prior training. Teachers in the current study were able to cover most of the curriculum, but often did not have time to review the previous lesson or spend time summarizing the current lesson, areas for future studies to consider. The fact that teachers were able to implement Strong Kids with fidelity and without prior training is important, as schools are often looking for programs that are feasible and easy for teachers to implement in the classroom (Carrizales-Engelmann et al., 2016).
Second, this study showed that students' socialemotional knowledge increased after participation in the Strong Kids program, consistent with previous research by Skiba (2017) and Gueldner (2007), though effect sizes were somewhat smaller in the current study. Strong Kids was intended as preventative; thus, time may be required to see the full effects of participation on students' knowledge.
Interventions designed to improve students' socialemotional knowledge regarding how to behave, and then providing support for desired behaviors, are crucial for students to succeed academically and emotionally (Carrizales-Engelmann et al., 2016;Kauffman et al., 2009). These types of interventions should be implemented in schools to help slow the growth of mental health problems such as EBD (Weist et al., 2018).
Third, the teachers and students rated the students' externalizing problems as above average both before and after the program. Although the curriculum was designed to lessen students' internalizing rather than externalizing symptoms, we wanted to evaluate whether the curriculum could lessen externalizing symptoms, considering research that has suggested the comorbidity between the two symptom categories (Masten et al., 2005). Results of the current evaluation revealed no significant effects on students' externalizing symptoms, which corresponds to the fact that Strong Kids was not designed to specifically address such symptoms.
The fourth research question examined whether the implementation of the Strong Kids curriculum decreased students' internalizing symptoms. Pretest scores for students in this study placed them in the above average range, and their posttest scores decreased to the average range. These results showed a medium effect size decrease in the students' internalizing symptoms, consistent with previous studies (Caldarella et al., 2019;Gueldner et al., 2019). One teacher and the instructional aide rated the students' internalizing symptoms as slightly above average at pretest with decreases at posttest. The other teacher rated the students' internalizing symptoms as below average at both pretest and posttest. These inconsistencies in the teachers' ratings are also consistent with previous research, which has found that some teachers have difficulty measuring students' internalizing symptoms (Caldarella et al., 2019;Center for Disease Control, 2013), as these symptoms are of a more private nature than externalizing symptoms. The fact that internalizing symptoms improved following Strong Kids implementation is important, given that such symptoms can result in increased social avoidance, somatic distress, loneliness, and negative selfperceptions (Masten et al., 2005;Weist et al., 2018). Internalizing disorders which go untreated can have long-term adverse effects reaching into adulthood, including mental illness, complicated relationships, unemployment, and suicidality (Bayer et al., 2011).
Concerning the social validity of the program, the teachers' and students' responses were predominantly positive, similar to the results of past studies (Caldarella et al., 2019;Gueldner et al., 2019). Teachers noted improvements in their students' behaviors and in their relationships with their students following implementation of the curriculum. Teachers agreed that the topics in the curriculum provided opportunities for discussions with the students they might not have had otherwise. Teachers did recommend some improvements for the program, as they felt some of the material was too immature for eighth grade students. They also recommended that the lesson on stress be split into two lessons to allow adequate time to teach the lesson topic. Overall, the teachers were pleased with Strong Kids and would be willing to teach the curriculum again. Most students indicated that they would participate in the program again. They identified a variety of benefits from participation in Strong Kids, including better understanding of their own and others' emotions, as well as improved relationships, anger management, and use of positive coping strategies. Students recommended more opportunities for physical activities and suggested adding material more appropriate for them to relate to at their age. The positive social validity reports are promising, given that a program is unlikely to be implemented if stakeholders are not satisfied with the program and do not find it acceptable (Luiselli & Reed, 2011).

Limitations and Directions for Future Research
Several limitations to this study should be considered. First, it lacked a control group. The school principal requested that there be no random selection or random assignment, explaining the opinion that all students in the Academic Success Class could benefit from the curriculum, and the school had no similar class to serve as a control group. Due to time constraints of the school's academic calendar, a waitlist control group was not feasible. For these reasons, a mixed methods study was chosen so that the treatment group could complete pretests and posttests for each variable. The measures were analyzed over time to determine changes in participants' levels of externalizing and internalizing symptoms. Another limitation of lacking a control group was that students' maturity, family, and home environments may have affected outcomes. Future studies could be improved by including an experimental single subject or randomized control group design.
The teachers both provided the treatment and rated the students' changes, which may have led to some bias in their responses due to a desire to demonstrate changes in students' symptoms. Only three teacher raters were involved in this study; thus, the varied teacher ratings may have been due to the teachers' characteristics rather than to change in the students.
Adding parent ratings could have helped improve the evaluation, though this would have been difficult, given the home situations of many of the participating students.
Another limitation was the sample size. Initially 10 students were participating in the study, but due to attrition only eight completed the program. Although the treatment group was small, all students in this class were considered at risk, and the school was looking for a social-emotional curriculum that could benefit them. The student and teacher reports showed that many students seemed to benefit from the curriculum. Future research should conduct a similar study with a larger sample of middle school students to determine the impact of the Strong Kids program on both internalizing symptoms. Another area to examine would be whether participation in the Strong Kids program impacts students' school attendance.
Additional research is needed measuring effects of Strong Kids on students over a longer time span. Future studies might also measure change in different settings (e.g., home in addition to classroom) and with multiple raters (e.g., parents as well as teachers). As some students may need more help than having Strong Kids implemented as a Tier 2 intervention, future research could investigate adding a Tier 3 intervention to improve student outcomes. Finally, the observations and reports that students' relationships with their teacher and peers improved with participation in the program could be expanded into an area for future study.

Implications and Conclusions
This study shows how middle grades teachers can work collaboratively to implement programs designed to promote SEL in students with or at-risk for EBD. Those seeking to help students with internalizing symptoms may find programs such as Strong Kids helpful in fostering social-emotional competence, particularly by engaging with students to make connections between such programs and students' real-life challenges (Caldarella et al., 2019). Schools should also consider proactively identifying at-risk students and then providing specialized SEL interventions as part of multi-tiered prevention. It is likely that some students may need additional individualized supports, beyond programs like Strong Kids, to address their social-emotional needs as they navigate the challenges of the middle grades.
The middle grades period is a time of social, emotional, and physical changes for students (Young, Caldarella, Richardson, & Young, 2011). SEL curricula such as Strong Kids can help students manage these changes by teaching them healthy coping strategies (Merrell et al., 2007). Results of the current study suggested that teachers were able to implement Strong Kids with fidelity and that the curriculum may be effective in lessening students' internalizing symptoms and increasing their socialemotional knowledge. Teachers and students found the curriculum to be predominantly positive. Future studies could include a larger sample size, a control group, and follow up data points or focus more directly on the impact on student-teacher relationships.