Findings from an Innovative Teen Pregnancy Prevention Program Evaluation of Positive Potential Middle School Program in December 2016 Rural Northwest Indiana Communities Final Impact Report for A Positive Approach to Teen Health Prepared by Z. Harry Piotrowski, MS, ITMESA, LLC, Oak Park, IL Don Hedeker, PhD, Department of Health Sciences, University of Chicago, Chicago IL Piotrowski ZH, Hedeker D. Evaluation of Positive Potential Middle School Program in Rural Northwest Indiana Communities: Findings From an Innovative Teen Pregnancy Prevention Program. Report Prepared for Office of Adolescent Health, U.S. Department of Health and Human Services (HHS), Washington DC, 2016. Acknowledgements: We recognize the people who made this evaluation possible. The authors especially want to thank the students, their teachers, and principals and superintendents in the participating schools who made all of this possible. We want to thank the many PATH staff, including Donna Golob, the Project Director and Airen Harris, the Onsite Evaluation Coordinator and the curriculum educators, Jaime Rogers, Jeremias Alicea, Toni Jongkind, and Jason Frederick, who traveled many miles daily and over five years to be role models for middle school youth. We thank Mathematica advisors for their input and guidance throughout the project, and the reviewers who made many valued suggestions in preparing the reports. We especially thank two ITMESA evaluation staff, Michelle Lee and Sherry Akey, for their expert assistance, task management and fortitude throughout this project. Finally, we thank and gratefully acknowledge Jacquelyn Crump at HHS/OASH, our Project Officer, for her support, always welcoming approach, and guidance. The Authors. This publication was prepared under Grant Number TP2AH000012-05-00 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. 2 Table of Contents I. Introduction ......................................................................................................... 7 A. Introduction and Study Overview ............................................................... 7 B. Research Questions................................................................................... 9 C. Primary and Secondary Research Questions .......................................... 10 II. Program and Comparison Programming........................................................ 11 A. Description of Program as Intended ......................................................... 11 B. Description of Counterfactual Condition ................................................... 13 III. Study Design ..................................................................................................... 14 A. Sample Recruitment................................................................................. 15 B. Study Design ............................................................................................ 16 C. Data Collection ......................................................................................... 18 1. Impact Evaluation .......................................................................... 18 2. Implementation Evaluation ............................................................ 19 D. Outcomes for Impact Analyses ................................................................ 19 E. Study Sample ........................................................................................... 22 1. Description of Sample: .................................................................. 22 F. Baseline Equivalence ............................................................................... 24 G. Methods ................................................................................................... 29 1. Impact Evaluation .......................................................................... 29 2. Implementation .............................................................................. 30 3. Sensitivity Analysis ........................................................................ 30 3 IV. Study Findings .................................................................................................. 31 A. Implementation Study Findings ................................................................ 31 B. Impact Study Findings.............................................................................. 34 C. Sensitivity Analysis ................................................................................... 40 V. Conclusion ........................................................................................................ 41 VII. Appendices ....................................................................................................... 47 A. Data used to address implementation research questions....................... 47 B. Cluster and youth sample sizes ............................................................... 49 C. Measures of baseline characteristics ....................................................... 51 D. Methods used to address implementation research questions ................ 54 E. Sensitivity of impact analyses .................................................................. 56 F. Program Impact on ”ever sexual intercourse” full regression models....... 57 G. Summary grid of program Impacts ........................................................... 61 H. Additional details on methods .................................................................. 62 I. Grades 6, 7, and 8 curriculum: sessions, activities, learning objectives, theoretical rationale ................................................................ 63 4 Tables Embedded in the Narrative III.1. Behavioral outcomes used for primary impact analysis research questions. ..................................................................................................... 20 III.2 Behavioral outcomes used for secondary impact analyses research questions. ..................................................................................................... 21 III.3A. Baseline equivalence on key demographic measures and baseline measures for youth in the Intervention and Comparison groups completing grade 8 3-months and grade 9 12-months follow-up grade 8 instruction for primary research questions. ................................................... 23 III.3.B.1. Gender: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups grade 9 12-months follow-up grade 8 intervention for secondary research questions. ...................................................................................... 25 III.3.B.2. Race and Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups grade 9 12-months follow-up grade 8 intervention for secondary research questions. ............................................................... 26 III.3.B.3.a Gender by Race/Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups, boys grade 9 12-months follow-up grade 8 intervention for secondary research questions. ............................................ 27 III.3.B.3.b Gender by Race/Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups, girls grade 9 12-months follow-up grade 8 intervention for secondary research questions. ............................................ 28 IV.1. Post-intervention estimated effects to address the primary research questions using data from 12-month follow-up in grade 9 and 3 month follow-up in 8th grade. ................................................................................... 35 IV.2. Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions.................... 36 IV.3. Gender: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. ..................................................................................................... 37 IV.4. Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. ...................................................................................... 38 5 IV.5a. Gender by Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions...................................................................... 39 IV.5b. Gender by Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions...................................................................... 39 6 I. Introduction A. Introduction and Study Overview Teen parenthood is more widespread in the U.S. than in any other developed country and it is most common in rural America. 1,2 Nearly all adolescent pregnancies are unplanned and one in four pregnancies are among teenagers.1, 2 Adolescent birth rates in rural counties are nearly one-third higher compared to the rest of the country.2 While the birth rate among teens between 1990 and 2010 was cut in half (50%) in major urban centers and by 40% in suburban counties, the decline in rural counties was 31%. 3 Youth Risk Behavior Survey-Middle School data show that 5-20% of sixth-graders and 14-42% of eight-graders have engaged in sexual intercourse. 4 About 19% of the disparity in rural and non-rural teen birth rates can be explained by differential poverty rates.3 In Indiana, with a large number of rural communities and counties with high poverty rates, the teen birth rate is above the national level and the gap between Indiana’s and the nation’s teen birth rates has increased from 2.7 to 3.6 births per 1,000 teenage girls 1990-2010.4 Early age at first sexual intercourse among young adolescents in the 6th and 7th grade is an important issue for public health policy.1, 3 Research demonstrates that onset of sexual behavior is associated with risky health behaviors such as tobacco, alcohol and other drug use, as well as violence and poor school performance. 5,6,7,8 Substance use, 1 Kearney MS, Levine PB. Why is the teen birth rate in the United States so high and why does it matter? J Econ Persp. 2012;26(2):141-166. 2 Ng AS, Kaye K. Sex in the (non) city: Teen childbearing in rural America. Washington, DC: The National Campaign to Prevent Teen and Unplanned Pregnancy, Washington, DC, 2015. 3 Waldorf B, Carriere D. Poverty and teen childbearing in rural Indiana. Purdue Extension, Center for Rural Development. Lafayette, IN, 2014. 4 Moore JM, et al. Sexual behavior of middle school students: 2009 Youth Risk behavior survey results from 16 locations. J Sch Health. 2013;83(1):61-68 5Hoffman SD, Maynard RA. (Eds.) (2008). Kids having kids: economic costs and social consequences of teen pregnancy. Washington, DC: Urban Institute Press, 2nd edition, 2008 7 peer aggression and early initiation of sex often occur together among young adolescents (ages 11-14). Planned community-based interventions can reduce adolescent pregnancy and harmful behaviors. In rural communities that want to implement effective teen pregnancy programs in middle school, there are a limited number of evidence-based programs that demonstrate slowing onset of early adolescent sexual behavior. 9,10,11 Several middle school programs have demonstrated reduction of sexual behavior with young adolescents.9-11 However, community, culture and student characteristics, such as region, religious beliefs, or race and ethnicity, can moderate and reduce positive impacts.9 There is limited evidence on programs that demonstrate a positive impact on reduction of sexual behavior in early adolescence in rural communities with largely white populations.9-11 Furthermore, national representative data from 2006-2013 shows significant declines in receipt of formal school- based sex education instruction with more rapid declines among adolescents living in nonmetropolitan or rural area. 12 6 Beets MW, Flay BR et al. Use of social and character development program to prevent substance use, violent behaviors, and sexual activity among elementary-school students in Hawaii. Am J Public Health. 2009;99(8):1438-1445. 7Rodgers JL, Koval A. Epidemic models of the onset of social activities: Applications to adolescent sexuality, smoking, drinking, and religious involvement. In Laursen B, Little TD, Card NA. (Eds.) Handbook of Developmental Research Methods. New York, NY: Guilford, 2012, pages 706-724. 8 Donovan JE, Jessor R, Costa FM. Syndrome of problem behavior in adolescence: A replication. J Consult Clin Psychol. 1988;56:762–765. 9Markham CM, Tortolero SR, Peskin FM, et al. Sexual risk avoidance and sexual risk reduction interventions for middle school youth: A RCT. J Adol Health. 2012:50(3):279-288. 10 Alford S, Bridges E, Gonzales T, Davis L, Houser D. (3rd Edition) Science and success: sex education and other programs that work to prevent teen pregnancy, HIV, and sexually transmitted infections. Advocates for Youth, Washington, DC, 2012. 11 Office of Adolescent Health. Teen Pregnancy Prevention (TPP) resource center: evidence based programs. Office of Adolescent Health, Dept. of HHS, September 11, 2015. http://www.hhs.gov/ash/oah/oah- initiatives/teen_pregnancy/db/, http://www.hhs.gov/ash/oah/oah-initiatives/teen_pregnancy/db/tpp-searchable.html. Accessed January 15, 2016. 12Duberstein LD, Maddow-Zimet I, Boonstra MS. Changes in adolescents’ receipt of sex education, 2006-2013. J Adolesc Health 2016;58:621-627. 8 Evidence-based school programs represent an effective strategy to reduce the occurrence of harmful behaviors. Communities in the U.S. are interested in implementing evidence-based and evidence-informed programs that can help young people reduce their risk for unintended pregnancy, HIV, and other sexually transmitted infections (STIs).9,10 There is limited evidence on programs that demonstrate a positive impact on prevention or reduction of sexual behavior in early adolescence in largely rural white communities in the United States.10 There are only a limited number of evidence-based programs in the 11-14 years age range, white populations, and rural communities.10,11 The Positive Potential three-year longitudinal program was developed to address this need. The Positive Potential middle school program goals are to: (a) reduce the occurrence or delay the onset of sexual behaviors, (b) reduce the occurrence of other risk behaviors, and (c) have a positive impact on psychosocial attributes that promote positive youth development among predominantly white, rural communities. This evaluation examines the impact of the program on 8th grade students 3 months and 9th grade students, 12 months after completion of the grade 8 instruction. This report is one part of an ongoing implementation and effectiveness evaluation of the Positive Potential longitudinal 3-year middle school intervention program. This evaluation was conducted as part of the Office of Adolescent Health’s Teen Pregnancy Prevention grant program and received funding in 2010. B. Research Questions This report describes the implementation to program fidelity and the rigorous evaluation impact of the Positive Potential longitudinal and innovative teen pregnancy program for rural middle school youth to delay the onset of proximal sexual behaviors that ultimately lead to pregnancy and fathering a child. 9 C. Primary and Secondary Research Questions The evaluation examined the impact of Positive Potential on middle school youth; specifically, 3 measures of risk avoidance sexual intercourse behavior (ever engaged; in past 12-months; and, in recent past 3-months); 4 measures of risk reduction sexual behaviors (non-use of condom; non-use of effective birth control with sexual intercourse in recent past 3-months; sex with multiple partners in lifetime, and oral sex). Available from the first author are the following evaluation outcome results between intervention and comparison groups: (1) 10 stage physical intimacy behavior hierarchy, for example, holding hands, touching above/below the waist, oral sex and sexual intercourse; (2) non- sexual risk behavior outcomes, and (3) positive youth development outcomes. Primary Research Questions: (1) What is the impact of the Positive Potential intervention relative to business-as- usual health instruction on the occurrence of ever engaged in sexual intercourse 12 months after completion of grade 8 instruction as reported by 9th-grade students? (2) What is the impact of the Positive Potential intervention relative to business-as- usual health instruction on the occurrence of sexual intercourse in the past 12 months one year after completion of grade 8 instruction as reported by 9th-grade students? (3) What is the impact of the Positive Potential intervention relative to business-as- usual health instruction on the occurrence of sexual intercourse in past 3 months as reported by 8th grade students 3 months after grade 8 instruction? 10 Secondary Research Questions: A series of secondary research questions were asked. One research question focused on 8th grade students asking: What is the impact of the Positive Potential intervention relative to business-as-usual health instruction on ever having sexual intercourse? Other secondary research questions focused on the 9th grade sample but included five additional sexual behavior outcomes: (1) sexual intercourse in the past 3 months; (2) sexual intercourse in the past 3 months without a condom; (3) sexual intercourse in the past 3 months without birth control, including condoms; (4) sex with two or more people; and (5) oral sex. The outcomes were looked at for the full sample of students (all 8th graders or all 9th graders), by gender (9th grade boys and 9th grade girls), by race (9th grade white-non- Hispanic or 9th Hispanic non-white youth), and by race and gender (9th grade white non- Hispanic boys, 9th grade Hispanic non-white boys, 9th grade white non-Hispanic girls, and 9th grade Hispanic non-white girls). The grade 9 gender subgroup research questions were posed at the start of the study. The race/ethnicity and gender by race/ethnicity 9th grade research questions were explored after preliminary grade 7 analyses indicated sexual behavior outcome differences between Hispanic non-white and white non-Hispanic subgroups. II. Program and Comparison Programming A. Description of Program as Intended The Positive Potential program is a longitudinal developmentally appropriate holistic intervention for middle school students in grades 6, 7, and 8. It is a new group- and school- based program developed primarily for middle school youth in predominantly white, rural 11 communities. 13,14 The program is based on psychosocial theoretical models of planned change interventions for 11 to 14 year-old adolescents in middle schools. 15 The curriculum focuses on promoting attitudes, skills and behaviors that support positive youth development. The emphasis is on a child’s positive possible self and future self, and focus on positive educational attainment, positive school performance and positive goal orientation. Health promotion activities center on risk-avoidance and risk-reduction of sexual behaviors and prevention of harmful behaviors such as use of alcohol, tobacco, and drugs; peer aggression, such as engaging in fighting, physical and cyber bullying; and viewing pornography. Positive Potential uses an instructional model that is responsive to adolescent change between grades 6 and 9 to deliver developmentally appropriate information each year.9,10,11 Initial sessions in grade 6 focus on thinking about and planning healthy futures. Next, 7th-grade students gain knowledge to make healthy choices and acquire skills to plan positive actions and to avoid and to reduce health risks. Instruction in the 8th grade promotes skill development to practice new behaviors and to recognize the hierarchy of risk with sexual behaviors. Maintenance of knowledge and reinforcement of positive changes becomes an increasing focus at end-of-academic-year annual assemblies. Students are encouraged to engage in activities and readings about risk-avoidance and developmental health-promotion strategies throughout the three years in the group sessions, during take home “charges,” journaling, and in discussion with adults and peers. 13 PATH, Inc., Positive Potential Young Adolescent Risk Reduction and Health Promotion Program. Program Materials. Distributed by PATH, Inc., Portage, IN, Website www.positiveteenhealth.org. Accessed January 15, 2016. 14 Piotrowski H, Lee MA. Evaluation of fidelity and implementation of the Positive Potential Risk Reduction and Health Promotion longitudinal middle school program. Manuscript submitted for publication, 2016. 15Dunkel C, Kerpelman J. (Eds.) Possible Selves: Theory, Research and Applications. New York, NY: Nova Science, 2004. 12 (See last Appendix I for learning objectives, methods, and theoretical framework of the Positive Potential program in grades 6, 7, 8.) Positive Potential is a supplement to the health/physical education curricula youth already receive as part of their regular school education. The program is composed of three, independent blocks of five classroom sessions and one end-of-the-year class assembly in grades 6, 7, and 8 and one start-of-the-year class assembly in grade 9. Students attend the 45- to 50-minute classroom sessions on five consecutive days in grades 6 (“Be the Exception” block), 7 (“Push the Limits” block), and 8 (“Unstoppable” block). Instruction is provided by a specially trained male–female team and features engaging and participatory interactions and multimedia presentations. PATH staff were the male-female teams implementing Positive Potential. The students also attend a 45-minute assembly at the end of each grade and at the start of grade 9. The booster assemblies, presented by program team of four health educators, are multimedia events that review content and reinforce past instruction learning objectives. In each of the four multi-media booster sessions, emphasis is on maintenance of knowledge, skills and new positive behaviors across the middle school years and beginning high school.13,14 Youth are offered a total of 15.5 hours of intervention across the 15 classroom sessions (12.5 hours) and 4 assemblies (3 hours). B. Description of Counterfactual Condition The resources for sexual health were very limited in the rural communities served throughout the program target area. Within the communities surrounding the middle schools participating in the comparison and intervention groups, no family planning or reproductive health clinics are available to provide youth with education on related 13 subjects. Youth would have to visit a primary physician or hospital to receive reproductive health instruction or assistance. The comparison-school youth, as well as the intervention-school youth, continued to participate in the usual health education instruction, after-school activities, or other community activities. Therefore, the difference between the two groups was the offer of Positive Potential as additional health education instruction. School districts in the study sample taught standard health education, which addressed STI and HIV prevention, use of condoms and contraceptives, and abstinence from sex and sexual intercourse. Districts were able to determine when, and in what order, the topics were taught—including postponing health education until high school observed for one Intervention middle school. Health education teachers used “Teen Health” Course 2, Teacher and Student Text (2009, McGraw-Hill) as a primary text. Comparison group students also attended assemblies at their schools that were coordinated to occur at the same time as the Positive Potential assemblies attended by intervention group students. The comparison group assemblies focused on topics not related to the Positive Potential instruction. An outside speaker presented information on general health and exercise. Nationally recognized speakers presented at the assembly each year and avoided any content that was presented in the intervention groups. III. Study Design The evaluation consisted of (1) mixed-methods implementation design to describe fidelity to core program elements and (2) a cluster randomized controlled trial to measure differences in student outcomes that can be causally attributed to the Positive Potential program. The sections below describe the sample recruitment, research design, data collection, outcomes for impact 14 analysis, study sample, baseline equivalence for intervention and comparison groups, and analytic approach for implementation and impact analyses. A. Sample Recruitment The recruitment of target schools began with 29 public middle and elementary schools with a 6th grade selected from five counties of northwest Indiana that consisted of predominantly rural and largely white communities. The goal was to obtain the agreement and cooperation of the principal and or superintendent to participate in the study, thus allowing students to be recruited to participate in the evaluation (survey administrations) and in the intervention (5 instructional sessions a year for three years and four assemblies). While in grades 6, 7, and 8 non-consenting students were not permitted to participate in the program instruction. As an incentive to enroll in the evaluation study, schools randomized to the comparison group and intervention group elected Positive Potential instruction in the third year of the project with entering grade 6 students: 5 of 7 intervention schools and 5 of 7 control schools selected this option. School eligibility criteria to participate were as follows: included a 6th grade class; in a rural community or in a county or areas of the counties commonly and locally regarded as rural. Eligible schools were identified in five counties that were adjacent or near the PATH office and considered counties with high adolescent health risk based on birth rates and rates of sexual behavior among adolescents. The 29 schools that had some prior contact with PATH youth and parent programs within the past 10 years or had staff that were familiar to PATH were approached for recruitment. Charter, specialized academy and special education schools were excluded. A total of 16 schools were recruited over several months in the 2011-12 school year. As schools were recruited, they were paired and randomized within pair to be a 15 comparison school or an intervention school. Establishing agreement to participate and randomization continued over a 6 month period. The schools were randomized and recruited over several months in the first cohort and first target year (2011-2012). A second cohort of students was recruited in 2012-2013 from the 14 schools with no change in the condition to which they were assigned. Of the sixteen schools that originally agreed to participate and were randomized to condition, two schools in one pair without memoranda of agreement later dropped out. Before joining the evaluation by providing grade school rosters and making arrangements for consent distribution and subsequent baseline survey administration, one of these schools elected to drop out due to a change in school leadership. The second school joined the evaluation on a trial basis. After collection of student consents and baseline surveys and receiving grade 6 instruction, the school decided not to continue. All survey data from this school were destroyed. B. Study Design The evaluation to estimate the impact of the Positive Potential program was a cluster randomized controlled trial. The study team blocked schools into pairs, based on the number of enrolled students on the 6th grade roster. An independent evaluator used a table of random numbers to perform randomization within each pair of schools, with a 50:50 probability of assignment to intervention or comparison status for 16 schools. Students in the 6th grade were identified, recruited, and enrolled into the study program and administered baseline surveys starting with cohort 1 in the 2011-12 school year. The following procedure was used to recruit and enroll students. After discussion with the principal/superintendent, a class roster was secured that included all 6th grade students. In a discussion with the health education teacher, youth were eligible if they were 16 in the 6th grade, able to read and comprehend English at least at a 5th-grade level, and provided parental consent and student assent. The classroom teacher or school administrator distributed consent packets to the 6th-grade class to take home to obtain parental consent and to provide assent to participate in the study. To ensure the decision to participate in the study was not biased by knowledge of group assignment, students were not told about school assignment (intervention or comparison group) until after the collection of consent forms and often until after baseline survey administration. Similarly, in almost all schools, the principal and health education teacher did not know the school’s assignment until after consent was obtained and often until after the baseline questionnaire had been administered. Some teachers were told of school assignment in order to prepare a class curriculum. They were told not to communicate with students or parents about school study status. For cohort 2 in the 2012-13 school year, schools remained in their assigned study group status. As in the previous year, teachers were told not to discuss school assignment with incoming 6th grade students or parents until after consent forms were returned. Each school received $1,000 for participating in the evaluation study. Additionally, schools selected incentives for their students returning consent forms. Options included: (1) a class pizza party if over 90% of students returned consent forms; (2) a t-shirt for each student who returned the consent form; or (3) a $5 gift card for each student who returned a consent form. All students received the same incentive for completing follow-up surveys: $20 per survey or, in the case of a handful of 9th graders who completed post-classroom administration $50. 17 Procedures for recruitment of students were the same for intervention and comparison groups and between cohort 1 and cohort 2 with one exception. With cohort 2, recruitment and enrollment of students in some schools started at school orientation before the official school start date. From discussion with teachers in the intervention and in the comparison groups, it is unlikely that students in cohort 1 or cohort 2 knew about the group status of the school until after the consent process and often not until the baseline surveys were administered. Institutional review board approval was obtained for the evaluation study. C. Data Collection 1. Impact Evaluation Baseline survey and Positive Potential instruction for the intervention group usually began within one or two weeks after consents were returned and the student was identified as a study participant and assigned a tracking code. Group administration of surveys was paper-pencil questionnaires. The combined sample, cohorts 1 and 2, were used to address research questions. During the staggered recruitment of schools, cohort 1 students completed baseline surveys between 10/25/2011 and 5/1/2012. Cohort 2 students completed baseline surveys in the fall of 2012. Surveys were administered 7 times (before and 3 months after classroom instruction in grades 6, 7, and 8, and 12-months after grade 8 classroom instruction with students in the 9th grade). The surveys conducted (1) at baseline (before grade 6 instruction), (2) at 3 months after grade 8 instruction completed in grade 8 and (3) at 12 months after grade 8 instruction completed in grade 9 were used in this report. Within each cohort, the timing and mode of survey administration were the same for the paired intervention and comparison group schools. 18 2. Implementation Evaluation Because Positive Potential was a new program, multiple-methods data sources provided information on implementation during each grade and for continuous staff feedback. Staff feedback is seen as maintaining fidelity and thus part of the intervention.13, 14 Implementation data on classroom instruction and assemblies were collected for each grade. Appendix A provides a breakdown of the implementation elements assessed, the types of data used to assess the element, the frequency of data collection, who collected the information. Implementation data methods of analysis, and results are summarized in this report and presented in detail in a separate report.14 D. Outcomes for Impact Analyses Tables III.1 and III.2 present details about primary and secondary impact measures. Students reported yes or no on sexual intercourse behavior items in grade 8, 3 months after grade 8 instruction and in grade 9, 12-months after grade 8 instruction: ever had sexual intercourse, sexual intercourse in past 3 months, and sexual intercourse in past 3 months with risky behavior (such as alcohol use or not using a condom) and sexual intercourse in the recent past 3-months without a condom or without effective birth control. 19 Table III.1. Behavioral outcomes used for primary impact analysis research questions. Timing of measure Outcome name Description of outcome relative to program 1. Sexual Note: Survey items are prefaced with the following instruction: “Sexual In grade 9, 12 intercourse in past intercourse is defined as making love or going all the way. By sexual months after grade 8 12 months intercourse we mean a male putting his penis into a female’s vagina.” sessions. This applies to outcomes 1-5, The variable is a yes/no measure of whether a person has had sexual intercourse in the past 12 months. The measure is taken directly from the following item on the survey: “Have you had sexual intercourse in the last 12 months?” The variable is constructed where respondents who responded yes, they have had sex, are coded as 1 and who responded no are coded as 0. Students who completed the survey but did not respond to the item, thereby a missing item response, are not included in the analytic sample for this outcome. These students are missing item response and are excluded from analysis. 2. Ever had sexual The variable is a yes/no measure of whether a person has ever had In grade 9, 12 intercourse sexual intercourse. The measure is taken directly from the following item months after grade 8 on the survey: “Have you ever had sexual intercourse?” sessions. The variable is constructed where respondents who responded yes, they have ever had sexual intercourse, are coded as 1, and who respond no are coded as 0. Students who completed the survey but did not respond to the item, thereby a missing item response, are not included in the analytic sample. 3a. Sexual The variable is a yes/no measure of whether a person has had sexual In grade 8, 3 months intercourse in the intercourse in the past three months. The measure is taken directly from after grade 8 last 3 months the following item on the survey: “Now please think about the past 3 sessions. months, have you or your partner had sexual intercourse even once?” The variable is constructed where respondents who responded yes, they had sexual intercourse in the last 3 months are coded as 1, and who responded no are coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. 3b. Sexual The variable is a yes/no measure of whether a person has had sexual In grade 9, 12 intercourse in the intercourse in the past three months. The measure is taken directly from months after grade 8 last 3 months the following item on the survey: “Now please think about the past 3 sessions. months, have you or your partner had sexual intercourse even once?” The variable is constructed where respondents who responded yes, they had sexual intercourse in the last 3 months are coded as 1, and who responded no are coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. 4. Sexual The variable is a yes/no measure of whether a person has had sexual In grade 9, 12 intercourse intercourse without using a condom. The measure is taken directly from months after grade 8 without a condom the following item on the survey: “In the past 3 months, have you or your sessions. in last 3 months partner had sexual intercourse without using a condom, even once?” The variable is constructed where respondents who responded yes, are coded as 1 and who responded no are coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. 20 Table III.2 Behavioral outcomes used for secondary impact analyses research questions. Timing of measure Outcome name Description of outcome relative to program 5. Sexual intercourse The variable is a yes/no measure of whether a person has had sexual In grade 9, 12 without effective intercourse without using effective birth control. The measure is taken months after grade 8 birth control in the directly from the following item on the survey: “In the past 3 months, sessions. last 3 months. have you or your partner had sexual intercourse without using an effective method of birth control even once?” The item is preceded with the following instruction. “The next question is about your use of effective methods of Birth Control. By effective methods, we mean the following: condoms, birth control pills, the shot (Depo Provera), the ring (NuvaRing), IUD (Mirena or Paragard), implant (Implannon), the patch.” The variable is constructed where respondents who responded yes, are coded as 1 and who responded no are coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. 6. Sex with 2 or more Note: This survey item was prefaced with the following instruction: “For In grade 9, 12 different people the next few questions, sex includes other kinds of sexual activity. Sex months after grade 8 is now defined as vaginal, oral, or anal sex with another person.” sessions. The variable is a check one of six options: I have never had sex. One person. 2 to 3 people. 4 to 5 people. 6 to 7 people. Check if other. The measure is taken directly from the following item on the grade 9 survey. During your life, with how many different people have you ever had sex?” The variable is constructed where respondents who checked 2-to-3 people or more people were coded as 1; and, who checked one person or never had sex were coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. 7. Oral sex The variable is a check one of options: I have never had sex; yes, no. In grade 9, 12 The measure is taken directly from the following item on the grade 9 months after grade 8 survey. Have you ever engaged in oral sex with another person? sessions. The variable is constructed where respondents who checked Yes were coded as 1; and, who checked never had sex or no were coded as 0. Youth who indicated they had never had sexual intercourse on the survey were coded No (0) for this item, which enables the youth to be included in the analytic sample. Note: The analytic sample for primary impact analyses are 1,415 for 3-month follow-up grade 8 respondents and 1,374 for 12 month follow-up grade 9 respondents. The sample size for the secondary analysis is dependent upon the number of respondents in that subgroup. The effective sample for analysis was reduced with separate subgroup by gender (boys, 668; girls, 706), race/ethnicity (white non-Hispanic, 1,185; Hispanic non-white, 189) and race/ethnicity by gender (white non-Hispanic boys, 574; Hispanic non- white boys, 94; and, white non-Hispanic girls, 611; Hispanic non-white girls, 95). Student responses to sexual behavior items were collected for the first time at the 12-month follow-up in grade 7 because not all schools provided permission for their inclusion in the grade 6 baseline questionnaire. The response option “other sex” in item “a” was not defined because not all schools provided permission to include oral or anal sex behaviors. A separate ever engaged in oral sex item was included at the grade 9 12-month follow-up. Sexual behavior measures, items 1-7, are from similar items in teen pregnancy prevention research.9,10,11 Sub-item “a” for composite item 8 was adapted from prior research.7,16,17 16 Piotrowski H. Evaluation of community- based abstinence education programs with adolescents, 2007-2010. Unpublished raw data, 2011 Hennessy M, Bleadley A, Fishbeing M, Jordon A. Validating an index of adolescent sexual behavior using 17 psychosocial theory and social trait correlates. AIDS Beh. 2008;12(2):321-338. 21 The source of impact analysis for the primary research questions were based on 8th grade and 9th grade surveys. The analytic samples of 1,415 8th graders and 1,374 9th graders were based on total number of youth who provided demographic data, baseline of student characteristics and all outcomes. The same 8th and 9th grade analytic samples were used to answer primary and secondary research questions. E. Study Sample In 2011-12, 16 schools were randomized to condition within 8 pairs. Two schools in the same pair block dropped out within 4 months after recruitment. See Appendix B for school cluster and youth sample sizes. School-level attrition was 12.5%. Of the 2,931 students enrolled in non-attriting schools at the time of random assignment (those who received a consent/assent form), 1,776 (970 intervention and 806 comparison) youth consented and were enrolled into the study. At the start of the study there were 827 students in cohort 1 (421 intervention and 406 comparison) and 949 in cohort 2 (549 intervention and 400 comparison). The analytic samples consisted of 1,415 grade 83- month follow-up youth and 1,374 grade 9 12-month follow-up youth who provided complete data on demographics, baseline predictors and responses to the three primary and several secondary research questions about sexual behaviors. See Table III.3A below for sample sizes for grades 8-9, boys and girls gender subgroups, for ethnicity/race white non-Hispanic and Hispanic non-white subgroups, and gender by ethnicity/race subgroups. 1. Description of Sample: Grade 8 and 9 youth are about 51% male, 96% white and 11.7% Hispanic and 2.3% non-white and 40% 13 and 57% 14 years of age in grade 8 and 52% 14 and 45% 15 years of age in grade 9. Baseline characteristics are as follows: opinions about parent belief that 22 Table III.3A. Baseline equivalence on key demographic measures and baseline measures for youth in the Intervention and Comparison groups completing grade 8 3-months and grade 9 12-months follow-up grade 8 instruction for primary research questions. Intervention Intervention Intervention Comparison versus Intervention Comparison versus mean or % mean or % comparison mean or % mean or % comparison (standard (standard difference (standard (standard difference p- Me deviation) deviation) p-value deviation) deviation) value Demographic and Completed in grade 8 3-Month follow-up Completed in grade 9 12-Month follow-up Baseline Measures after grade 8 intervention after grade 9 intervention . . All . . All . Age 11.5 (0.5) 11.5 (0.5) 0 11.5(0.5) 11.5 (0.5) 0.1 (0.197) Boys 48.1% 46.5% 1.6 (0.392) 50.0% 47.1% 2.9 (0.257) White non-Hispanic 85.0% 85.4% -0.4 (0.491) 85.1% 87.5% -1.4 (0.845) Belief, knowledge . . . . . . Opinion of parents’ . . . . . . belief about sex before marriage: - Sex before 4.0% 6.4% -2.4 (0.020) 3.9% 6.5% -2.6 (0.038) marriage is OK - Don’t know parents’ 53.5% 48.4% 5.1 (0.224) 52.7% 49.2% 3.5 (0.596) belief - Should not have 42.4% 45.1% -2.7 (0.676) 43.3% 44.2% -0.9 (0.820) sex before marriage Risk Avoidance 78.6 (22.2) 79.8 (22.3) -1.2 (0.593) 78.8 (22.1) 80.4 (21.9) -1.6 (0.426) Non-sexual behaviors . . . . . . Suspended/expelled 14.4% 9.3% 5.1 (0.102) 14.9% 8.9% 6.0 (0.001) Substance use 8.2% 9.2% -1.0 (0.763) 7.8% 8.8% -1.0 (0.612) Fighting 37.6% 31.9% 5.7 (0.461) 38.4% 30.7% 7.7 (0.003) Bullying 17.4% 16.4% 1.0 (0.326) 17.2% 15.7% 1.5 (0.285) Sample Size 764 651 . 718 656 . Source: Grades 8 and 9 sample baseline surveys. See Methods section for description of variables. Note: Difference calculations may not be exact due to rounding. Characteristics with p-values at 0.05 or below are bolded. Raw or observed means and standard deviations are reported. Equivalence was examined by regressing each baseline student covariate on the intervention indicator variable, a series of school pair indicators, and cohort indicator, while clustering standard errors at the school level and classroom level for the subgroup analytic samples. Multilevel logistic regression analyses were performed for binary characteristics and multilevel linear regression analyses were performed for continuous characteristics. P values are adjusted for clustering standard errors. The four non-sexual risk behaviors are presented in two formats in this report: continuous ordinal composite with multiple items (0-1.00 range); and, dichotomous item (yes=1, no=0 format). For all regression analyses with the student baseline four behaviors, the response measure of interest is measured as a composite of the two to five sub-items. For example, a ‘yes’ response to 2 of 5 sub-items for substance use can result in a substance use composite score of 0.40. At the same time, a “yes’ response to one or more of the 5 sub-items can result in a substance use score of 1. While the composite score provides more information in the logistic regression, we report the prevalence of non-sexual risk behavior to describe the sample and for ease in comparison with other adolescent research that includes participant predictors of sexual behavior. For example, the overall 1,374 student composite mean for baseline report of ever substance use was 2.22. The prevalence of baseline report of ever substance use was 9.2%. We performed a sensitivity analysis with the two ways of coding the four non- sexual risk behaviors. 23 sex is OK before marriage, 5.5%; knowledge about avoiding risky sexual behaviors, 79%; suspended or expelled, 12%; ever used substances such as alcohol, drugs, tobacco, 9%; ever engaged in a fight, 35%; and ever bullied someone, 16%. F. Baseline Equivalence Focusing on the full 9th grade sample of 1,374 youth (718 in Positive Potential and 656 in the comparison condition) there are no statistically significant differences on age, gender, or race/ethnicity (Table III.3.A). There are three statistically significant differences on baseline measures of: opinion of parents’ belief that sexual intercourse before marriage is OK; reports of suspensions or expulsions; and, reports of fighting. Focusing on the full 8th grade sample of 1,415 youth (764 in Positive Potential and 651 in the comparison condition), there are no statistically significant differences on age, gender, or race/ethnicity (Table III.3.A). The groups had a statistically significant difference on baseline opinion of parents’ belief that sex before marriage is OK. The subgroups based on gender, race/ethnicity, or both gender and race/ethnicity did not have statistically significant differences on age, race/ethnicity, or gender (Tables III.3. B.1, B.2, B.3a, B.3b). There were statistically significant differences for some subgroups on opinion of parents’ belief that sex before marriage is OK, on reports of suspended/expelled and reports of fighting. 24 Table III.3.B.1. Gender: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups grade 9 12-months follow-up grade 8 intervention for secondary research questions. Intervention Intervention Intervention Comparison versus Intervention Comparison versus mean or % mean or % comparison mean or % mean or % comparison (standard (standard difference (standard (standard difference deviation) deviation) p-value deviation) deviation) p-value Demographic and Completed in grade 8 3-Month follow-up Completed in grade 9 12-Month follow-up Baseline Measures after grade 8 intervention after grade 9 intervention . Boys Girls Age 11.5 (0.5) 11.5 (0.5) 0 11.5 (0.5) 11.5 (0.5) 0 White non-Hispanic 81.6% 85.1% 3.5 (0.355) 83.6% 89.6% -6.0 (0.513) Belief, knowledge . . . . . . Parent belief . . . . . . Opinion of parents’ belief about sex before marriage: 5.2% 6.4% -1.2 (0.887) 2.5% 6.6% -4.1 (0.011) - Sex before marriage is OK 62.1% 58.2% 3.9 (0.460) 43.4% 41.2% 2.2 (0.786) - Don’t know parents’ Belief 32.5% 35.3% -2.8 (0.622) 54.1% 52.2% 1.9 (0.288) - Should not have sex before marriage 73.2 (23.2) 75.3 (24.0) -2.0 (0.319) 84.3 (19.3) 84.9 (18.8) -0.5 (0.632) Risk Avoidance Suspended/ expelled 23.9% 15.5% 8.4 (0.004) 5.8% 3.1% 2.7 (0.064) Substance use 10.8% 10.0% 0.8 (0.459) 4.7% 7.7% -3.0 (0.076) Fighting 56.2% 45.3% 10.9 (0.004) 20.6% 17.8% 2.8 (0.574) Bullying 18.6% 16.5% 2.1 (0.374) 15.9% 14.9% 1.0 (0.577) Sample Size 359 309 359 347 Source: Grade 9 sample baseline surveys. See Methods section for description of variables. Note: Statistically significant intervention and comparison group differences for Grade 8 3-month youth survey responses were minimal. Baseline equivalence on key demographic measures and baseline measures for youth completing Grade 8 3-month follow-up for subgroup secondary research questions are available from the author. Baseline equivalence for secondary research questions for Grade 8 3-month follow-up are available from the first author. Difference calculations may not be exact due to rounding. Characteristics with p-values near 0.05 or below are bolded. Raw or observed means and standard deviations are reported. Equivalence was examined by regressing each baseline student covariate on the intervention indicator variable, a series of school pair indicators, and cohort indicators, while clustering standard errors at the school level and classroom level for the subgroup analytic sample. Multilevel logistic regression analyses were performed for binary characteristics and multilevel linear regression analyses were performed for continuous characteristics. P values are adjusted for clustering standard errors. 25 Table III.3.B.2. Race and Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups grade 9 12-months follow-up grade 8 intervention for secondary research questions. Intervention Intervention Intervention Comparison versus Intervention Comparison versus . mean or % mean or % comparison mean or % mean or % comparison (standard (standard difference (standard (standard difference deviation) deviation) p-value deviation) deviation) p-value Demographic and Completed in grade 8 3-Month follow-up Completed in grade 9 12-Month follow-up Baseline Measures after grade 8 intervention after grade 9 intervention . White non-Hispanic Hispanic non-white Age 11.5 (0.5) 11.5 (0.5) 0 11.5 (0.6) 11.4 (0.5) 0.1 (0.287) Boys 50.9% 45.8% 5.1 (0.079) 44.8% 56.1% -12.7 (0.470) Belief, knowledge . . . . . . Opinion of parents’ . . . . . . belief about sex before marriage: - Sex before 3.4% 6.2% -2.8 (0.060) 6.5% 8.5% -2.0 (0.496) marriage is OK - Don’t know 54.1% 50.1% 4.0 (0.615) 44.8% 42.6% 2.2 (0.515) parents’ Belief - Should not have 42.4% 43.6% -1.2 (0.779) 48.7% 48.9% -0.2 (0.868) sex before marriage Risk Avoidance 78.6 (22.0) 80.1 (21.9) -1.5 (0.577) 79.9% 81.9% -2.0 (0.274) Non-sexual . . . . . . behaviors Suspended/expell 13.7% 8.1% 5.6 (0.001) 21.5% 14.6% 6.8 (0.319) ed Substance use 7.5% 9.2% -1.7 (0.372) 9.3% 6.1% 3.2 (0.441) Fighting 37.1% 29.2% 7.9 (0.005) 45.7% 41.4% 4.3 (0.776) Bullying 16.8% 15.5% 1.3 (0.327) 19.6% 17.0% 2.6 (0.687) Sample Size 611 574 . 107 82 . See note Table III.3.B.1. 26 Table III.3.B.3.a Gender by Race/Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups, boys grade 9 12-months follow-up grade 8 intervention for secondary research questions. Intervention Intervention Intervention Comparison versus Intervention Comparison versus . mean or % mean or % comparison mean or % mean or % comparison (standard (standard difference p- (standard (standard difference deviation) deviation) value deviation) deviation) p-value Demographic and Completed in grade 8 3-Month follow-up after Completed in grade 9 12-Month follow-up Baseline Measures grade 8 intervention after grade 9 intervention . White non-Hispanic Boys Hispanic non-white Boys Age 11.5 (0.5) 11.5 (0.5) 0 11.5 (05) 11.5 (0.5) 0 Belief, knowledge . . . . . . Opinion of parents’ belief about sex before marriage: . . . . . . - Sex before marriage is OK 4.5% 6.0% -1.5 (0.591) 10.4% 8.7% 1.7 (0.923) - Don’t know parents’ belief 63.3% 59.7% 3.65 (0.489) 54.1% 50.0% 4.1 (0.818) - Should not have sex before marriage 32.1% 34.3% -2.25 (0.797) 35.5% 46.3% -10.8 (0.883) Risk Avoidance 73.3 (22.9) 74.6 (23.7) -1.3 (0.683) 72.9 (25.6) 79.3 (25.1) -6.4 (0.218) Non-sexual behaviors . . . . . . Suspended/expelled 22.5% 15.2% 7.4 (0.018 ) 33.3% 17.3% 16.0 (0.044) Substance use 9.9% 10.2% -0.3 (0.878) 16.6% 8.7% 7.9 (0.132) Fighting 54.9% 44.1% 10.8 (0.009) 64.5% 52.1% 12.4 (0.110) Bullying 18.3% 16.7% 1.6 (0.483) 20.8% 15.2% 5.6 (0.636) Sample Size 311 263 . 48 46 . See Note Table III.3.B.1. 27 Table III.3.B.3.b Gender by Race/Ethnicity: Baseline equivalence on key demographic measures and baseline measures for the youth in Intervention and Comparison groups, girls grade 9 12-months follow-up grade 8 intervention for secondary research questions. Intervention Intervention Intervention Comparison versus Intervention Comparison versus . mean or % mean or % comparison mean or % mean or % comparison (standard (standard difference p- (standard (standard difference deviation) deviation) value deviation) deviation) p-value Demographic and Completed in grade 8 3-Month follow-up after Completed in grade 9 12-Month follow-up Baseline Measures grade 8 intervention after grade 9 intervention . White non-Hispanic Girls Hispanic-non-white Girls Age 11.4 (0.5) 11.5 (0.5) -0.1 (0.692) 11.5 (0.6) 11.4 (0.5) 0.1 (0.948) Belief, knowledge . . . . . . Opinion of parents’ belief about sex before marriage: . . . . . . - Sex before marriage is OK 2.3% 6.4% -4.1 (0.021) 3.3% 8.3% -5.0 (0.123) - Don’t know parents’ belief 44.6% 42.1% 2.5 (0.831) 37.2% 33.3% 3.9 (0.618) - Should not have sex before marriage 53.1% 51.5% 1.6 (0.357) 59.4% 58.4% 1.0 (0.746) Risk Avoidance 84.2 (19.6) 84.9 (19.1) -0.7 (0.706) 85.7 (18.0) 85.2 (17.2) 0.5 (0.691) Non-sexual behaviors . . . . . . Suspended/expelle d 4.6% 2.2% 2.4 (0.056) 11.8% 11.1% 0.7 (0.941) Substance use 5.0% 8.3% -3.3 (0.093) 3.3% 2.7% 0.6 (0.264) Fighting 18.6% 16.7% 1.9 (0.683) 30.5% 27.7% 2.8 (0.546) Bullying 15.3% 14.4% 0.9 (0.717) 18.6 19.4 -0.8 (0.752) Sample Size 300 311 . 59 36 . See Note Table III.3.B.1. 28 G. Methods 1. Impact Evaluation A three-level hierarchical mixed effects logistic regression model was used to estimate effects of Positive Potential on the outcomes identified for the primary and secondary research questions. (Proportional and non-proportional odds ordinal logistic regression models were used with the physical intimacy behavior reported by 9th graders. 18,19,20 These methods with results are available from the first author.) The same logistic regression model was used to address all primary and secondary research questions about sexual behavior outcomes. In the regression adjusted difference between average outcome of students in the intervention schools and students in the comparison schools, the intervention group indicator variable coefficient provided the estimated effect of the Positive Potential program. Both classrooms and schools were treated as random intercepts. Classroom was treated as a random effect for two reasons: instruction occurred at the classroom level; and, preliminary analyses with other student characteristics revealed clustering level intra-class correlations at .02, which suggests that within a school students were grouped in a class based on some student characteristic(s). The model also included two fixed design effects: six blocked pair indicators (with the 7th pair serving as the omitted category) and a cohort indicator (equals 0 for the first cohort and 1 for the second cohort). Demographic data were used to create additional covariates: gender (male coded 1), age (baseline age minus 11 in baseline group comparison and survey age 18 Hoffman L Longitudinal analysis: modeling within-person fluctuation. New York; Routledge, 2015. 19 Hedeker D, Gibbons RD. Longitudinal Data Analysis. Hoboken, NJ: Wiley, 2006. 20 Hedeker D, Gibbons RD, Flay BR. Random-effects regression models for cluster data with an example from smoking prevention research. J Cons Clin Psychol. 1994;62(4):757-765. 29 minus 11 in regression analysis of impact), race (one indicator: white non-Hispanic, coded 1 and Non-white or Hispanic, coded 0). A student was coded Hispanic for ethnicity and non-white for race (Black, Native Hawaiian, American Indian or Asian) if the youth selected Hispanic or non-white in any survey. Additional details on other baseline measures can be found in Appendix C. Effects were estimated using an intent-to-treat approach, which does not consider the number of sessions attended. Bonferroni multiple comparison method adjusted the p value threshold to 0.0167 to determine statistical significance for the three primary measures of sexual intercourse behavior.18-19 Otherwise, statistical significance was set at p value 0.05; and, 95% confidence intervals were computed for adjusted odds ratios. Analyses were not weighted. 2. Implementation The methods used to address implementation research questions are presented in Appendix D. Implementation research questions were about whether or not benchmark targets were attained (and exceeded). These are discussed below. Percentages and averages were used to address the four implementation topics: adherence, quality, counterfactual and context. 3. Sensitivity Analysis The primary research question item about sex in the past 12 months for grade 9 showed no inconsistent responses to warrant recoding their sexual behavior responses. The only exception was with grade 8 students on one survey item for a secondary research question. six students reported inconsistent sexual behavior across survey items (e.g., indicating they were sexually active in the recent past months but also checked no to 30 ever sexual intercourse). For the benchmark analysis, the 6 students were recoded as “yes” to ever engaged in sexual intercourse because they also responded yes to non- condom use or none-use of effective birth control. A similar recoding was performed with sexual intercourse in the last 3 months for 6 grade 8 students. As a sensitivity analysis, we estimated impacts where these students were analyzed based on their raw, inconsistent findings. Appendix E presents results of all the sensitivity analyses Additional sensitivity analyses were performed to examine whether or not Hispanic non-white subgroup negative outcome results possibly varied by school (a similar approach used by Markham, et. al.9 Seven post hoc full-model all-predictor logistic regression analyses were performed to examine whether any particular school held undue influence on the primary outcome of ever sexual intercourse. Each analyses removed one matched school-pair and the adjusted odds ratio and p-value was examined. IV. Study Findings There were two goals in this evaluation: (1) to determine if Positive Potential delayed onset of sexual behavior, and (2) to assess implementation to better understand the context of Positive Potential to enable replication in middle school communities. Section IV presents the results of the implementation analysis followed by impact findings about sexual behavior impacts. A. Implementation Study Findings Implementation study focused on the extent the Positive Potential three-year program achieved and maintained fidelity standards measured by adherence and quality benchmark targets. All percentages reported below meet or exceeded target benchmarks. Indicators of program context and counterfactual comparison group experiences were examined for indication of any differential instruction not in accordance with the core 31 principals of Positive Potential. See Appendix I at the end for session activities, objectives, and theoretical framework for each grade. Adherence: The analyses of implementation demonstrate high implementation fidelity for grades 6, 7 and 8 Positive Potential instruction.13 Over 92% of the scheduled activities for each grade were completed as planned based on independent observations of a 10% sample and 90% of health educator team self-assessments. Over 86% of the participants attended the end-of-year assembly in each grade. At least 80% of youth attended at least 80% (4 of 5) of the sessions in each grade. Among the offered assemblies, 94% were completed for intervention schools and 83% for comparison schools. Average minutes for sessions was 45.0 minutes (by grade, 47.6 to 48.4; range by day of session, 40-56). Over 92% of the time, the same male-female educator conducted the session. Some comparison schools (17%) declined to have an assembly with an outside speaker. Quality: Regular school teachers observed and reported excellent or very good ratings of different components of instruction across grade and school an average of 88% of the time. Over 74% of students strongly agreed or agreed with positive ratings of instruction overall, use of supporting materials, and educator teaching, across grade and school. Over 91% of the time based on educator-pairs self-assessment and 85% of the time based on 10% independent sample assessments, results pointed to positive ratings of the educational quality of the activities and sessions, and positive ratings of student participation during the activities and the sessions overall. Students strongly agreed or agreed more than 78% of the time, across grades and schools, that the assembly will have a positive impact on own behaviors, attitudes, goals and plans about risk behaviors, sexual 32 behavior, school performance and their future. Quality ratings with 10% sample by independent observers showed consistent agreement more than 75% of the time across grade about the team’s use of teaching strategies to improve learning, positive impact on student understanding, effective presentation of materials and in 8 other areas of instruction. Counterfactual: Some intervention versus comparison group differences were observed in student ratings of content taught outside of Positive Potential. These differences were noted between the comparison and intervention groups when 8th and 9th grade students reported about instruction they had received about condoms and birth control, STDs and HIV, and sex and sexual behavior abstinence. Grade 9 intervention group students had significantly more often reported receiving instruction in the 7th or 8th grade about use of condoms and birth control, 65.6% versus 54.4% (p<0.001). Grade 8 comparison group students had significantly more often reported receiving instruction in the 6th and 7th grade about STDs and HIV (72.1% versus 64.7%, p<0.003) and about abstinence (66.8% versus 58.8%, p<0.003). There were no significant intervention- comparison group grade 9 differences in student report of STI/HIV instruction (76.2%, 74.9%) and for sex and sexual intercourse abstinence instruction (71.5%, 78.6%). There was no significance intervention-comparison group grade 8 difference in student report of condom and birth control instruction in grades 6-7 (50.5%, 52.1%). The observed modest differences reported by 8th and 9th grade students between intervention and comparison group suggest that there was not sufficient differences in instruction to differentially impact youth. 33 Context: Ad hoc focus groups with students and discussions with school teachers indicated that some of the students in a comparison school and some in one intervention school in the 6th and 7th grades were exposed to a special speaker to present sessions on the topic of abstinence. In general, ad hoc discussions with 7th and 8th grade school health education teachers showed that they maintained the usual health education instruction about sexual behavior in both the comparison and intervention schools. All 14 schools except one intervention school conducted the usual sexual behavior risk reduction and risk avoidance health education classes within grades 6-8. One middle school delayed instruction until the 9th grade. Though there was evidence of modest intervention versus comparison group differences in unplanned instruction about topics addressed in Positive Potential, in summary, implementation results demonstrated high fidelity across grades 6, 7, and 8 in the Positive Potential program. B. Impact Study Findings Focusing on the three primary research questions, there are no statistically significant effects of Positive Potential on the sexual behaviors of 8th or 9th graders, after accounting for multiple comparisons (Table IV.1). Among 9th graders, 15.74% of the Positive Potential and 19.82% of the comparison group had engaged in sexual intercourse in the past 12 months, a finding that was not statistically significant after adjusting for multiple comparisons using the Bonferroni approach. Fewer 9th grade intervention youth (18.11%) than 9th grade comparison youth (22.10%) reported ever engaging in sexual intercourse, again a finding that is not statistically significant after adjusting for multiple 34 Table IV.1. Post-intervention estimated effects to address the primary research questions using data from 12- month follow-up in grade 9 and 3 month follow-up in 8th grade. Comparison Intervention compared to Outcome measure for primary Intervention prevalence comparison mean difference (p-value research questions prevalence rates rates of difference) Sexual intercourse past 12 months (grade 9 12-month grade 8 instruction follow-up) 5.74% 19.82% - 4.08 (0.031) a Ever sexual intercourse (grade 9 12- a month grade 8 instruction follow-up) 18.11% 22.10% - 3.99 (0.039) Sexual intercourse in past 3 months (grade 8 3-month grade 8 instruction follow-up) 6.02% 4.76% 1.26 (0.282) Source: 1,374 student surveys 12-months after end of grade 8 instruction and 1,415 student surveys 3 months after grade 8 instruction. See Table III.1 for a description of the primary research questions outcome measures. Note: The prevalence rates reported here are unadjusted while the reported p-value is based on a multilevel model. The impact analyses adjusted for the following covariates: baseline demographics (gender, race/ethnicity and age at time of the survey); baseline characteristics (parent opinion about sex before marriage, risk avoidance knowledge, and non-sexual risk behaviors (school suspension/expelled, substance use, fighting, bullying); cohort 1 versus 2, and school time at grade 9 questionnaire; and, six school pair blocks. School time for grade 9 survey was included in the analysis to reduce possible bias because administration ranged from the first week of September to January even though the mean was the end of September. The error terms were adjusted for non-independence with school and classroom random effects. See Appendix F for the logistic regression analysis with student baseline predictors and design predictors. All analyses were performed with the full analytic sample of 1,374 students who completed the 12-month follow-up surveys. Supermix software routine was used for the mixed effects 3-level (student, classroom, school) logistic regression analysis models. The unadjusted intra-class correlations (or ICC) for the three analyses presented were: 0.00, 0.00, and 0.01, respectively. The adjusted odds ratios and 95% confidence intervals are as follows: sexual intercourse in past 12 months, 0.70 (0.518-0.970); ever sexual intercourse, 0.73 (0.542-0.984); and, sexual intercourse in past 3 months, 1.32 (0.791-2.22). The intervention and comparison group mean percentages are observed values. The p values, odds ratios and 95% confidence intervals are adjusted to covariates in the multilevel logistic regression models. aMultiple testing adjustment was performed. Bonferroni correction was performed on the three primary impact measures. This was to ensure that the overall experiment-wise-risk for three tests remains at . Ever had sexual intercourse and sexual intercourse in the past 12 month p-values were not below 0.0167 and thereby did not attain statistical significance. comparisons. Finally, among 8th graders, 6.02% of intervention youth and 4.76% of comparison youth reported having had sexual intercourse in the past 3 months. Examining the secondary research questions that focus on 9th graders as a whole, no statistically significant findings were found for any of the five sexual behaviors of interest: sexual intercourse in the past 3 months; sexual intercourse in the past 3 months without a condom; sexual intercourse in the past 3 months without birth control; sexual intercourse with 2 or more different people; and, ever had oral sex (Table IV.2). For four of 35 Table IV.2. Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. Intervention compared with Outcome measure for secondary Intervention Comparison comparison difference research questions Grade 9 prevalence rate prevalence rate (p-value of difference) . Grade 9 Full Sample Sexual intercourse in past 3 months 13.51 16.62 -3.11 (0.052) Sexual intercourse in past 3 months without condom 7.80 8.69 -0.89 (0.725) Sexual intercourse past 3 months without birth control 5.29 6.10 -0.81 (0.501) Sex with 2 or more different people 7.52 8.69 -1.17 (0.179) Ever had oral sex with another person 18.66 17.53 1.13 (0.568) Sample size 718 656 . Source: Grade 9 1,374 student surveys 12 months after end of grade 8 instruction. See Table III.1 for a description of the primary research questions outcome measures. Note: The prevalence rates reported here are unadjusted while the reported p-value is based on a multilevel model. The impact analyses adjusted for the following covariates: baseline demographics (gender, race/ethnicity and age at time of the survey); baseline characteristics (parent opinion about sex before marriage, risk avoidance knowledge, and non-sexual risk behaviors (school suspension/expelled, substance use, fighting, bullying); cohort 1 versus 2, and school time at grade 9 questionnaire; and, six school pair blocks. School time for grade 9 survey was included in the analysis to reduce possible bias because administration ranged from the first week of September to January even though the mean was the end of September. The error terms were adjusted for non-independence with school and classroom random effects. See Appendix F for the logistic regression analysis with student baseline predictors and design predictors. All analyses were performed with the full analytic sample of 1,374 students who completed the 12-month follow-up surveys. Supermix software routine was used for the mixed effects 3-level (student, classroom, school) logistic regression analysis models. the five (all but oral sex), the percentage of intervention youth reporting the behavior was lower than the percentage of comparison youth. Focusing on gender subgroups of youth in 9th grade, no statistically significant findings were observed for girls (Table IV.3). For boys, there were three statistically significant findings: (1) sexual intercourse in the past 12 months; (2) ever had sexual intercourse; and (3) sexual intercourse in the past 3 months. All three favored the intervention group - that is, the boys who received Positive Potential reported the behavior less frequently than boys in the comparison group. 36 Table IV.3. Gender: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. Intervention Intervention compared with compared with Outcome measure comparison comparison for secondary Intervention Comparison difference Intervention Comparison difference research questions prevalence prevalence (p-value of prevalence prevalence (p-value of Grade 9 rate rate difference) rate rate difference) . Boys Girls Sexual intercourse in past 12 months 16.43 23.30 -6.87 (0.002) 15.04 16.71 -1.67 (0.558) Ever had sexual intercourse 19.22 26.21 -6.99 (0.002) 16.99 18.44 -1.45 (0.613) Sexual intercourse in past 3 months 13.65 20.39 -6.74 (0.002) 13.37 13.26 0.11 (0.290) Sexual intercourse in past 3 months without condom 7.52 9.38 -1.86 (0.155) 8.08 8.07 0.01 (0.942) Sexual intercourse past 3 mon. without birth control 5.01 6.15 -1.14 (0.287) 5.57 6.06 -0.49 (0.692) Sex with 2 or more different people 10.31 12.62 -2.31 (0.138) 4.74 5.19 -0.45 (0.489) Ever had oral sex with another person 18.66 20.05 -1.40 (0.473 18.66 15.27 3.39 (0.065) Sample size 359 309 . 359 347 . See note Table IV 2. Examining effects for race/ethnicity subgroups of youth in 9th grade, the results are mixed (Table IV.4). Within the white non-Hispanic sample there are four statistically significant findings, all of which favor the intervention group: lower prevalence of sexual intercourse in the past 12 months; lower prevalence of ever had sexual intercourse; lower prevalence of sexual intercourse in the past 3 months; and, lower prevalence of sex with 2 or more people. Among the Hispanic non-white subgroup sample, there are six statistically significant findings all of which favor the comparison group with lower prevalence for: sexual intercourse in the past 12 months; ever had sexual intercourse; sexual intercourse in the past 3 months; sexual intercourse in the past 3 months without a condom; sexual intercourse with two or more people; and, ever had oral sex. 37 Table IV.4. Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. Intervention Intervention compared with compared with Outcome measure comparison comparison for secondary Intervention Comparison difference Intervention Comparison difference research questions prevalence prevalence (p-value of prevalence prevalence (p-value of Grade 9 rate rate difference) rate rate difference) White non-Hispanic Hispanic non-white Sexual intercourse in past 12 months 12.60 20.56 -7.96 (0.001) 33.64 14.63 19.01 (0.005) Ever had sexual intercourse 14.89 23.00 -8.11 (0.001) 36.45 15.85 20.60 (0.004) Sexual intercourse in past 3 months 10.47 17.07 -6.60 (0.002) 30.44 13.41 17.03 (0.006) Sexual intercourse in past 3 months without condom 5.29 8.54 -3.25 (0.116) 22.43 9.70 12.73 (0.008) Sexual intercourse past 3 mon. without birth control 3.60 5.75 -2.15 (0.212) 14.96 8.54 6.42 (0.165) Sex with 2 or more different people 5.07 8.71 -3.64 (0.016) 21.50 8.54 12.96 (0.018) Ever had oral sex with another person 16.37 18.74 -2.37 (0.676) 31.78 9.76 22.02 (0.001) Sample size 611 574 107 82 See Note Table IV.2 Focusing on the final set of estimated effects of interest, for gender and race/ethnicity subgroups, there are a total of 10 statistically significant findings (Tables IV.5a and IV.5b). Five of the ten statistically significant outcomes are for white non- Hispanic boys and all favor the intervention group (that is, lower rates for the Positive Potential participants). Specifically, statistically significant lower prevalence of sexual activity were found among Positive Potential program participating youth for: (1) sexual intercourse in the past 12 months; (2) ever had sexual intercourse; (3) sexual intercourse in the past 3 months; (4) sexual intercourse in the past 3 months without a condom: and, (5) sexual intercourse with 2 or more different people. The other five all have lower rates for the comparison group - one for Hispanic non-white boys and four for Hispanic non- white girls. 38 Table IV.5a. Gender by Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. Intervention Intervention compared with compared with Outcome measure comparison comparison for secondary Intervention Comparison difference Intervention Comparison difference research questions prevalence prevalence (p-value of prevalence prevalence (p-value of Grade 9 rate rate difference) rate rate difference) . Boys Girls White non-Hispanic White non-Hispanic Sexual intercourse in past 12 months 12.86 24.71 -11.85 (0.002) 12.33 17.04 -4.71 (0.689) Ever had sexual intercourse 16.08 27.76 -11.68 (0.002) 13.67 18.97 -5.30 (0.493) Sexual intercourse in past 3 months 10.29 21.67 -11.38 (0.001) 10.67 13.18 -2.51 (0.932) Sexual intercourse in past 3 months without condom 4.82 9.51 -4.69 (0.021) 5.67 7.72 -2.05 (0.953) Sexual intercourse past 3 mon. without birth control 3.22 5.70 -2.48 (0.118) 4.00 5.79 -1.79 (0.851) Sex with 2 or more different people 6.43 12.93 -6.50 (0.016) 3.67 5.14 -1.47 (0.916) Ever had oral sex with another person 16.08 22.05 -5.97 (0.242) 16.67 15.76 0.91 (0.294) Sample size 311 263 300 311 . See Note Table IV.2. * NE = p value not estimable with multi-level logistic regression analysis. Table IV.5b.Gender by Race/Ethnicity: Post-intervention estimated effects using data from grade 9, 12-months after instruction, to address the secondary research questions. Intervention Intervention compared with compared with Outcome measure comparison comparison for secondary Intervention Comparison difference Intervention Comparison difference research questions prevalence prevalence (p-value of prevalence prevalence (p-value of Grade 9 rate rate difference) rate rate difference) . Boys Girls Hispanic non-white Hispanic-non-white Sexual intercourse in past 12 months 39.59 15.22 24.37 (0.192) 28.61 13.89 14.72 (0.004) Ever had sexual intercourse 39.59 17.38 22.21 (0.446) 33.90 13.89 20.01 (0.002) Sexual intercourse in past 3 months 35.42 13.04 22.38 (0.124) 27.12 13.89 13.23 (0.011) Sexual intercourse in past 3 months without condom 25.00 8.70 16.30 (NE)* 20.34 11.11 9.23 (NE) Sexual intercourse past 3 mon. without birth control 16.67 8.70 4.97 (NE) 13.56 8.33 5.23 (NE) Sex with 2 or more different people 35.42 8.70 26.72 (0.023) 10.17 5.56 4.61 (0.483) Ever had oral sex with another person 35.42 10.87 24.55 (0.056) 28.81 11.11 17.71 (0.009) Sample size 48 46 59 36 . See Note Table IV.2. * NE = p value not estimable with multi-level logistic regression analysis. 39 Appendix F presents the full model regression results with demographic and baseline characteristics predictors on program impact on had ever engaged in sexual intercourse. There were strong associations between grade 6 baseline non-sexual risky behaviors and grade 9 overall percentage and boys and girls youth report of ever engaged in sexual intercourse. These results indicate a strong association between 6th grade student self-report of school suspension or being expelled, substance use, fighting and bullying and 9th grade report of ever engaged in sexual intercourse. Both white non- Hispanic girls subgroup (odds ratio 2.59, p<0.042) and Hispanic non-white girls subgroup (odds ratio 14.99, p<.063) predicted high likelihood of reporting in their freshman year of ever engaging in sexual intercourse. Baseline youth characteristics can be informative to refashion elements of a risk behavior prevention program. (Similar associations were noted between ever engaged in non-sexual risky behaviors reported in the 6th grade and physical intimacy behavior escalation in freshman year of high school. See first author for more information.) Appendix G presents a summary grid of the statistically significant results and sample sizes with risk avoidance and risk reduction primary and secondary sexual behavior research questions for different subgroups: (1) by 8th and 9th grades, (2) by gender for 9th grade, (3) by race/ethnicity for 9th grade, and (4) gender by race/ethnicity for 9th grade. C. Sensitivity Analysis As discussed earlier, sensitivity analyses were conducted to see if the effects were robust to different approaches to handling youth with inconsistent survey responses about sex, and whether the findings for the Hispanic non-white subgroups were driven by outlier 40 school pairs. The sensitivity results showed that the benchmark findings were robust to different assumptions about the sample. V. Conclusion The study is a cluster randomized controlled trial of student self-report of sexual behavior on two points in time: three months after 8th grade instruction and long-term 12 months after 8th grade instruction (when students were in 9th grade). Students provided survey responses on seven measures of sexual behavior. The program objectives were to examine differences in the prevalence of sexual intercourse and risky sexual behavior measures between two groups of students who continued usual health education except the intervention group was offered the Positive Potential program in grades 6-9: 12.5 hours of instruction across 15 sessions, 5 per school year, and 3 hours of additional programming across 4 assemblies, 1 per year. Implementation data with multiple-methods over the three-year program period provided strong evidence of high fidelity and maintaining instruction according to the core principles of the curriculum. The impact results indicated there was evidence of a positive intervention group impact on multiple measures of sexual behavior 12 months after grade 8 follow-up as reported by the grade 9 boys group. Follow-up analyses indicated positive intervention impact on white non-Hispanic sub-group and white non-Hispanic boys. Positive impact was observed for risk avoidance sexual intercourse and risk reduction sexual intercourse behaviors. There also was evidence of negative intervention program impact among the Hispanic non-white subgroup of students. In the Positive Potential longitudinal program, white non-Hispanic youth overall and white non-Hispanic boys specifically reported statistically significant lower occurrence of 41 (1) sexual intercourse in the past 12 months, (2) had ever sexual intercourse, (3) sexual intercourse in the past 3 months, (4) sexual intercourse without a condom in the past 3 months and (5) sexual behavior with multiple partners. Hispanic non-white youth overall and Hispanic non-white girls subgroup specifically in the Positive Potential program had statistically significant higher prevalence of (1) sexual intercourse in the past 12 months, (2) ever sexual intercourse and (3) sexual intercourse in the past 3 months. Other negative impacts included higher occurrence of sexual intercourse in past 3 months without a condom and sex with 2 or more people reported among Hispanic non-white group overall; sex with 2 or more different people reported by Hispanic non-white boys; and, ever had oral sex reported by Hispanic non-white girls. The divergent results among white non-Hispanic and Hispanic non-white students in the Positive Potential program versus comparison group were explored further. Additional sensitivity analyses, such as excluding pair of intervention and comparison school one pair at a time and reviewing possible skewed sample size across school pairs did not provide an explanation for the divergent positive versus negative ethnicity/race differences in sexual behavior outcomes The substantial divergent negative impact on white non- Hispanic youth versus Hispanic non-white youth is under further study. Conducting analyses separately for boys and girls is a method supported by other evaluations of impact in sex education studies which observed mixed effects on girl’s and boys’ sexual behavior. 21,22,23,24 Grossman et al point out that conducting the analyses separately avoids the assumption that the covariates have the same effects on having had sex regardless of gender. 21 Grossman J, Tracy AJ, Charmaraman L, et al. Protective effects of middle school comprehensive sex education with family involvement. J School Health. 2014;84(11):747. 42 The analytic sample for this study reported similar rates of sexual initiation as seen in the 2015 Indiana Youth Risk Behavior Survey (YRBS). For 9th graders in Indiana completing the YRBS, the rates were 19.8% overall, 20.6% boys, and 19.2% girls, which are lower than the United States rates of 24.1%, 27.3%, and 20.7%, respectively. 25 The comparison group, which had not received any aspect of Positive Potential by 9th grade, reported rates of 22.1% overall, 26.2% for boys, and 18.4% for girls. Although the positive findings of this study are promising and the negative findings of concern, there are some limitations among the inferences made about the several intervention impacts. Among the 29 schools deemed eligible, 13 did not participate and two schools dropped out after randomization. And 39% of students in the school did not consent/assent to participate. Though this is an efficacy trial, these events reduced the sample size and may have limited the findings. We were not able to follow-up with students who did not complete the baseline and the 3-month and 12-month follow-up surveys because Grade 6 instruction began the week after baseline group survey administration and follow-up data on missing student baseline characteristics was not available. Students often elected not to participate in the several group survey administrations and not participate when located and contacted. Students did not respond to one or more baseline questions and to sexual intercourse questions. The sample size to provide better understanding about the negative or adverse impact on 22 Coyle KK, et al. Draw the line/respect the line: a randomized trial of a middle school intervention to reduce sexual risk behaviors. Am J Public Health.2004;94(5):843-851. 23 Tortolero SR, Markham CM, Peskin MF, et al. It’s Your Game: Keep It Real: Delaying sexual behavior with an effective middle school program. J Adol Hlth 2010;46:169-179. 24 Clark LF, Miller KS, Nagy SS, etal. Adult identity mentoring: reducing sexual risk for African-American seventh grade students. J Adol Health.2005:37:337.e1-337.e10. 25 Kann L, Kinchen S, Shankilin SL, et al. Youth Risk Behavior Surveillance – United States, MMWR 2014;63(SS04),1-168. June 13, 2014. and Indiana 2015 YRBS, https://nccd.cdc.gov/youthonline/App/Results.aspx?LID=IN, accessed 8/1/2016. 43 Hispanic or non-white middle school youth was a limiting factor in efforts to reveal tenable hypotheses. Self-report of sexual intercourse behavior and other risk behaviors as well as choosing to skip those items may be subject to cognitive and social biases. Lack of information, whether from focus groups, from ad hoc youth initiated discussions with the team or school teacher initiated discussions with the team, about non-planned school or out of school instruction/exposure may not have been fully documented. Generalizability to other rural populations or urban populations is limited, particularly with a different baseline risk profiles. There are several strengths in this study. Use of a randomized controlled trial design with multi-level mixed effects regression estimates and design and student predictor adjusted standard errors produced more accurate estimates. Baseline behavior and outcome sexual behavior items were adapted from prior research thereby increasing validity of student responses.11, 16 Analyses indicated minimal non-equivalence among groups at different group administration survey time periods. Other strengths include: strong retention of participants for more than three years; use of an intent-to-treat analytic approach to enable rigorous assessment of the program; demonstration of high fidelity with core principles of Positive Potential program implementation, medium to large effect sizes for risk reduction and risk avoidance sexual behavior measures among white non-Hispanic boys. Finally, this study is unique because we recruited a sample of predominantly rural white middle school youth, the target student population of the Positive Potential longitudinal program, thereby maximizing the match between curriculum design, intervention sample, and evaluation sample. 44 A review of other evidence-based teen pregnancy prevention programs with middle school youth demonstrated that the prevalence observed program positive outcomes with white non-Hispanic youth in the Positive Potential program were similar in magnitude compared to sexual behavior education interventions with middle school populations.10,21-24 However, there is minimal discussion of statistically significant negative impact results of teen pregnancy prevention programs among youth characteristics such as different ethnicity/race subgroups or by gender.11 There have been a number of reports about iatrogenic outcomes of evaluations of interventions in group-delivery formats among youth with harmful adolescent behaviors. 26,27,28,29 For example, Lipsey27 in a 1992 meta- analysis concluded that approximately 29% of controlled interventions that focused on youth problem behaviors produced iatrogenic interventions. (See Tables 1-2 in 2016 updated findings from HHS Teen Pregnancy Prevention evidence review29 that identify “adverse effects” with three programs/studies.) It is common for short- and long-term effects of teen pregnancy prevention programs and education evaluation programs to vary among particular subgroups and for some ethnicity/race or gender subgroups to show significant impacts, even when the average intervention effect is not significant.11, 21-23 If newly introduced sexual behavior evidence- based programs are only minor variations on what is already being done in various settings such as in schools or in after-school programs, the overall average program 26 Chaney B. Reconsidering findings of “No Effects” in randomized control trials: modeling differences in treatment impacts. Am J Eval. 216:37(1):45-62. 27Lipsey MW. Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. In T D Cook, H Cooper, DS Corduroy H, Harman LV, Hedges RJ, et al. (Eds.). Meta-analysis for explanation: A casebook. New York: Russell Sage Foundation.1992; 83-125. 28 Dishion T J. McCord, J., & Poulin, F. (1999). When interventions harm: Peer groups and problem behavior. Am Pschol.1999;54(9):755–764. 29Rhule DM. Take care to do no harm: harmful interventions for youth problem behavior. Prof Psych Res Practice. 2005;36(6):618-625. 45 effects may be relatively small and thus hard to measure. From a programmatic viewpoint, it may be more appropriate to find targeted evidence-based programs that address specific youth subgroups and focus on specific prevention outcomes. 30,31 The Positive Potential longitudinal program was developed primarily for middle school youth in predominantly white, rural communities. It was evaluated with a similar population and in similar a setting. The Positive Potential program reduced the occurrence of sexual intercourse among white non-Hispanic youth and, in particular, white non- Hispanic girls. At the same time, evidence indicated that Hispanic non-white youth in the Positive Potential program may have experienced negative and adverse outcomes, that is, higher prevalence of sexual intercourse behaviors. This study affirms the positive impact of Positive Potential program on middle school adolescents with (1) benefit of health risk reduction to prevent and reduce the occurrence of risky sexual behaviors and (2) benefit of health risk avoidance to delay the onset and prevent the occurrence of sexual intercourse behaviors. The findings of this study suggest that Positive Potential can be particularly effective with middle school youth. Our findings add to the needed literature documenting the value of teen pregnancy prevention programs limited to rural and largely white communities. 30 Hedeker D. Methods for multilevel ordinal data in prevention research. Prevention Science. 2015;16(7):997- 1006. Hedeker D, Mermelstein R, Weeks K. The threshold of change model: An approach to analyzing stages of 31 change data. Ann Behav Med. 1999;21(1);61-70. 46 Appendix A. Data used to address implementation research questions for Positive Potential grades 6, 7, and 8 group sessions and for grade assemblies at the end of grade 6, 7, and 8 and the beginning of grade 9 Types of data used to assess whether the element of the Party responsible Implementation intervention was implemented Frequency/sampling of data for data element as intended collection collection Adherence . . . How often were Frequency of sessions offered All group sessions and assemblies Program staff sessions offered? and implemented was tracked delivered for each school for both Evaluation staff How many were 100%. Assemblies offered and cohorts for all grades were captured in offered? implemented were tracked 100%. MIS, 100% sample. Session and assembly tracking was performed for all intervention schools. Assembly tracking was performed for all comparison schools. What and how Student attendance was captured Attendance was measured by the Program staff much was for all group sessions offered in number of sessions out of 18 total that Evaluation staff received? grades 6, 7, and 8. Attendance at the student received. Attendance was each of four assemblies was maintained in the MIS, 100% sample. estimated from school roster for In addition to date of attendance, length intervention and comparison (number of minutes) of program schools. sessions was captured in MIS. Frequency was captured with daily attendance records at the participant level for group sessions and estimated with class roster list for participants at assemblies. Student attendance at all group sessions and assemblies are captured in the MIS What content was Number of activities offered and Educators self-assess 95% of the Program staff delivered to implemented are captured on sessions. A 10% sample of sessions Evaluation staff youth? observation forms for each of five was observed by Independent sessions for each grade. Educator evaluation staff. The 10% sample was self-assessment forms and distributed to represent day of session independent observation forms (1-5), educators (2 pairs) and grade with activities for each (grade 6, 7, 8). day/session indicated what content (activities and sessions) was delivered. See Appendix I of the curriculum for grades 6, 7 and 8 and a list of the activities for each session. Who delivered List of PATH staff members hired Data on all staff members are available Program staff material to youth? and trained to implement program, to program staff. Evaluation staff male and female educator pair for a class, throughout length of 100% recording of male-female project. Background qualifications educator pair tracked, and any cross- of staff members from staff over due to absences was also tracked. applications maintained. 100% recording of data were entered Who delivered the sessions and into the implementation/fidelity data file assemblies was tracked for all for all grades. grades. 47 Types of data used to assess whether the element of the Party responsible Implementation intervention was implemented Frequency/sampling of data for data element as intended collection collection Quality . . . Quality of youth Several sources were used: Student assessment of group class Evaluation staff engagement with educator self-assessment and instruction, 95% sample, using 8-10 program independent assessment of items with 3-4 survey versions using quality of instruction; student Likert-scale ratings to obtain responses assessment of instruction at the on several learning objectives end of the 5-session group outcomes. instruction; student assessment of assembly at the end of the Educator self-assessment of instruction, assembly; school teacher rating of 95% sample and independent 10% instruction of the block of five sample. sessions. Teacher assessment of instruction using rating scale of each session, 75% sample of participating intervention students. All three grades and both cohorts were included in the sample. Student assessment of assembly, over 90% sample per attendance at the assembly (program participants and non-program participants). Structure convenient 10% sample by independent observations. Data were collected in an implementation/fidelity data file for grades 6-9. Counterfactual . . . Experiences of Survey items were added to Students in the grade 8 3-month follow- Evaluation staff comparison student questionnaires. Each up survey were asked to report-on or condition survey asked the student to check reflect-on instruction in grades 6 and 7; a yes or no to whether they and, students in the grade 9 12-month received instruction in the post-program survey were asked to following areas: condoms or other report-on or reflect-on instruction in methods of birth control; sexually grades 7 and 8; 95% sample was transmitted diseases and HIV; attained. Data were entered into the abstinence from sex or sexual student questionnaire response file. intercourse. Context . . . External events Discussion with teachers and Discussions with teachers were ad hoc Program staff affecting feedback from student participant events. And student focus groups were Evaluation staff implementation focus groups. events conducted when available. Note: MIS = Management Information System by independent evaluator, ITMESA, LLC. 48 Appendix B. Cluster and youth sample sizes by intervention status. Number Time Period Sample Size Response Rate Interven- Compar- Interven- Compar- Number of Clusters . Total tion ison Total tion ison 1. At beginning of Recruitment 16 8 8 X X X study, schools enrolled 2. Contributed at least Baseline before 14 7 7 87.5% 87.5% 87.5% one youth at baseline grade 6 instruction 3. Contributed at least 12 month follow-up 14 7 7 87.5% 87.5% 87.5% one youth at 12-month grade 6 instruction, follow-up after grade 6 survey in grade7 instruction, 4. Contributed at least 3-month follow-up 14 7 7 87.5% 87.5% 87.5% one youth at 3-month grade 7 instruction, follow-up after grade 7 survey in grade 7 instruction 5. Contributed at least 12 month follow-up 14 7 7 87.5% 87.5% 87.5% one youth at 12-month grade 7 instruction, follow-up after grade 7 survey in grade 8 instruction 6. Contributed at least 3-month follow-up 14 7 7 87.5% 87.5% 87.5% one youth at 3-month grade 8 instruction, follow-up after grade 8 survey in grade 8 instruction 7. Contributed at least 12-month follow-up 14 7 7 87.5% 87.5% 87.5% one youth at 12-month grade 8 instruction, follow-up after grade 8 survey in grade 9 instruction Number of youth . . . . . . . (Combined cohorts) 8a. In non-attriting sites Recruitment 2,931 1,408 1,523 X X X at time of assignment* 9. Who consented . 1,776 970 806 60.6% 68.9% 52.9% 10. Contributed a Baseline in grade 6 1,747 953; 794 59.6% 67.7% 52.1% baseline survey 11. Contributed at 12- 12 month follow-up 1,622 883 739 55.3% 62.7% 48.5% month follow-up after grade 6 instruction, grade 6 instruction survey in grade 7 12. Contributed at 3- 3-month follow-up 1,655 904 751 56.5% 64.2% 49.3% month follow-up 7 after grade 7 instruction, grade 7 instruction survey in grade 7 13. Contributed at 12- 12 month follow-up 1,537 833 704 52.4% 59.2% 46.22% month follow-up in after grade 7 instruction, grade 7 instruction survey in grade 8 46. Contributed at 3 3 month follow-up 1,580 863 717 53.9% 61.3% 47.1% months follow-up after grade 8 instruction Grade 8 instruction, Primary Research Analytic Sample 1,415 764 651 48.3% 54.3% 42.7% Questions Secondary Research Boys 677 393 345 . . . Questions Subgroup Samples . Girls 738 371 306 . . . 49 Number Time Period Sample Size Response Rate Interven- Compar- Interven- Compar- Number of Clusters . Total tion ison Total tion ison . White non- 1,206 650 556 . . . Hispanic . Boys 570 319 251 . . . . Girls 636 331 305 . . . . Hispanic non- 209 114 95 . . . white . Boys 101 49 52 . . . . Girls 108 65 43 . . . 15. Contributed at 12 12-month follow-up 1,469 703 766 50.1% 49.9% 50.3% months follow-up to in grade 9 grade 8 instruction Primary Research Analytic Sample 1,374 718 656 46.9% 60.0% 43.1% Questions Secondary Research Boys 668 359 309 . . . Questions Subgroup Samples . Girls 706 359 347 . . . . White non- 1,185 611 574 . . . Hispanic . Boys 574 311 263 . . . . Girls 611 300 311 . . . . Hispanic non- 189 107 82 . . . white . Boys 94 48 46 . . . . Girls 95 59 36 . . . Note: Clusters (i.e., schools) were randomly assigned to condition in 2011-12. For cohort 1, this count includes grade 6 students who were enrolled in the schools at the time of random assignment in 2011-12. For cohort 2, this count includes grade 6 students who were enrolled in the schools in 2012-13, one year after school random assignment. Calculations are based on recruitment of Cohort 1: 1,414 students (671 Intervention and 743 Comparison); and, Cohort 2 1,517 students (737 Intervention and 780 Comparison). 50 Appendix C. Measures at baseline characteristics: youth belief, knowledge and non-sexual risk behaviors used in regression analyses of sexual behavior outcomes. Response Derivation of Scale Score for Measures at Baseline Options Analysis Beliefs and knowledge at baseline 1. Student opinion about parents’ belief about sex check one: Three dichotomous options are before marriage (number of items,1) yes = 1 dummy coded 0 or 1; yes (1) to not checked = 0 option checked and no (0) to other two options. “my parents believe I should NOT have sex before marriage” is the reference category. Which statement is true about your parents or check one: Three dichotomous options are guardian? yes = 1 dummy coded 0 or 1; yes (1) to not checked = 0 option checked and no (0) to other two options. “my parents believe I should NOT have sex before marriage” is the reference category. a. My parents believe it is OK to have sex before check one: Three dichotomous options are marriage. yes = 1 dummy coded 0 or 1; yes (1) to not checked = 0 option checked and no (0) to other two options. “my parents believe I should NOT have sex before marriage” is the reference category. b. I really DON’T KNOW what my parents think check one: Three dichotomous options are yes = 1 dummy coded 0 or 1; yes (1) to not checked = 0 option checked and no (0) to other two options. “my parents believe I should NOT have sex before marriage” is the reference category. c. My parents believe I should NOT have sex before check one: Three dichotomous options are marriage. yes = 1 dummy coded 0 or 1; yes (1) to not checked = 0 option checked and no (0) to other two options. “my parents believe I should NOT have sex before marriage” is the reference category. 2. Sexual avoidance knowledge (3) each item: Scale range 0 – 1.00. yes, I agree = 2 Each item is re-coded 0, 0.5, 1.0; don’t know = 1 mean is computed from the three no, I do not items; 1.0 is positive for sexual agree = 0 risk avoidance knowledge. Cronbach alpha is 0.75 a. The best way to avoid an STD is not to have sex. each item: Scale range 0 – 1.00. yes, I agree = 2 Each item is re-coded 0, 0.5, 1.0; don’t know = 1 mean is computed from the three no, I do not items; 1.0 is positive for sexual agree = 0 risk avoidance knowledge. Cronbach alpha is 0.75 b. Sexual activity can cause problems for teenagers. each item: Scale range 0 – 1.00. yes, I agree = 2 Each item is re-coded 0, 0.5, 1.0; don’t know = 1 mean is computed from the three no, I do not items; 1.0 is positive for sexual agree = 0 risk avoidance knowledge. Cronbach alpha is 0.75 51 Response Derivation of Scale Score for Measures at Baseline Options Analysis c. The best way to avoid pregnancy is to abstain each item: Scale range 0 – 1.00. from sex. yes, I agree = 2 Each item is re-coded 0, 0.5, 1.0; don’t know = 1 mean is computed from the three no, I do not items; 1.0 is positive for sexual agree = 0 risk avoidance knowledge. Cronbach alpha is 0.75 Non-sexual risk behaviors at baseline 3. Suspended or expelled (2) each item: Scale range 0 – 1.00. Have you ever yes = 1, no = 0 Mean is computed from the two items; 0 is positive for not expelled nor suspended. Cronbach alpha is 0.49. a. been expelled from school? each item: Scale range 0 – 1.00. yes = 1, no = 0 Mean is computed from the two items; 0 is positive for not expelled nor suspended. Cronbach alpha is 0.49. b. had an in-school or out-of-school suspension? each item: Scale range 0 – 1.00. yes = 1, no = 0 Mean is computed from the two items; 0 is positive for not expelled nor suspended. Cronbach alpha is 0.49. 4. Substance use (5) each item: Scale range 0 – 1.00. Have you ever yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. a. tried cigarette smoking, even one or two puffs? each item: Scale range 0 – 1.00. yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. b. had a drink of alcohol, other than a few sips?” each item: Scale range 0 – 1.00. yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. c. smoked cigarettes daily, that is at least one each item: Scale range 0 – 1.00. cigarette every day for 30 days? yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. d. used marijuana? Marijuana is also called grass, each item: Scale range 0 – 1.00. pot or weed? yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. e. used drugs (cocaine, heroin, or ecstasy or meth)? each item: Scale range 0 – 1.00. yes = 1, no = 0 Mean is computed from the five items; 0 is positive for no substance use. Cronbach alpha is 0.75. 52 Response Derivation of Scale Score for Measures at Baseline Options Analysis 5. Fighting (2) each item: Scale range 0 – 1.00.: yes = 1, no = 0 Mean is computed from two items; 0 is positive for not fighting. Cronbach alpha 0.85. Have you ever each item: Scale range 0 – 1.00.: yes = 1, no = 0 Mean is computed from two items; 0 is positive for not fighting. Cronbach alpha 0.85. a. been in a physical fight? each item: Scale range 0 – 1.00.: yes = 1, no = 0 Mean is computed from two items; 0 is positive for not fighting. Cronbach alpha 0.85. b. hurt someone in a fight? each item: Scale range 0 – 1.00.: yes = 1, no = 0 Mean is computed from two items; 0 is positive for not fighting. Cronbach alpha 0.85. 6. Bullying (2) each item: Scale range 0 – 1.00. Have you ever yes = 1, no = 0 Mean is computed from two items; 0 is positive for not bullying. Cronbach alpha 0.49 a. cyber bullied someone else? (For example, each item: Scale range 0 – 1.00. bullying through the internet, email, chat rooms, yes = 1, no = 0 Mean is computed from two instant messaging, social networking sites, items; 0 is positive for not websites, or texting.) bullying. Cronbach alpha 0.49 b. physically bullied someone else on school each item: Scale range 0 – 1.00. property? yes = 1, no = 0 Mean is computed from two items; 0 is positive for not bullying. Cronbach alpha 0.49 Note: The “Parent belief about sex” measure and the sexual risk avoidance knowledge measure are based on prior research.16 The measure of suspension/expulsion is a student self-report. The risk behavior measures (substance use, fighting, bullying), are adapted from the national biennial administered high school Youth Risk Behavior Survey. Cronbach alpha is a measure of consistency with score ranges of 0.49 to 0.85 in this evaluation. Higher values indicate greater reliability. 53 Appendix D. Methods used to address implementation research questions for Positive Potential grades 6, 7, and 8 group sessions and for grade assemblies at the end of grade 6, 7, and 8 and the beginning of grade 9. Implementation element Methods used to address each implementation element . Adherence Percentages were reported for sessions and activities offered. The number of sessions How often were sessions offered was calculated for each school. Total number of sessions or activities offered is offered? How many were the sum of the sessions or activities the sum offered. Average session duration is offered? calculated as the average of the session lengths offered, measured in minutes. Average grade session frequency is calculated as the total number of sessions offered for grade 6, 7 and 8 programs. Number of assembly offered was divided by number offered at each school for all grades. Average percentage of sessions attended for each student is calculated as the number of How much was received? sessions that each student attended divided by 15 total sessions offered. Overall average is the grand mean for the students who attended at least one out of 15 sessions. Average number of minutes of sessions was computed from actual number recorded for 100% sampling. Minutes of assembly attendance was estimated at 45 minutes (their typical length). Assembly attendance was calculated as number of intervention or comparison group students on the roster minus the frequency of those who had dropped out at the school. The percentage of assembly attendance is an estimate based on enrolled and non-enrolled study students since the assembly was offered to the entire class. Percentages are used throughout to report how much was received Total number of activities implemented, divided by the total number prescribed in the What content was delivered curriculum. Activities implemented divided by activities prescribed in the curriculum were to youth? obtained for each session by the educator self-assessment, and 10% sample from independent observations. Total number of staff delivering the program is a simple count of two paired educators Who delivered material to implementing the program during classroom group instruction. Percentage of staff trained youth? is calculated as the number of staff members who were trained divided by the total number of staff who delivered the program. Delivery of assembly is a count of the two pairs (four educators) present. Percent of sessions that 2 educators presented is obtained by dividing the number of times 2 educators presented by number of times 2 educators were prescribed (scheduled) as presenters. . Quality An indicator of staff-participant interactions is calculated as the percentage of observed Quality of staff-participant interactions in which the independent evaluator scored and rated the interaction on a five- interactions point scale, with a 4 or 5 (strongly agree, agree) indicating high quality Benchmark of the quality of youth engagement and quality of instruction is based on the Quality of youth following completed for each of the three grades and for the assemblies: Student engagement with program Assessment after completion of 5 sessions: 75% average for each survey item and across all items for each of three grades with a check of strongly agree, agree. Two other options on the survey item are disagree or strongly disagree. Student Assessment of Assembly: 75% average for each item and across all items for each of three grades with a check of strongly agree or agree. Three other options on the survey were undecided, disagree or strongly disagree. The items were specific for each program year and for the assembly. Teacher assessment ratings of 80% checking excellent or very good on items taping instruction by the pair of educators. The other options for the item are average or needs improvement Educator self-assessment and independent observer of quality of instruction of sessions and session activity using a rating scale of quality of instruction (yes/no rating or fair/good/excellent rating on student participation and overall rating) provided percentages to compare with a benchmark standard. The benchmark standard was an average of 75% positive rating by educator and independent observer. Overall average percentage yes and high rating was computed across all activities. Self-educator ratings and independent observer ratings enabled assessment of whether or not 75% benchmark was attained with overall session ratings of student learning and participation and other areas of instruction. Average was computed across all sessions. 54 Implementation element Methods used to address each implementation element . Counterfactual The data on the survey question on experiences of Intervention and Comparison students Experiences of at 8th and 9th grade follow-up are presented as percentages responding Yes to three counterfactual condition questions about past instruction. These data are obtained from 1,415 3-month grade 8 follow-up student questionnaires and 1,374 12-month grade 9 student questionnaires. The survey items were added to the study questionnaire. This three question bear directly on whether or not there may have been differences in student exposure to topics or content that is similar to Positive Potential. Youth were asked to reflect and think about any instruction they many have received in three sexual behavior areas: (1) condom or other methods of birth control; (2) STDs/STIs; and, (3) abstinence from sex or sexual intercourse. Youth in the 8th grade were asked about instruction that occurred in the 6th or 7th grade. Youth in the 9th grade were asked about instruction that occurred in the 7th or 8th grade. We compared the percentages of students in the Intervention group with the Comparison group for each question. A chi-square test was performed using a p value of 0.05 to determine if there was a statistically significant difference between percentages responding “yes” in the two groups. Q114. Did you have instruction in the 7th or 8th grade (6th or 7th grade) about condom or other methods of birth control? In school, part of health or physical education class ___yes ____no Q115. Did you have instruction in 7th or 8th grade (6th or 7th grade) about sexually transmitted diseases and HIV, also known as STDs/STIs In school, part of health or physical education class ____yes ____no Q116. Did you have instruction in 7th or 8th grade (6th or 7th grade) about abstinence from sex or sexual intercourse? In school, part of health or physical education class ____yes ____no . Context At least quarterly, several staff which included educators and evaluators, held discussions External events affecting to monitor implementation. These included evaluation staff and educator staff discussion implementation of results of meetings with teachers. Ad hoc discussions with students before and after sessions and focus groups results were also discussed. Note: We discuss only some of methods used to address implementation research questions. The methods discussed below were the basis for the results reported in the Implementation Study Section. The extensive report on Implementation and Fidelity of Positive Potential Program is available from the first author. 55 Appendix E. Sensitivity of impact analyses using data from Grade 8 to address the secondary research questions and data from Grade 9 to address secondary research question and Grade 9 physical intimacy behavior research question. Benchmark Approach and Sensitivity Approach and . Analysis Result Analysis Result Recoded 6 students with Behavioral outcome: inconsistent responses to sexual Response not recoded (i.e. raw comparison compared to behavior items to a “yes” response data analyzed) intervention group (n = 1,145)p (n = 1,409) 2.2% > 0.10 2.0% > 0.10 1. Ever sexual intercourse: Grade 8 All 1.6% > 0.10 1.4% > 0.10 2. Sexual intercourse in last 3 months: Grade 8 All 0.5% > 0.10 0.1% > 0.10 Boys . 2.7% > 0.10 2.5% > 0.10 Girls Logistic Logistic 7 different 7 different logistic 3. Ever sexual intercourse regression with regression with all logistic regression Grade 9 all 7 paired 7 paired regression models Hispanic non-white intervention and intervention and models estimated, with 6 subgroup only n=189 comparison comparison estimated, with paired schools (n=189) schools (n=189) 6 paired intervention and intervention and comparison comparison schools (n=189) schools (n=189) Adjusted odds Adjusted odds For each of 7 For each of 7 ratios, statistically ratios, statistically logistic logistic significant significant regressions, regressions, difference with difference with excluding one excluding one Intervention Intervention group school pair at a school pair at a group with higher with higher time, time, intervention occurrence than occurrence than intervention Hispanic non- comparison comparison group Hispanic non- white subgroup group white subgroup indicated indicated statistically statistically significantly significantly higher higher occurrence than occurrence than comparison comparison group. Results group. Results were no different were no different excluding one- excluding one- pair (two schools) pair (two at a time. schools) at a time. Source: Grade 8 3-month follow-up surveys and Grade 9 12-month follow-up surveys. Note: Table III.1 for a more detailed description of each measure and Chapter III fora description of the impact estimation methods. In sensitivity approach 1 six student responses to ever sexual intercourse and sexual intercourse in the last 3 months at grade 8 3-month follow-up were recoded from a “no” response to a “yes” response based on a “yes” response to one or more items about condom use or frequency of condom use and birth comparison use or frequency of birth control use in the last 3 months. Approach 2: additional recoding inconsistent responses. The three sets of sensitivity analyses and findings illustrate that the results were robust to alternate analytic decisions and approaches. 56 Appendix F.1a. Program Impact on “ever sexual intercourse” for full sample and two subgroups with multilevel logistic regression model student baseline covariates from grade 9, 12-months after grade 8 sessions. Full Sample Boys Girls Mean percentage Mean percentage Mean percentage Outcome measure: Difference Difference Difference Ever sexual intercourse (p-value) (p-value) (p-value) Group: Comparison 22.10% 26.21% 18.44% Intervention 18.11% 19.22% 16.99% Group Difference - 3.99% (0.039) a - 6.99% (0.002) - 1.45% (0.613) (Intervention-Comparison) Sample Size 1,374 668 706 Appendix F1b. Regression coefficients for analysis of “ever sexual intercourse” for full sample and two subgroups with multilevel logistic regression model student baseline covariates from grade 9, 12-months after grade 8 sessions. Full Sample Boys Girls Adjusted Adjusted Adjusted Outcome measure: Odds Ratio Odds Ratio Odds Ratio Ever sexual intercourse (p-value) (p-value) (p-value) Baseline predictor measure . . . (adjusted odds ratios) Group: Control vs Intervention 0.73 (0.039) a 0.51 (0.003) 1.11 (0.613) Boys 0.88 (0.514) x x White, non-Hispanic. 0.64 (0.027) 0.74 (0.313) 0.57 (0.051) Age (in years) 1.33 (0.097) 1.57 (0.062) 1.20 (0.476 Opinion of parents’ belief about . . . sex before marriage: Sex before marriage is OK 1.65 (0.096) 1.22 (0.659) 2.60 (0.024) Don’t know 0.85 (0.344) 1.04 (0.843) 0.79 (0.322) Should not have sex before marriage 0.59 (0.127) 0.37 (0.043) 0.83 (0.755) . . . Non-sexual risk behaviors Suspended or expelled 2.28 (0.023) 2.41 (0.051) 1.71 (0.445) Substance Use 15.59 (0.001) 54.11 (0.001) 8.16 (0.100) Fighting 2.27 (0.001) 2.52 (0.001) 2.42 (0.006) Bullying 2.01 (0.022) 3.14 (0.008) 1.18 (0.717) Cohort 0.98 (0.940) 1.18 (0.434) 0.80 (0.320) School Time 0.93 (0.555) 0.86 (0.425) 0.99 (0.983) Sample Size 1,374 668 706 Source: Grade 9 results from 1,374 student surveys 12-months after end of grade 8 instruction. Note: Statistical significance probability for each of the three primary research questions is adjusted to 0.167. The p value of 0.039 is not statistically significant. The regression analyses with estimated youth predictors of ever engaged in sexual intercourse are presented for the full grade 9 analytic sample and different grade 9 subgroups. The youth predictors were included in all regression models. The intervention and comparison group mean percentages are observed means. Subgroup analyses were performed: for the following: overall full sample, race/ethnicity subgroups (white non-Hispanic and Hispanic non-white), and gender subgroups (boys and girls) and boys and girls within each race/ethnicity subgroup. Ever sexual intercourse behavior outcome was adjusted as follows: baseline demographics of gender and 57 race/ethnicity and age at time of the survey. Baseline youth predictors were also included (parent opinion about sex before marriage, sexual risk avoidance knowledge, suspended or expelled from school, substance use, fighting and bullying. Cohort 1 versus 2, school time (months since baseline), and, six school pair blocks were included in the analyses with the baseline measures. The six school pair block results are not reported and are available from the author. The error terms were adjusted for non- Independence with school and classroom random effects. Supermix software was used for the mixed effects 3-level (student, classroom, school) logistic regression analysis models. a See note about Bonferroni correction in Table IV.1 on page 36. 58 Appendix F.2. Program Impact on “ever sexual intercourse” for six subgroup samples with multilevel logistic regression model student baseline covariates from grade 9, 12-months after grade 8 sessions. Hispanic White non-Hispanic White non-Hispanic Hispanic non-white Hispanic non-white White non-Hispanic non-white Boys Girls Boys Girls Mean percentage Mean percentage Mean percentage Mean percentage Mean percentage Mean percentage Outcome measure: Difference Difference Difference Difference Difference Difference Ever sexual intercourse (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) Group: Comparison 23.00% 15.85% 27.76% 19.97% 17.38% 13.89% Intervention 14.89% 36.45% 16.08% 13.67% 39.59% 33.90% Group Difference - 8.11% (0.001) 20.60% (0.004) - 11.68% -5.30% 22.38% 20.01% (Intervention-Comparison) (0.002) (0.493) (0.124) (0.002) Sample Size 1,185 189 574 611 94 95 F2b. Regression coefficients for analysis of “ever sexual intercourse” for six subgroup samples with multilevel logistic regression model student baseline covariates from grade 9, 12-months after grade 8 sessions. White non- White non- Hispanic Hispanic White non- Hispanic non- Hispanic non- . Hispanic non-white Boys Hispanic Girls white Boys white Girls Adjusted Adjusted Adjusted Adjusted Adjusted Adjusted Outcome measure: Odds Ratio Odds Ratio Odds Ratio Odds Ratio Odds Ratio Odds Ratio Ever sexual intercourse (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) Baseline predictor measure (adjusted odds ratios) . . . . . . Group: Control vs Intervention 0.56 (0.001) 3.81 (0.004) 0.51 (002) 0.84 (0.493) 1.78 (0.446) 32.17 (0.002) Boys 0.95 (0.790) 0.68 (0.378) x x x x White, non-Hispanic. x x x x x x Age (in years) 1.44 (0.058) 0.87 (0.755) 1.57 (0.062) 1.24 (0.479) 0.42 (0.293) 1.05 (0.937) Opinion of parents’ belief about sex before marriage: . . . . . . Sex before marriage is OK 1.79 (0.082) 1.56 (0.577) 1.22 (0.659) 2.59 (0.042) 0.48 (0.590) 14.99 (0.063) Don’t know 0.89 (0.549) 0.68 (0.361) 1.04 (0.843) 0.84 (0.551) 0.68 (0.604) 0.48 (0.270) Should not have sex before marriage 0.57 (0.145) 0.75 (0.769) 0.39 (0.043) 0.77 (0.599) 0.13 (0.938) 8.38 (0.329) Non-sexual risk behaviors . . . . . . Suspended or expelled 2.17 (0.060) 3.34 (0.153) 2.41 (0.051) 2.81 (0.228) 39.48 (0.012) 0.42 (0.579) Substance Use 11.86 (0.004) 92.39 (0.105) 54.11 (0.001) 11.62 (0.670) NE 0.01 (0.461) Fighting 2.13 (0.001) 3.31 (0.016) 2.52 (0.001) 2.63 (0.009) 6.82 (0.033) 5.39 (0.066) 59 White non- White non- Hispanic Hispanic White non- Hispanic non- Hispanic non- . Hispanic non-white Boys Hispanic Girls white Boys white Girls Adjusted Adjusted Adjusted Adjusted Adjusted Adjusted Outcome measure: Odds Ratio Odds Ratio Odds Ratio Odds Ratio Odds Ratio Odds Ratio Ever sexual intercourse (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) Bullying 2.58 (0.006) 0.66 (0.627) 3.14 (0.008) 0.85 (0.784) 0.036 (0.053) 5.29 (0.227) Cohort 1.04 (0.790) 0.91 (0.824) 1.18 (0.434) 0.77 (0.301) 1.56 (0.551) 1.12 (0.843) School Time 0.93 (0.561) 0.82 (0.565) 0.86 (0.425) 0.95 (0.834) 0.91 (0.887) (1.28 (0.649) Sample Size 1,185 189 574 611 94 95 See note in Appendix F.1 above. 60 Appendix G. Summary grid of program impacts: statistically significant mean percentage differences among outcome sexual behavior measures between Positive Potential and Comparison groups. Sexual Behavior Positive Potential - Comparison Group Percentage Difference At Grades 8 And 9 Measure . Results on Primary (P) and Results on Secondary Research Questions Grade 9 Secondary Research Questions Race/Ethnicity Subgroups and Grade8 and Grade 9 Gender by Race/Ethnicity Subgroups . Grade Grade 9 Grade 9 Grade 9 White Hispanic White White Hispanic Hispanic 8 All All Boys Girls All All Boys Girls Boys Girls Risk Avoidance . . . . . . . . . . Sexual P NS - NS - 7.86% + 19.01% - 11.85% NS NS + intercourse in 6.87% 14.72% past 12 months Ever had sexual NS P NS - NS - 8.11% + 20.60% - 11.68% NS NS + intercourse 6.99% 20.01% Sexual P NS - NS - 6.60% + 17.03% - 11.38% NS NS + intercourse in NS 6.74% 13.23% past 3 months Risk Reduction . . . . . . . . . . Sexual NS NS NS NS NS + - NS NE NE intercourse in 12.73% 4.69% past 3 months without a condom Sexual NS NS NS NS NS NS NS NS NE NE intercourse past 3 months without birth control Sex with 2 or . NS NS NS - 3.64% + 12.96% - 6.50% NS + 26.72% NS more different people Ever had oral sex . NS NS NS NS + 22.02% NS NS NS + 17.71% Sample size, N: . . . . . . . . . . Intervention 764 718 359 359 611 107 311 300 48 59 Comparison 651 656 309 347 574 82 263 311 46 36 Total 1,415 1.374 668 706 1,185 189 574 611 94 95 Source: Grade 8 surveys 3-months after end of grade 8 instruction and grade 9 surveys 12-months after end of grade 8 instruction. Note: This grid is a summary of results from Tables IV.1 and IV.5. Only statistically significant values p< 0.05 are displayed for ease in interpretation. Statistical significance was adjusted at p=0.167 for on the three primary research questions indicated by “P”. Elsewhere, a p <.05 defined statistical significant. A“+” indicates a positive impact of Positive Potential; a “–“ indicates a negative impact of Positive Potential, and, NS is not statistically significant. NE indicates not estimable due to small sample size. The only statistically significant group difference in grade 8 was with Hispanic non-white subgroup ever sexual intercourse: intervention vs comparison, 21.04%, 10.10%, a 10.94% difference, p< 0.028. Outcome results for grade 8 research questions by gender and race/ethnicity subgroups are available from the first author. 61 Appendix H. Additional Details On Methods The separation of ethnicity/race into two groups reflected the fact that the Positive Potential program was developed for and evaluated in rural communities with largely white populations. One of the goals of the evaluation was to examine the impact of Positive Potential on the subgroup of middle school white non- Hispanic youth. The second reason is that the Hispanic (11.7%) and non-white (2.3%) samples were relatively small and could be combined and serve as a contrast with the white non-Hispanic sample. Youth baseline characteristics were also included as predictors in all regression analyses. These are presented in Appendix D. A stratified set of analyses were performed on the secondary research questions.18-19 For a prior secondary each research questions, the regression models were performed separately for grade 8 group and for grade 9 group and for grade 9 boys and girls youth: For post hoc secondary research questions, the regression models were performed separately for grade 9 two ethnicity/race subgroups, white non-Hispanic and Hispanic non-white students; and within each ethnicity/race subgroup, boys and girls subgroups separately. The unconditional or main effects in each analysis were adjusted for the design, demographic and baseline factors.18-20, 30-31, 32 Primary and secondary research question with Grade 8 and Grade 9 total groups and Grade 9 boys and girls subgroups were proposed a prior. We tested an interaction term for intervention status conditioned on the ethnicity/race subgroup indicator (intervention=1, control=0). The estimated regression coefficient for the intervention measured the differences in Positive Potential impacts between white non-Hispanic and Hispanic non-white student subgroups. The post hoc test of group by ethnicity/race interaction was performed using the deviance difference test (likelihood ratio test) in the full logistic regression model with ever sexual intercourse outcome among grade 9 students. A significant deviance of 5211.178 without the interaction term and 5195.290 with the additional interaction term, chi- square of 15.888, 1df, p < .001 led to ethnicity/race subgroup analyses followed by gender by ethnicity/race subgroup analyses with the seven sexual behavior measure. The objectives were to test and compare the impact of Positive Potential on boys versus girls youth within white non-Hispanic subgroup of 1,185 students and within the limited sample size Hispanic non-white subgroup of 189 students (gender, intervention, comparison samples were: 94 boys subgroup, 48, 46; 95 girls subgroup, 36,95). Four baseline non-sexual risk behaviors were coded as dichotomous (y/n) for reporting prevalence description and were coded as continuous (0.0-1.0) composite score for regression analysis. Logistic regression analyses were performed with the dichotomous coding scheme to determine if there were differences in outcomes. For counterfactual data analysis, chi-square tests were performed to compare the percentages of student questionnaire responses in the intervention versus the comparison groups about past usual physical education/health instruction in grades 6-8 on the following three topics: condoms and birth control, STDs and HIV, and abstinence from sex and sexual intercourse. 32 Piotrowski H, Hedeker D. Ordered Sexual Activity Categories and Mixed-Effects Regression Models for Ordinal Outcomes: Going Beyond A Binary Indicator To Evaluate and Describe Adolescent Teen Pregnancy Prevention Program Impact. Presentation at the American Evaluation Association Annual Meeting, Nov 9-14, 2015, Chicago, IL. 62 Appendix Table I.1 Be The Exception grade 6 outline: learning objectives, methods, theoretical framework. (Material developed and tables prepared by Michelle A Lee.) Day. Theoretical Activity Title Learning Objective(s) Instructional Method Framework 1.1 Introduction After the introduction of the educators, youth will be able to identify something N/A Not applicable personal about each of the health educators. 1.2 Ice Breaker After participating in the Icebreaker activity, youth will be able to identify Interactive Game Not applicable- similarities about each other. Introduction only 1.3 Undivided After participating in the Undivided activity, youth will be able to identify the Lecture with visual Holistic (Whole Five Parts of the Whole Person (physical, mental, social, emotional, and aids Person) Theory spiritual) Youth will recognize how day-to-day decisions can affect each area of the Whole Person. Youth will discuss how choices made today will help or hinder their lives now and possibly in the future. 1.4 True Value After participating in the True Value activity, youth will be able to recognize Lecture with visual Theory of Possible that all people have value and purpose. aids engaging Selves responses 1.5 Legacy After participating in the Legacy activity, youth will be able to recognize they Lecture with visual Theory of Possible create their own legacies. aids Selves 1.6 Positive and After participating in the Positive/Negative Influences activity, youth will be Group discussion with Theory of Possible Negative able to distinguish the positive and negative people influencing their lives. visual aid Selves Influences 1.7 Transition After Transition and Charge, youth will be to recall what they have learned and Lecture Theory of Planned and Charge apply it to the charge received at the end of the day’s session. Youth will be Behavior able to define their personal legacy. Youth will be encouraged to discuss their response with a trusted adult. 2.1 P2 Review After the P2 Review, youth will be able to recall what they learned on Day One Discussion & Review Not applicable by sharing their responses from the assigned charge questions 2.2 Tug of War After participating in the Tug of War activity, youth will be able to describe the Interactive Arcs Model of effects of negative peer pressure demonstration group Motivational Design discussion, role play 2.3 N.I.C.E After participating in the N.I.C.E activity, youth will be able to understand when Media Snippet & Cognitive Theory of and how to use the N.I.C.E. refusal skills Discussion Multimedia Learning 2.4 Exit Strategy After participating in the Exit Strategy activity, youth will be able to implement Worksheet; brainstorm Problem Based refusal skills in negative peer pressure situations ways to resist the Learning pressures. 2.5 No Regrets After participating in the No Regrets activity, youth will be able to predict No Regrets booklet- Social Learning possible outcomes from decisions made in the No Regrets book group discussion. Theory 2.6 Transition After the Transition and Charge, youth will be able to recall what they learned Lecture with visual Social Norming and Charge and apply it to the charge received at the end of the day’s session. aids. Take Home theory Challenge Cognitive Theory of Multimedia Learning 3.1 P2 Review After the P2 Review, youth will recall what they’ve learned in the previous Discussion & Review Not applicable session and will be encouraged to discuss what they wrote down for their questions charge. 63 Day. Theoretical Activity Title Learning Objective(s) Instructional Method Framework 3.2 Positive/Nega After the Positive/Negative Future activity, youth will be able to define a Workbook, discussion Theory of Possible tive Future possible positive future and negative future. Youth will be able to discuss a Selves possible negative and positive future. Youth will be able to compare a positive and negative future. 3.3 Baggage After participating in the Baggage Check activity, youth will be able to Demonstration using Theory of Reasoned Check recognize the various responsibilities a teen may have. Youth will be able to role play with Action recognize the consequences that a sexual relationship can have and how it classmates followed Social Cognitive can affect their lives and their future. by discussion Theory 3.4 Ashley’s After watching Ashley’s story, youth will be able to recognize the potential to Short Video Social Norming Story overcome obstacles and achieve their positive futures. Theory 3.5 A.C.T Skills After participating in the A.C.T. activity, youth will be able to restate three skills Media Snippet & Cognitive Theory of for decision-making. Discussion Multimedia Learning 3.6 Transition After the Transition and Charge, youth will be able to recall what they learned Visual aids. Take Social Norming and Charge and apply it to the charge received at the end of today’s session. Home Challenge 4.1 P2 Review After the P2 Review, youth will be able to recall what they learned in the Discussion & Review Not applicable previous sessions by sharing their responses from the assigned charge questions 4.2 Influences After participating in the Influences and Outcomes activity, youth will be able Media Snippets, Social Norming and to recognize that choices have positive and negative outcomes Discussion Theory Outcomes 4.3 Unhealthy After participating in the Unhealthy Relationships activity, youth will be able to Discussion ARCS Model of Relationships identify specific behaviors and feelings that are most common in unhealthy Motivational Design relationships 4.4 Strong After participating in the Strong Foundations activity, youth will be able to Visual Aids, Cognitive Theory of Foundations identify the five stages in a healthy relationship. Discussion, Lecture Multimedia Learning 4.5 Transition After the transition and Charge, youth will recall what they learned and apply it Take Home Challenge Social Norming and Charge to the charge received at the end of today’s session. 5.1 P2 Review After the P2 Review, youth will be able to recall what they learned in the Discussion & Review Not applicable previous sessions by sharing their responses from the assigned charge questions 5.2 No Regrets After participating in the No Regrets review, youth will be able to examine Discussion & Review Theory of Reasoned Act Review potential positive and potential negative consequences of their choices. questions Social Cognitive Theory 5.3 True After participating in the True That/That’s Wack activity, youth will be able to Slide Presentation, Cognitive Theory of That/That’s identify the four types of bullying. Discussion Multimedia Learning Whack Youth will be able to restate the four tips to help in a bullying situation 5.4 It’s Not Too After participation, youth will be encouraged to persevere regardless of past Discussion Social learning theory Late mistakes and future obstacles. 5.5 Measure Your After participating in the Measure Your Life activity, youth will be able to Visual Aids, ARCS Model of Life recognize how choices made in the present can potentially affect their futures Discussion & Motivational Design engagement 64 Day. Theoretical Activity Title Learning Objective(s) Instructional Method Framework 5.6 For What It’s After the For What It’s Worth activity, youth will be able to recognize that Visual Aid, Lecture, Holistic (Whole Worth mistakes are not a determinant of personal self-worth Discussion Person) Theory: Youth will be able to recognize their innate value despite past mistakes or Bandura's Theory of outward physical appearance. Self Efficacy 5.7 Final After the transition and Charge, youth will be able to make healthy choices in Workbook, Discussion Holistic (Whole Transition order to reach their goals and leave a positive legacy Person) Theory: and Charge Bandura's Theory of Self Efficacy 65 Positive Potential Theoretical Framework Holistic (Whole Person) Theory: parts of any whole cannot exist and cannot be understood except in their relation to the whole. The parts of an individual include physical, emotional, social, mental and spiritual and choices made affect both individual parts as well as the whole). Theory of Possible Selves: a person's motivation is determined by a balance of positive and negative ways people see themselves in the future. Individuals able to imagine both possible positive and negative futures are more likely to work toward life goals and achieve future success. Theory of Planned Behavior: specifies the nature of relationships between beliefs and attitudes. According to these models, people's evaluations of, or attitudes toward behavior are determined by their accessible beliefs about the behavior, where a belief is defined as the subjective probability that the behavior will produce a certain outcome. ARCS Model of Motivational Design-(Attention: Relevance: Confidence and Satisfaction). This activity gets students Attention through Active participation (adopting role-play or other hands-on methods to get learners involved with the material or subject matter) Social learning theory: people learn new behavior through observational learning of the social factors in their environment. (Bandura, Rotter) Problem Based Learning: Students work as problem-solvers in small collaborative groups. (Macmaster University) Social Norming: Individuals are strongly influenced by what they think their peers are doing or thinking. Students will often act according to what they think is “normal” or what their peers think is normal. (Berkowitz and Perkins) Cognitive Theory of Multimedia Learning-people learn more deeply from words and pictures than from words alone (Mayer) Theory of Reasoned Action: a person's volitional (voluntary) behavior is predicted by his/her attitude toward that behavior and how he/she thinks other people would view them if they performed the behavior. A person's attitude, combined with subjective norms, forms his/her behavioral intention. (Fishbein & Ajzen) Social Cognitive Theory: Human functioning is the product of a dynamic interplay of personal, behavioral, & environmental influences (Bandura) Bandura's Theory of Self Efficacy: Youth can regulate their level of physiological activation through their belief in self-efficacy, which is to say their beliefs in their own capabilities. Educators then challenge students to move forward past any regretful decisions and imagine themselves in the positive possible future self and that they CAN Make healthy choices for their future. (Bandura) Note: Indiana State Standards for Health and Wellness are available at: http://www.doe.in.gov/sites/default/files/standards/health-and- wellness/2010_health_education_standards_literacy.pdf (Accessed June 22, 2016) 66 Appendix Table I.2 Push the Limits grade 7 outline: learning objectives, instructional methods, theoretical framework Day. Instructional Activity Title Objective(s) Method Theoretical Framework 1.1 Introduction Introduction of Educators by classroom teacher. N/A Not applicable 1.2 After participating in the Icebreaker activity, youth are challenged to learn Interactive Game Not applicable- Ice Breaker how to make difficult decisions. Introduction only 1.3 After the Rundown and Review, youth will be able to recall what they’ve Lecture with visual Holistic (Whole Person) Rundown and aids Theory Review learned in 6th grade. 1.4 After participating in The Gift Part 1 activity, youth will be able to consider Positive Youth The Gift Object Lesson options based on instant/delayed gratification and make a choice. Development 1.5 After participating in the Act Like a Man vs Lady Like activity, youth will be Act Like a Man vs Lecture with visual Positive Youth able to identify common stereotypes, summarize the effects of Lady Like aids Development stereotyping and recognize that stereotypes can be hurtful. 1.6 Building Self After participating in the Building Self Confidence activity, youth will be Positive Youth Slide Presentation Confidence able to identify strategies for increasing one’s self- confidence. Development 1.7 After the Transition and Charge, youth will recall what they’ve learned and Lecture/Discussion Transition and apply it to the charge they receive at the end of the session which is to Theory of Possible Charge identify their strengths and weaknesses. Youth will be encouraged to Selves discuss their response with a trusted adult. 2.1 After the Rundown and Review, youth will recall what they’ve learned in Lecture/Visual Rundown and Theory of Possible the previous session and will be encouraged to discuss what they wrote Aids/Question and Review Selves down for their charge Answer 2.2 After participating in the Whole Truth activity, youth will be able to identify Lecture/Visual The Whole Truth types of media and recognize negative messages in the media. Aids/Question and Social Norming Theory Answer 2.3 After participating in the Counter Culture activity, youth will be able to Video, Slide Counter Culture identify possible effects of viewing pornography. Presentation Social Norming Theory Discussion 2.4 After participating in the Entrapment exercise, youth will define Slide Presentation, pornography; Youth will discover the effects pornography can have on the discussion Social Norming; Whole Entrapment whole person and their future; and Youth will develop skills on how to Person/Holistic Theory avoid the pornography trap. 2.5 After the Transition and Charge, youth will recall what they’ve learned and Slide Presentation; Theory of Possible Transition and create a list of five major goals they hope to achieve and five things that discussion Selves Charge could hold them back. 3.1 After the Rundown and Review, youth will recall what they’ve learned in Discussion & Review Rundown and Theory of Possible the previous session and will be encouraged to discuss what they wrote questions Review Selves down for their charge. 3.2 After participating in the Split Decisions activity, youth will recognize how Video, Slide Theory of Possible Split Decisions choices they make today can affect their lives now and in their future. Presentation Selves 67 Day. Instructional Activity Title Objective(s) Method Theoretical Framework 3.3 After participating in the Enlightenment activity, youth will understand how sexual activity puts them at risk for STD/STIs; Youth will understand how Object Lesson, Slide Social Norming, Whole Enlightenment sexual activity puts them at risk for pregnancy; Youth will understand how Presentation Person/Holistic Theory STI’s are spread, the symptoms they cause and how to best prevent STI’s and pregnancy. 3.4 Transition and After the Transition and Charge, youth will recall what they’ve learned and Positive Youth Discussion Charge identify three resources they can access to achieve their goals. Development 4.1 After the Rundown and Review, youth will recall what they’ve learned in Discussion & Review Social Norming Rundown and the previous session and will be encouraged to discuss what they wrote questions Cognitive Theory of Review down for their charge. Multimedia Learning 4.2 After participating in the Journey activity, youth will be able to identify Booklet, lecture, Theory of Possible Journey several careers/jobs related to their own interests and skills. discussion Selves 4.3 After participating in the Stages/Timeline activity, youth will be able to Discussion/Visual Theory of Possible Stages/Timeline compare and contrast the benefits of waiting (delayed) or instant Aids Selves; Theory of gratification. Possible Selves 4.4 After the Transition and Charge, youth will recall what they’ve learned and Take home activity Transition and Theory of Possible create a timeline with a minimum of 10 action steps for the next stage of Timeline Charge Selves their lives. 5.1 After the Rundown and Review, youth will recall what they’ve learned in Discussion, Question Rundown and Theory of Possible the previous session and will be encouraged to discuss what they wrote and Answer Review Selves down for their charge. 5.2 After participating in the Counter Culture activity, youth will be able to Discussion, Question Theory of Possible Counter Culture identify with other youth/peers who have overcome obstacles and living a and Answer Selves, Social Norming positive healthy life. Theory 5.3 After participating in the Building My Legacy, youth will identify the Slide Presentation, Social Norming Theory; Building My obstacles they have overcome as well as obstacles overcome by others Discussion, Question Theory of Possible Legacy (family, peers, and community). & Answer Selves 5.4 After participating in the Q&A, youth will be better informed by having their Question & Answer, Q&A Not applicable questions answered. discussion 5.5 After participating in The Gift Part 2 activity, youth will be able to compare Object Lesson Theory of Possible The Gift Part II and contrast the benefits of waiting (delayed) or instant gratification. Selves, Positive Youth Development 5.6 After the Transition and Charge, youth will sign a commitment to them- Commitment Card Final Transition Social Norming, Theory selves to make healthy choices that will help them reach their goals and and Charge of Possible Selves leave a positive legacy 5.3 After participating in the Building My Legacy, youth will identify the Slide Presentation, Social Norming Theory; Building My obstacles they have overcome as well as obstacles overcome by others Discussion, Question Theory of Possible Legacy (family, peers, and community). & Answer Selves 68 Positive Potential Theoretical Framework Holistic (Whole Person) Theory: parts of any whole cannot exist and cannot be understood except in their relation to the whole. The parts of an individual include physical, emotional, social, mental and spiritual and choices made affect both individual parts as well as the whole). Theory of Possible Selves: a person's motivation is determined by a balance of positive and negative ways people see themselves in the future. Individuals who are able to imagine both possible positive and negative futures are more likely to work toward their life goals and achieve future success. Theory of Planned Behavior: specifies the nature of relationships between beliefs and attitudes. According to these models, people's evaluations of, or attitudes toward behavior are determined by their accessible beliefs about the behavior, where a belief is defined as the subjective probability that the behavior will produce a certain outcome. ARCS Model of Motivational Design-(Attention: Relevance: Confidence and Satisfaction). This activity gets students Attention through Active participation (adopting role-play or other hands-on methods to get learners involved with the material or subject matter) Social Learning Theory: people learn new behavior through observational learning of the social factors in their environment. (Bandura, Rotter) Problem Based Learning: Students work as problem-solvers in small collaborative groups. (Macmaster University) Social Norming: Individuals are strongly influenced by what they think their peers are doing or thinking. Students will often act according to what they think is “normal” or what their peers think is normal. (Berkowitx and Perkins) Cognitive Theory of Multimedia Learning-people learn more deeply from words and pictures than from words alone (Mayer) Theory of Reasoned Action: a person's volitional (voluntary) behavior is predicted by his/her attitude toward that behavior and how he/she thinks other people would view them if they performed the behavior. A person's attitude, combined with subjective norms, forms his/her behavioral intention. (Fishbein & Ajzen) Social Cognitive Theory: Human functioning is the product of a dynamic interplay of personal, behavioral, & environmental influences (Bandura) Bandura's Theory of Self Efficacy: Youth can regulate their level of physiological activation through their belief in self-efficacy, which is to say their beliefs in their own capabilities. Educators then challenge students to move forward past any regretful decisions and imagine themselves in the positive possible future self and that they CAN Make healthy choices for their future. (Bandura)Note: Indiana State Standards for Health and Wellness are available at: http://www.doe.in.gov/sites/default/files/standards/health-and-wellness/2010_health_education_standards_literacy.pdf (Accessed June 22, 2016) 69 Appendix Table I.3 Unstoppable grade 8 outline: learning objectives, instructional methods, theoretical framework Instructional Theoretical Day/Act Title Learning Objective(s) Method Framework 1.1 Introduction Introduction of Educators by classroom teacher. N/A Not applicable- Introduction only 1.2 Rundown and After the Rundown and Review, youth will be able to recall what they have Question and Not applicable Review learned in 6th and 7th grade. Answer 1.3 What goes around After participating in the What Goes Around Comes Around activity, youth will be Interactive Game Holistic (Whole comes around. able to identify the consequences of having multiple sexual partners and its Person) Theory effects on all Five Parts of the Whole Person. 1.4 Bonding Power After participating in the Bonding Power activity, youth will increase their Object Lesson Holistic (Whole knowledge about how sexual relationships create strong bonds between two Person) Theory people. Youth will define Oxytocin and be able to explain the role it plays in the body during sexual activity. Youth will recognize the effects that a sexual relationship can have on the Five Parts of the Whole Person. 1.5 Road to Romance After participating in the Road to Romance Activity, youth will identify the stages Lecture with visual Positive Youth of physical intimacy and which behaviors are low risk, high risk, or no risk for aids Development contracting STD’s or becoming pregnant. Youth will identify personal limitations and boundaries regarding sexual physical contact. Youth will identify the specific types of sexual physical contact that fit within their personal boundaries; Youth will identify benefits of waiting to have sex. 1.6 Transition and After participating in the in the Transition and Charge, youth will recall what they Slide Presentation Positive Youth Charge have learned and apply it to the charge they receive at the end of the session Development which is to choose their own sexual boundary and explain why they set it there; Youth will be encourage to discuss their response with a trusted adult 2.3 After participating in the Streamline activity, youth will identify the six most Object lesson, Slide ARCS Model of common STDs among their age group and whether they are viral or presentation, Motivational Design Streamline bacterial/treatable or incurable; how they are transmitted; signs and symptoms of Question and each; risk reduction of condoms; and abstinence as the only way of preventing Answer, discussion STDs, and unplanned pregnancy. 2.4 After the Transition and Charge, youth will recall what they’ve learned and apply Question and Transition and it to the charge they receive at the end of the session which is to describe the Answer, and Positive Youth Charge best way to avoid consequences from sexual activity, answer why they might discussion Development want to practice abstinence, and write down who they can talk to about it. 3.1 After the Rundown and Review, youth will recall what they’ve learned in the Question and Rundown and Positive Youth previous session and will be encouraged to discuss what they wrote down for Answer and Review Development their charge. discussion 3.2 After participating in the Multiple Choices activity, youth will identify the various Cognitive Theory of Multiple Choices types of contraception available including their effectiveness and possible Slide Presentation Multimedia Learning drawbacks. 70 Instructional Theoretical Day/Act Title Learning Objective(s) Method Framework 3.3 After participating in the Multiple Choices game, youth will be able to recall the Multiple Choices Positive Youth different types of contraception available to them including its effectiveness and Slide Presentation Game Development drawbacks. 3.4 After participating in the Are You In Control activity, youth will be able to under- Object Lesson Theory of Planned Are you in Control stand the effects alcohol has on the Whole Person. Behavior Youth will be able to identify how alcohol can alter sexual decisions. 3.5 After participating in the Baby Mama Drama activity, youth recognize the Slide Presentation Social Norming Baby Mama dangers of, and connection to, teen drinking and sexual activity. Youth will Drama identify the benefits of not compromising their personal boundaries. 3.6 After the Transition and Charge, youth will recall what they’ve learned and apply Question and Theory of Possible Transition and it to the charge they receive at the end of the session, which is to describe how Answer Selves; Theory of Charge they can avoid or get out of a risky sexual situation and describe how engaging Possible Selves in sexual activity can affect their future goals. 4.1 After the Rundown and Review, youth will recall what they’ve learned in the Question and Rundown and Theory of Possible previous session and will be encouraged to discuss what they wrote down for Answer, discussion Review Selves their charge. 4.2 After participating in the Refusal Skills activity, youth will be able to identify Question and Social Norming, Refusal Skills effective problem solving/refusal skills. Answer, discussion, Problem Based worksheet Learning 4.3 After participating in the Streamline 2 activity, youth will demonstrate the ability to Role Play Social Norming Streamline II refuse sex and choose abstinence with a partner; Youth will be able express confidence in their ability to say “no 4.4 After the Transition and Charge, youth will recall what they’ve learned and apply Journaling, question Theory of Possible it to the charge they receive at the end of the session which is to describe an and answer, Selves Transition and assigned scenario where there is risky behavior, identify the type of pressure discussion Charge involved, including the risks and ways to refuse the pressure; and Youth will be encouraged to discuss their response with a trusted adult. 5.1 After the Rundown and Review, youth will recall what they’ve learned in the Question and Rundown and Theory of Possible previous session and will be encouraged to discuss what they wrote down for Answer, discussion Review Selves their charge. 5.2 After participating in the Possible Future activity, youth will recognize personal Slide Presentation, Theory of Possible strengths and weaknesses; Youth will create images of their possible self, Video, worksheet Possible Futures Selves, Positive expected self, and feared self; Youth will create goals that align with their desired Youth Development future possible self. 5.3 After the Transition and Charge, youth will if they choose sign a commitment to Question and Transition and Theory of Possible themselves to make healthy choices that will help them reach their goals and Answer, discussion Charge Selves leave an amazing legacy 71 Positive Potential Theoretical Framework Holistic (Whole Person) Theory: parts of any whole cannot exist and cannot be understood except in their relation to the whole. The parts of an individual include physical, emotional, social, mental and spiritual and choices made affect both individual parts as well as the whole). Theory of Possible Selves: a person's motivation is determined by a balance of positive and negative ways people see themselves in the future. Individuals who are able to imagine both possible positive and negative futures are more likely to work toward their life goals and achieve future success. Theory of Planned Behavior: specifies the nature of relationships between beliefs and attitudes. According to these models, people's evaluations of, or attitudes toward behavior are determined by their accessible beliefs about the behavior, where a belief is defined as the subjective probability that the behavior will produce a certain outcome. ARCS Model of Motivational Design-(Attention: Relevance: Confidence and Satisfaction). This activity gets students Attention through Active participation (adopting role-play or other hands-on methods to get learners involved with the material or subject matter) Social Learning Theory: people learn new behavior through observational learning of the social factors in their environment. (Bandura, Rotter) Problem Based Learning: Students work as problem-solvers in small collaborative groups. (Macmaster University) Social Norming: Individuals are strongly influenced by what they think their peers are doing or thinking. Students will often act according to what they think is “normal” or what their peers think is normal. (Berkowitx and Perkins) Cognitive Theory of Multimedia Learning-people learn more deeply from words and pictures than from words alone (Mayer) Theory of Reasoned Action: a person's volitional (voluntary) behavior is predicted by his/her attitude toward that behavior and how he/she thinks other people would view them if they performed the behavior. A person's attitude, combined with subjective norms, forms his/her behavioral intention. (Fishbein & Ajzen) Social Cognitive Theory: Human functioning is the product of a dynamic interplay of personal, behavioral, & environmental influences (Bandura) Bandura's Theory of Self Efficacy: Youth can regulate their level of physiological activation through their belief in self-efficacy, which is to say their beliefs in their own capabilities. Educators then challenge students to move forward past any regretful decisions and imagine themselves in the positive possible future self and that they CAN Make healthy choices for their future. (Bandura) Note: Indiana State Standards for Health and Wellness are available at: http://www.doe.in.gov/sites/default/files/standards/health-and- wellness/2010_health_education_standards_literacy.pdf (Accessed June 22, 2016) 72