Findings from an Innovative Evaluation of Haitian- Teen American Responsible Teen Pregnancy Prevention Program in Eastern Final Impact Report for Massachusetts Boston Medical Center August, 2016 Prepared by Mathilda B. Ruwe MD, MPH, Ph.D. Youth and Family Enrichment Services (YOFES) Lois McCloskey, Dr.PH Boston University School of Public Health Alan Meyers, MD, MPH Boston University School of Medicine Nicole Prudent, MD, MPA Boston University School of Medicine Marie Ketly Foureau-Dorsinville, MD Boston Medical Center Recommended Citation: Ruwe MB, McCloskey L. Meyers A, Prudent N, and Foureau-Dorsinville M. Evaluation of Haitian American Responsible Teen. Findings from the Replication of an Evidence-based Teen Pregnancy Prevention Program in Eastern Massachusetts, August 2016. Acknowledgements: The authors extend their gratitude to the following: US Department of Health and Human Services, Office of Adolescent Health for providing funding that made this research possible; the project officers Joanne Jensen and Victor Medrano for their leadership and guidance; Lauren Scher and the TPP Evaluation Technical Assistant Team of Mathematica Policy Research for reviewing and providing technical evaluation advice; Program site partners, for providing classroom space and facilitating the recruitment process, including principals and site coordinators from 9 high schools in eastern Massachusetts [Community Academy of Science and Health, Cambridge Rindge and Latin, English High School, Everett High School, Malden High School, Mattahunt Elementary, Newcomers Academy, Randolph High School, Tech Boston Academy, West Roxbury Academy and Urban Science Academy]; and leadership of two community based organizations [Youth And Family Enrichment Services and South Shore Haitians United for Progress]; implementation /evaluation collaborators, including Suze Prudent Ph.D. from Haitian Mental Health Network who helped with designing the lesson plan for the PTSD awareness session; Ms. Vivien Morris from the Boston Public Health Commission who helped in adapting the nutrition curriculum for the control group; and John Capitman Ph.D. who assisted with the early phase of evaluation design and with adapting the student health questionnaire to the HART curriculum; the staff of the Haitian-American Responsible Teen program, namely Dominique Dupont-Dubois MPH, Joel Piton M.Ed., Elise Myrthil, Michelle Cineus, Glorie Vital, Anthony Williams, Pierre Alexis, Jessica Francois, and Smith Guillaume, for their role in facilitating recruitment and implementing the curriculum; and research assistants Christy Benjamin, BS, MPH, Dr. Linda Lubin, Myriam Georges, and Noadya Legrand for their role in collection of survey data as well as class observation data and for conducting data entry. We are also grateful to the students who participated in the program. The Authors. This publication was prepared under Grant Number 5 TP2AH000005 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. TABLE OF CONTENTS Content page I. Introduction A. Introduction and Study Overview 4-5 B. Primary Research Questions 5 II. Program and Comparison Programming A. Description of Program as Intended 6-7 B. Description of Counterfactual Condition 7-8 III. Study Design A. Sample Recruitment 8-9 B. Study Design 9 C. Data Collection 9-11 D. Outcomes for Impact Analyses 11 E. Study Sample 12 F. Baseline Equivalence 13-14 G. Methods 15-17 IV. Study Findings A. Implementation Study Findings 17-18 B. Impact Study Findings 18-19 V. Conclusion 20-21 Appendices 25-36 3 I. INTRODUCTION A. Introduction and Study Overview This report describes the implementation and impact of a teen pregnancy prevention program funded by OPHS/OAHTPP PREP Tier2-201. The initiative was created in response to an increase in HIV, other sexually transmitted infections, and pregnancy among teens in the United States, with widening racial and ethnic disparities in the period from 2005-2009. 1 Cultural and linguistic factors make addressing teen pregnancy and sexually transmitted diseases particularly challenging for immigrant youth. The large number of Haitians in the greater Boston area presents a clear need for a prevention intervention specifically geared to Haitian youth. Based on the American Community Survey brief there are projected to be about 830,000 Haitians nationwide, with five states (Florida, New York, Massachusetts, New Jersey and Connecticut) having the highest percentage. One hundred thousand Haitians live in Massachusetts, 2 predominantly in Eastern Massachusetts, with the majority in the greater Boston area. Most project participants resided in the predominantly minority neighborhoods of Mattapan, Hyde Park, Dorchester and Roxbury, which in 2010 had a combined population of 215,926. About 30% of residents live below the federal poverty level; 25% have less than a high school diploma; and 25% are non-English speaking immigrants from countries such as Haiti (10%) and the Dominican Republic (9%), where formal education is unevenly available at the primary and secondary levels. As a result of the prior educational deficits and other social and economic challenges for new immigrant families, the youth are at increased risk for dropping out or failing before completing high school. In the 4 targeted neighborhoods, in fact, nearly half of black youth fail to complete high school. 3 Comprehensive education interventions driven by socio-cognitive theory have gained wide appeal as a means to improving health outcomes. However, further research is required to 4 demonstrate that they can be efficacious in varying settings and among minority populations, 4,5 even in cases in which 2 programs have followed the same approach, the impacts on youth outcomes have differed. 6 In the absence of rigorous and consistent evidence about what is most effective in specific contexts and populations, it is important to test a range of program approaches.6 We implemented and evaluated the “Haitian-American Responsible Teen” (HART) program in 9 public high schools and 2 community settings in the greater Boston area (Suffolk and Middlesex counties). We culturally adapted “Becoming a Responsible Teen” (BART), an 8- session curriculum originally designed for African-American youth. 7 B. Primary Research Question(s) What is the impact of the HART curriculum, relative to a control nutrition/fitness curriculum, on the proportion of youth who report at the 6-month follow-up survey: 1. Ever having had sex 2. Having had sexual intercourse in the prior 3 months 3. Having had sexual intercourse without using a condom in the prior 3 months 4. Having had sexual intercourse without using any effective birth-control methods in the prior 3 months II. PROGRAM AND COMPARISON PROGRAMMING HART is a cultural adaptation of BART, an evidenced-based 8-week sexual education curriculum designed for African-American teens. 8 The BART curriculum combines concepts of the “Information, Motivation, Behavioral Skills” theory7 and Bandura’s social cognitive theory. 9 The BART curriculum was originally tested in 3 settings—a community health center, 10 a residential drug rehabilitation center, 11,12 and a juvenile reformatory. 13,14 In the community health center setting, the curriculum was effective in delaying sexual activity, reducing the frequency of sexual activity, and increasing condom use. Among residential drug program 5 participants, the program was associated with reduced sex and number of partners, and increased condom use. 15 No impact was observed among youth in the juvenile reformatory setting. In the published literature, only 1 replication of BART has been evaluated among Haitian teens, and the study focused only on impacts of mediating factors, such as knowledge, intention to use condoms in the future, safer-sex self-efficacy, and attitudes about condom use and the ability to use condoms. 16 Our study is the first to assess impacts of an adaptation of the sexuality education curriculum on behavioral outcomes among Haitian teens. A. Description of Program as Intended The HART curriculum consists of 10 lessons: the 8 lessons from the BART curriculum, and 2 additional lessons as part of the culturally-specific adaptation. The additional lessons include one on anatomy and one on Post-Traumatic Stress Disorder (PTSD). 17 The lesson on PTSD was designed to increase awareness about the problem and its impact on decision making for this population with a high prevalence of traumatic experience. Further, HART started in September 2010, a few months after Haiti was struck by an earthquake that resulted in thousands of deaths, children being orphaned, and chaotic displacements, including many youth who were transported to the US as new immigrants. All materials were translated into Haitian Creole and contextualized for Haitian teens. Role-play scenarios were used to stimulate discussion and dispel cultural myths and commonly held but inaccurate beliefs. The HART curriculum was delivered in 9 high schools with a large concentration of Haitian students, largely during after-school hours. The program was offered as part of a summer program that occurred both in school settings and at 2 community-based organizations serving Haitian families. The program was held once a week for 10 weeks during fall and twice a week for 5 weeks during spring and summer for 1.5-2 hours per session, over a total of 5 semesters and 3 summers. Three trained facilitators led groups ranging in size from 5 to 6 15 participants. Facilitators received technical support to resolve day-to-day implementation challenges reported in facilitator logs. Three senior facilitators participated in weekly staff meetings with the implementation and evaluation teams, where implementation issues and possible solutions were discussed. The HART program follows the structure and adapts the curriculum of BART’s core topic components. The program includes content focused on improving knowledge, attitude, and communication and negotiation skills related to HIV, drug and alcohol use, avoiding risky sexual situations, and using condoms correctly and consistently. The program discusses perceptions of risk, social and peer norms, values, and intentions. It also covers communication with parents and other adults as well as using the skills gained through the program to influence family members, friends, and peers. Additional content focuses on dispelling myths and improving knowledge of reproductive anatomy and physiology, as well as raising awareness about trauma and coping strategies. B. Description of Counterfactual Condition The control group received a nutrition curriculum adapted from the “FANtastic Kids” curriculum, 18 Boston Medical Center’s after-school fitness and nutrition curriculum, and “CANfit”, a California Adolescent Nutrition and Fitness program. 19 We offered a nutrition/fitness curriculum to maintain the engagement of the control group and to meet our responsibility to the participating schools and communities to offer a health promotion alternative that was relevant but distinct from the treatment curriculum. Nutrition was a salient topic to address in light of the high prevalence of obesity among Haitian immigrant children, which increases with each year of residence in the U.S. 20 The program aimed at building capacity for healthy eating habits and increased physical activity through peer leadership development. It has both nutrition and physical activity components. Like the treatment 7 curriculum, it is offered as a 10-lesson, group-based education intervention, delivered on a similar schedule, and also included the same PTSD session offered to youth in the treatment condition. III. RESEARCH DESIGN A. Sample Recruitment We targeted schools that were located in communities with a high concentration of Haitians and willing to sign a Memorandum of Understanding. The study took place in 9 schools, as part of an after-school program, and 2 community based organizations over 5 school semesters and 3 summers. Schools serving larger Haitian populations participated in more semesters than schools with fewer Haitian students. In the first semester 3 schools participated, and subsequently each semester between 3 and 8 schools participated in the study. Summer programs were incrementally added annually, one in year 2, two in year 3 and four in year 4, and occurred at both school and community based organization sites. In Appendix A, Table A.1 documents the number of schools and community-based organizations involved in each of the time frames. Students were recruited each semester and during the summer. To be eligible, youth had to be of Haitian descent, 13 to 19 years of age, attending 9th or 10th grades in the greater Boston area and surrounding counties, have signed assent and consent forms, and not currently participating in a sex education program. We added 11th grade for one school that served new arrivals from Haiti who typically had low literacy for grade level. During the fall and spring semesters of each program year, informed consent for parents and informed assent for youth were obtained through established systems for after-school enrollment. Participating schools provided space for the program, helped with parents’ recruitment, and facilitated the informed consent process. Both the treatment and control groups received the same parent consent and 8 student assent forms, available in both Haitian Creole and English. The consent process during summer programming and during the school year was the same, except that during summer eligible participants were identified among youth that were seeking summer opportunities. B. Study Design The study was a randomized controlled trial. Individuals were randomly assigned to treatment and control conditions after eligible students and their parents consented to participate in the study and students completed a baseline survey. The individual student was the unit of randomization. A stratified random assignment was done by gender and by age-groups (13 to 15 and 16 to 19 years old), using Research Randomizer, a web-based software program 21 to reduce potential baseline imbalance on key characteristics that may be related to outcomes. We randomly assigned 637 students who had met eligibility criteria to treatment (326) and control (311). One hundred siblings or related students who signed up concurrently were randomized as a single unit. In contrast, 38 students whose sibling or related participant who enrolled in the study a semester or a year later were randomly assigned separately to the treatment or control group. Interventions began within 1 week of randomization. C. Data Collection 1. Impact evaluation We used pencil and paper self-administered 14-page survey instruments to collect pre- test and outcome data from students. Outcome data were collected at 3 points in time—soon after completing the curriculum (immediate post-test) and at 6 months and 12 months from the time the program ended. Pretest data were collected prior to randomization (about 1 week before the beginning of the educational intervention). Table B.1 in Appendix B shows the timing of data collection efforts. Trained bilingual/bicultural research assistants provided the surveys to students and supervised the survey process. Participants were informed about incentives: meals 9 at program sessions and a stipend of $150, which was dispensed in small amounts ($10 for the pre-test, $5 for each session attended for a total of 10 sessions, and $20 for the post-test, $30 at 6 months and $40 at 12 months follow-up). Participants were assured of the confidentiality of their responses. Survey instruments were available in both Haitian Creole and English and students were given the opportunity to choose their preferred survey language. The timing of survey collection was the same for treatment and control group youth. The surveys in both the school- based and community-based settings were administered in classrooms; it took students 15-60 minutes to complete. 2. Implementation evaluation Facilitators and independent observers were the main sources of implementation data that assessed adherence and dosage as well as quality (Appendix B Table B.2.). For adherence and dosage, facilitators collected data on: (1) the number and frequency of offered sessions; (2) what and how much was received; (3) the content that was delivered to youth; and (4) who delivered the materials to youth. These data were collected every time class was offered. Similar data were collected for the control group. To measure quality of implementation, data were collected by facilitators and 2 independent observers. Facilitators completed a self-efficacy questionnaire in which they shared their experience of facilitating the class. The independent observers, who were fluent in both English and Haitian Creole, hired and trained by the program evaluator, conducted direct observation of classes. A purposeful sample of 10% of classes was collected at various points in time throughout the intervention period. Similar data were collected for the control group on a smaller sample. We used a validated instrument which was provided by the U.S. Department of Health and Human Services, Office of Adolescent Health and the Administration on Children, Youth and Families. 22 This instrument contains 7 questions that assess different aspects of delivery 10 quality and facilitator competence, as well as the overall quality of the session implementation. Delivery quality was measured on 5-point ordinal scale (1=poor, 3=average, 5=excellent). The assessment form also had a key that provided examples of situations that would satisfy each of these scales. The independent observer also submitted class completion logs, which served as an external validation tool for facilitator class completion logs. D. Outcomes for Impact Analyses The evaluation measured program impacts at 6 months post completion of the intervention. These are described in Table III.1. Table III.1. Behavioral outcomes used for primary impact analyses research questions Timing of measure Outcome name Description of outcome relative to program 1. Ever had sex Assesses delaying sex initiation and is 6 months after (delaying sex operationalized as proportion who reported ever program ends initiation) having had sexual intercourse among all participants, and measured on a dichotomous yes/no question (coded 1=Yes, 0=No) 2. Had sex in last 3 Assesses recent sexual intercourse and is 6 months after months (recent operationalized as proportion among all program ends sexual intercourse) participants who reported having had sexual intercourse in the prior 3 months measured on a dichotomous yes/no (coded 1=Yes, 0=No) 3. Recent sex Proportion who reported sexual activity without 6 months after without condom using condoms in the prior 3 months, measured on program ends use a dichotomous yes/no (coded 1=Yes, 0=No). Individuals who had not had sex in the past 3 months were coded as “no” for this variable. 4. Recent sex Proportion who reported sexual activity in the prior 6 months after without effective 3 months without using any birth control method, program ends birth control use measured on a dichotomous yes/no question (coded 1=Yes, 0=No)[a] Individuals who had not had sex in the past 3 months were coded as “no” for this variable. [a] If participants reported using any of the 7 named birth control methods (a) condoms, (b) birth control pills, (c) the shot (Depo-Provera), (d) the patch, (e) the ring (Nuva Ring), (f) IUD (Mirena or Paragard), (g) implant (Implanon) in the prior 3 months, they were coded as 0=no; otherwise they were coded as 1=yes. 11 E. Study Sample Of the 794 youth who expressed interest in participating in the program, 637 met the eligibility criteria, provided consent and assent, and completed a baseline survey. All 637 were then randomly assigned to the treatment group (326) or the control group (311); 552 youth (275 treatment, 277 control) participated in the 6-month follow-up survey, the basis for answering the study’s primary research questions. We adjusted for non-independence during analysis by including only one computer-identified sibling from sets of sibling that had been randomly assigned as a single unit. Appendix C details the flow of the sample from random assignment through follow-up. Handling of missing data for analytic sample We cross-checked all missing data with hard copies to rule out entry errors. Less than 4% of pretest data were missing and less than 2% of outcome data were missing. We used independent sample t-test as well as chi-square to compare the treatment group and control on rate of missing for both treatment and control and to compare demographics and behavioral characteristics of youth that had completed the 6-month survey to those that had not. We also compared the sample with missing outcome or missing baseline data versus non-missing data by demographic characteristics. There were no statistically significant differences between treatment and control groups in the rate or pattern of missing data at pre-test or at outcome assessments. Since the amount of missing data was very low, our benchmark analysis used casewise deletion; only participants with non-missing pre-test and outcome data were included in the analytic sample. As a sensitivity analysis (described later), we performed dummy variable adjustment on missing pretest data. 12 F. Baseline Equivalence To assess baseline equivalence, we first conducted bivariate analyses to determine similarity in the distribution of key demographic and behavioral variables at baseline. For continuous variables, we used independent sample t-test with p-value at alpha = 0.05 (two- tailed). For dichotomous or categorical scale variables, we assessed equivalence using Pearson chi-square and p-value at alpha = 0.05. Next, we conducted regression analyses to determine if any of the baseline variables were predicted by group membership (i.e., treatment versus control). This was accomplished by regressing key baseline covariates on a treatment indicator and stratification indicators (gender and age group). We did not find statistically significant differences between treatment and control in age, gender, language spoken at home, or being U.S. born. However, the treatment group had a higher proportion of youth who had lived in the U.S. for less than 4 years, by 9.5 percentage points. (See Table III.2a.) 13 Table III.2.a Summary statistics of key baseline measures for Haitian-American youth 13 to 19 years of age who completed 6 months follow-up survey[a] and had no missing data Baseline measure N Intervention N Comparison Intervention Intervention mean mean or % versus versus (standard (standard comparison comparison deviation) deviation) mean p-value of or % or % difference difference Age 256 15.3(1.4) 254 15.4 (1.4) -0.1 .71 Age Group 13 to 15, percent 256 60.6 254 63.4 -2.8 .51 Female, percent 256 53.9 254 52.0 1.9 .66 Grade 256 9.4(0.6) 254 9.4(0.6) -0.03 .49 Grade group 9th, percent 256 68.4 254 64.2 4.2 .32 Race, percent . . . . . . Black/African American 228 89.1 235 92.5 -3.5 . Asian 1 0.39 1 0.39 0.00 . American Indian or Alaska Native 3 1.17 0 0.00 1.17 .43 Native Hawaiian or other Pacific Islander 1 0.39 2 0.79 -0.40 . More than one race 4 1.56 2 0.79 0.78 . Unknown 19 7.4 14 5.5 1.9 . Hispanic, percent 256 4.3 254 5.1 -0.80 .66 U.S. Born, percent 254 23.2 252 25.8 -2.6 .50 Language Spoken Home English, percent 256 65.6 254 69.3 -3.7 .38 Length of time in the US of less than 4 years, percent 256 52.4 254 42.9 9.5 .03 Ever Had Sex, percent 252 29.4 249 32.1 -2.8 .50 Sex Last 3 months, percent 248 14.1 246 13.01 1.1 .72 Sex Without Condom Last 3 months, percent 248 3.63 246 5.69 -2.06 .28 Sex Last 3 Months Without Birth Control, percent 254 9.84 247 10.93 -1.09 .69 Analyses used sample of all those that completed a follow-up survey at 6 months follow-up. [a]The sample includes only one computer-identified primary sibling of each set of group-randomized siblings, and all cases with missing data are excluded. Due to rounding, some categories may not add to 100%. 14 G. Methods 1. Impact evaluation Our analytic approach focused on individuals who had completed the 6-month survey, including all youth who were randomized, regardless of class participation status. We adjusted for non-independence of data from siblings who were randomly assigned as a single unit by including only one computer-identified sibling from sets of sibling that had been randomly assigned as a single unit. We did this by first assigning a family ID to all youth participants based on information collected at enrollment, and related youth were assigned the same family ID. We then used a computer to identify primary (randomly selected) and secondary siblings in a family. We retained the primary siblings and excluded the rest in the impact analyses. We used multiple regression models for all outcomes to estimate treatment effects. According to Bloom et al, 23 there are 2 main approaches for improving the precision of randomized experiments: one is to include baseline covariates and the other is to incorporate strata as fixed effects. We used both of these features in our impact analysis. Our benchmark model used casewise deletion and included as covariates-- a pre-test value of the outcome, demographic variables that were used to conduct random assignment (gender, age group), and length of time the participant had lived in the United States of less than 4 years. Length time in the United States of over 5 years (a measure of assimilation) has been associated with increasing likelihood of ever having had sex 24 particularly among first generation immigrant adolescents from poor families of color who typically settle in under- resourced geographical settings. The threshold for this sample was 4 years and over. We also adjusted for multiple comparisons using a Bonferroni correction. We accounted for 4 primary outcomes, and set our new alpha level at 0.0125. All analyses were conducted using SPSS Version 20. 15 Sensitivity analyses We conducted sensitivity analyses to determine the impact of missing data and of covariate adjustment on the precision of the impact estimate. As a sensitivity analysis for covariate adjustment, we specified a model that included the baseline value of the outcome as the only covariate. We compared this model against our benchmark model and observed substantively similar results (see Table E.1, Models A and B). As a sensitivity analysis for missing baseline responses, we imputed a constant value for all missing data and created a dummy variable where “1 = missing baseline data” and “0 = not missing baseline data” and included this dummy variable in the model as a covariate, along with the baseline measure. We then compared this model with a dummy variable with the benchmark approach and observed substantively similar results (see Table E.1, Models A and C). According to simulation studies for missing pretest scores, the dummy variable adjustment method performed similarly to the more sophisticated methods such as multiple imputation or maximum likelihoods method, especially if missing data are rare. 25,26,27 Conversely, when posttest data are missing, case deletion according to simulation studies performed by Puma and colleagues worked as well as, or better than, all of the alternative methods across all of the missing data scenarios.25 2. Implementation evaluation Facilitator reported and observer assessment data, each pooled over a 3-year period, were used to assess adherence, dosage, and class quality, with the unit of analysis being the class session. Adherence was measured as the percentage of session elements delivered relative to the recommendations. In addition, the degree of agreement between observer data and facilitator data was assessed. Attendance was measured as the percent of classes attended as well as the percent of classes where 75% or more of youth attended. Seven quality indicators rated on 5 point scales (1 = poor, 3 = average, 5 = excellent) on observations were used to assess class 16 quality. These indicators included: (1) clarity of facilitator’s explanations of activities; (2) extent to which the facilitator tracked time during the session and activities; (3) extent to which presentations were rushed or hurried; (4) participants’ understanding of the material; (5) level of participation by group members in discussions and activities; (6) facilitator’s competence, which included: (a) knowledge of the program, (b) level of enthusiasm, (c) poise and confidence, (d) rapport and communication with participants, and (e) effectiveness of addressing questions/concerns; and (7) the overall quality of the program session. See Appendix D for a description of the implementation evaluation methods we employed. IV. STUDY FINDINGS A. Implementation Study Findings In this section we present implementation element-specific findings. We used data that we had submitted from spring 2012 to summer 2015 which were pooled for all program sites and present findings under three subheadings: (1) component adherence, (2) dosage, and (3) class quality. Component adherence. We found that 97.3% of session activities that were planned had been completed within the allocated time. The activity completion mean adherence rates based on classroom observations ranged from 80 to over 100 (the observations that had over 100 were few, and reflected activities that were carried over from a prior session). The agreement between facilitator-reported and observer-reported completion was more than 90%. Dosage. Median attendance was 70%, and 59% of the class maintained more than 75% attendance. Based on the reports from facilitators and the tracking tool from the Program Coordinator, reasons for non-attendance included moving out of state, moving out of district, conflict of schedule with other youth commitments, and lack of interest. 17 Class Quality. We found the overall observed quality to be 4 out of 5 across all 31 class observations. Six of 31 sessions were rated as average, including understanding HIV/AIDS, personalizing risks, spreading the word, and PTSD awareness. None of the observed classes had an overall quality rating (across the 7 quality domains) that was below average. We also assessed external factors that may have affected implementation and unplanned adaptations of the intervention or study design. We found that the consolidation of Boston Public Schools during the time we were recruiting school participation led to a delay in school officials’ buy in to the project. The only major adaptation to our study implementation was the addition of a summer camp each year to assure adequate numbers. B. Impact Study Findings Primary Outcomes We did not find statistically significant effects on any of the 4 behavioral outcomes (Table IV). For ever having had sex, 19.8% of youth in the treatment group reported that they had ever had sex versus 23.5% of youth in the control group, a non-statistically significant mean difference of -3.7%, p=.31, [95% CI:-10.7%, 3.4%]. For having had sex in the last 3 months, 22.2% of youth reported that they had sex in the last 3 months versus 19.7 % of youth in the control group, a non-statistically significant mean difference of 2.5%, p=.45 [95% CI:-4.0%, 8.9%]. For having had sex in the last three months without using condoms, 2.7% of youth in the treatment group reported having had sex in the last 3 months without using condoms versus 5.4% of youth in control group, a non-statistically significant mean difference of -2.7%, p=.17 [95% CI:-6.5%, 1.2%]. For having had sex in the last three months without using any effective birth control, 12.6% of youth in the treatment group reported having had sex in the last 3 months without using any effective birth control method compared to 8.9% youth in the control group, a non-statistically significant mean difference of 3.8%, p=.17, [95% CI:-1.6%, 9.2%]. 18 Table IV. Adjusted post-intervention estimated effects using data from the six-month survey to address the primary research questions Outcome measure Intervention Comparison mean p-Value 95%CI 95%CI (%) (%) difference lower Upper Ever had Sex 19.8 23.5 -3.7% .31 -10.7% 3.4% N=496 Sex in Last 3 months 22.2 19.7 2.5% .45 -4.0% 8.9% N= N=488 Sex Without Condom 2.7 5.4 -2.7% .17 -6.5% 1.2% N=488 Sex Without Effective Birth Control 12.6 8.9 3.8% .17 -1.6% 9.2% N=493 Sample Size . . . . . . Source: Haitian-American Responsible Teen impact data collected from fall 2012 to spring 2015, 6 months after the program ended. Notes: Mean differences are linear regression adjusted estimates of treatment effect controlled for gender, age-group, 13 to 15 versus 16 to 19 and length of time participant had lived in the US. 19 V. CONCLUSION We implemented the HART curriculum, which is a culturally adapted BART plus reproductive anatomy/physiology and PTSD awareness program designed for use by Haitian teens, and a counterfactual nutrition/fitness plus PTSD awareness which were concurrently offered. Programing was offered once a week for 10 weeks in fall and twice a week for 5 weeks in spring and summer. The average attendance rate was 70%, with 60% of class sessions serving at least 75% of participants. The program was implemented with fidelity to the model; 90% of class activities were completed. We found no differences between treatment and control groups in the full sample on self-reported sexual behaviors, including ever having had intercourse, recent sexual intercourse, condom non-use, and birth control non-use. Our study had the following limitations. First, our sample was purposeful rather than randomly selected from a target population; we targeted 9th and 10th grades and we had a hard-to- reach sample. Haitian Americans are not listed as such in the schools, and we depended on Site Coordinators to identify students who were Haitian. We were limited to schools that were willing to participate. However, to our knowledge random samples in clinical trials are rare. Another limitation of our 6-month follow-up results include the fact that 6 months may not provide enough time to observe behavior differentials between the treatment and control groups. Despite these limitations, our study has a number of strengths. This study is an individual-level randomized controlled trial and represents one of the largest trials of a BART- based curriculum to date, and only the second carried out among Haitian adolescents. Our final retained sample is more gender-balanced (54% female) compared to 72% in the original BART trial and 70% in the first reported Haitian-American BART-based trial.10 Further, our retention rates were higher than that of previous studies (84% in the treatment and 89% in the control 20 group) and our overall attrition rate of 13% and differential attrition rate of 5% are within the range of attrition rates considered acceptable for evidence standard. In conclusion, our randomized controlled trial of the Haitian-American Responsible Teen (HART) program, adapted from “Becoming a Responsible Teen” (BART), did not show program effects on the overall sample at 6 months post-intervention. The HART intervention is distinguished from BART by its focus on pregnancy prevention as a long-term goal and in having two additional lessons, one in reproductive anatomy and another on PTSD awareness. The original BART study (N=246) which had been tested and found effective among African- American youth in a medium-sized southern city10 found a reduction in unprotected intercourse, an increase in condom-protected intercourse, and an increase in behavioral skills related to safer sex among youth who received skill-based training when compared to their counterparts who received only information at one year. However, males were less likely than females in the study to sustain safer sexual behaviors, particularly condom use, although the sample was predominantly female (72%). In our study, the intervention was compared to an alternative study curriculum focused on nutrition information. Our work highlights the need to test interventions in multiple sites and multiple contexts, and to assess results over longer-term periods of time after the intervention, so that the evidence base is strong and generalizable across diverse populations. 21 1 Funding Opportunity Announcement (FOA) number: OPHS/OAHTPP PREP Tier2-2010. Teenage Pregnancy Prevention: Research and Demonstration Programs (Tier 2) and Personal Responsibility Education Program. Office of Adolescent Health Office of Public Health and Science and Administration on Children, Youth and Families Administration for Children and Families U.S. Department of Health and Human Services, 2010, available at http://www.hhs.gov/ash/oah/news/assets/funding_announcment_research.pdf 2 Buchanan AB, Nora G. Albert NG and Beaulieu D. The Population With Haitian Ancestry in the United States: 2009 American Community Survey Briefs, Issued October 2010, US Census Bureau 3 Office of Data and Accountability, Boston Public Schools 4-Year Graduation Rate By Cohort Group, available at http://www.bostonpublicschools.org/cms/lib07/MA01906464/Centricity/Domain/238/Cohort%2 02015%204-Y%20Graduation%20Rate%20Report.pdf 4 Card JJ. Teen pregnancy prevention: do any programs work? Ann. Rev Public Health 1999;20:257–285 5 Fisher, Jeffrey D. and Fisher, William A., "Theoretical Approaches to Individual-Level Change in HIV Risk Behavior" (2000). CHIP Documents. Paper 4. http://digitalcommons.uconn.edu/chip_docs/4 6 Goesling B, Colman S, Trenholm C (Mathematica Policy Research); Terzian M, Moore K (Child Trends). This report is available on the Internet at:http://aspe.hhs.gov/hsp/13/Reduce- TeenPregnancy/rpt_tppevidence.cfm 7 Fisher, Jeffrey D. and Fisher, William A., "Theoretical Approaches to Individual-Level Change in HIV Risk Behavior" (2000). CHIP Documents. Paper 4. http://digitalcommons.uconn.edu/chip_docs/4 8 St. Lawrence JS, Brasfield TL, Jefferson KW, et al. Cognitive-behavioral intervention to reduce African-American adolescents’ risk for HIV infection. J Consult Clin Psychol 1995[a]; 63:221-237 9 Bandura, A. (1989). Social cognitive theory. In R. Vasta (Ed.), Annals of child development. Vol. 6. Six theories of child development (pp. 1-60). Greenwich, CT: JAI Press 10 St. Lawrence JS, Brasfield TL, Jefferson KW, et al. Cognitive-behavioral intervention to reduce African-American adolescents’ risk for HIV infection. J Consult Clin Psychol 1995[a]; 63:221-237 11 St Lawrence JS, Jefferson KW, Alleyne E, Brasfield TL. (1995 Comparison of education versus behavioral skills training interventions in lowering sexual HIV-risk behavior of substance-dependent adolescents. J Consult Clin Psychol 1995[b]; 63:154-157. 22 12 St Lawrence JS, Crosby RA, Brasfield TL, O'Bannon RE 3rd. Reducing STD and HIV risk behavior of substance-dependent adolescents: a randomized controlled trial. J Consult Clin Psychol 2002 ;70 :1010-1021 13 St Lawrence J, Crosby R, Belcher L, et al. Sexual risk reduction and anger management interventions for incarcerated male adolescents: a randomized controlled trial of 2 interventions. J Sex Educ Ther. 1999; 24:9-17. 14 Robertson AA, St Lawrence JS, Morse DT, et al. Comparison of health education and STD risk reduction interventions for incarcerated adolescent females. Health Educ Behav. 2011; 38:241-250. 15 Kirby D. (2010) Interventions to Reduce Sexual risk: Curriculum and Group-Based Interventions for Youth. ETR Associates 16 Malow RM, Stein JA, McMahon RC, et al. Effects of a culturally adapted HIV prevention intervention in Haitian youth. J Assoc. Nurses AIDS Care. 2009; 20(2):110-121 17 Fawzi MCS, Betancourt TS, Marcelin L, Klopner M, Munir K, Muriel AC, Oswald C, Mukherjee JS. Depression and post-traumatic stress disorder among Haitian immigrant students: implications for access to mental health services and educational programming. BMC Public Health 2009; 482-493. Available at www.biomedcentral.com/1471-2458/9/482. 18 Rollins JV, Cook J, Thayer JC, Morris TV. FANtastic Kids: A Peer-Led Community-Based Fitness and Nutrition Intervention to Address Pediatric Overweight. J Am Diet Assoc. 2010; 110(9):A90.doi:10.1016/j.jada.2010.06.333. 19 Hinkle JA. Promoting Healthy Eating Among Youth: Engaging Communities Through Multi- Level Strategies California Adolescent Nutrition And Fitness Program .American Dietetic Association Food & Nutrition Conference & Expo October 4, 2004, Available at http://www.dannon-institute.org/pdf/promotinghealth1.pdf. Accessed October 5,2015 20 Strickman-Stein N, Gervais M-D, Ludwig DA, Messiah SE, Lipshultz SE, Miller TL. Body mass index as a function of length of United States residency among Haitian immigrant children. Ethnicity & Disease 2010; 20:22-28 21 Urbaniak GC, Plous S. (2011). Research Randomizer (Version 3.0) [Computer software]. Retrieved on April 22, 2011, from http://www.randomizer.org/ 22 OAH & ACFY, 2011, Program Observation Form for TPP Grantees. Available at http://www.hhs.gov/ash/oah/oah- initiatives/webinars/Program_Observation_Form_for_TPP_Grantees-July_2011.pdf 23 Bloom et al 2007,Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions” Educ Eval Policy An March 2007, Vol. 29, No. 1, pp. 30– 59 DOI: 10.3102/0162373707299550 23 24 National Research Council (US) and Institute of Medicine (US) Committee on the Health and Adjustment of Immigrant Children and Families; Hernandez DJ, editor. Children of Immigrants: Health, Adjustment, and Public Assistance. Washington (DC): National Academies Press (US); 1999. Chapter 6, The Health Status And Risk Behaviors Of Adolescents In Immigrant Families. Available from: http://www.ncbi.nlm.nih.gov/books/NBK224437/ 25 Groenwold RHH, Moons KGM, Vandenbroucke JP. Randomized trials with missing outcome data: how to analyze and what to report. Can Med Assoc J. 2014; 186; 186(15):1153-1157. doi:10.1503/cmaj.131353. 26 Puma, Michael J., Robert B. Olsen, Stephen H. Bell, and Cristofer Price (2009). What to Do When Data Are Missing in Group Randomized Controlled Trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education Available at: http://ies.ed.gov/ncee/pdf/20090049.pdf. Retrieved on: 27 Deke J and Puma M Coping with Missing Data in Randomized Controlled Trials Evaluation Technical Assistance Brief #3 Publisher: Washington, DC: U.S. Department of Health and Human Services, Office of Adolescent Health Available at: https://www.mathematica-mpr.com//our-publications-and- findings/publications/coping-with-missing-data-in-randomized-controlled-trials. Retrieve: 5/13/2016 24 APPENDIX A: SAMPLE RECRUITMENT Table A.1. Number of sample sources and implementation sites Data collection effort Cohort 1 Cohort 2 Cohort 3 Cohort 4 Cohort 5 Cohort 6 Cohort 7 Cohort 8 Spring Summer Fall Spring Summer Fall Spring Summer 2012 2012 2012 2013 2013 2013 2014 2014 Number of source 3 23[*] 6 6 9[*] 8 4 38[*] schools Number of 3 1 6 6 2 8 4 4 implementation sites [*]Include schools that did not participate during fall and spring and those that had no formal collaboration agreement with the HART project. Students from these source schools attended program at one of the participating implementing sites. 25 APPENDIX B: DATA COLLECTION EFFORTS Table B.1. Data collection efforts used in the impact analysis of HART and timing Data collection Cohort 1 Cohort 2 Cohort 3 Cohort 4 Cohort 5 Cohort 6 Cohort 7 Cohort 8 effort Spring 2012 Summer Fall 2012 Spring Summer Fall 2013 Spring Summer 2012 2013 2013 2014 2014 Baseline survey 4/5/12- 7/11/12- 10/16/12- 2/1/13- 7/15/13- 9/19/13- 3/6/14- 7/9/14- 5/2/2012 7/16/12 1/15/13 7/19/13 2/11/14 5/20/14 7/26/14 5/9/13 Start date of 4/9/12 7/12/12 10/23/12 3/1/13 7/17/13 10/21/13 3/13/14 7/13/14 programming Immediate post- 5/24/12- 8/14/12- 1/22/13- 5/20/13- 8/21/13- 1/27/14- 5/21/14- 8/14/14- test 6/14/12 8/15/12 4/11/13 8/29/13 5/6/14 6/27/14 10/26/14 7/23/13 6 months follow- 12/11/12- 2/19/13- 7/30/13- 11/13/13- 2/28/14- 7/30/14- 11/12/14- 2/12/15- up 12/17/12 3/2/13 1/13/14 4/2/14 11/6/14 2/19/15 4/30/15 2/26/14 12 months 6/6/13- 8/15/13- 2/3/14- 5/20/14- 8/20/14- 2/4/15- 5/20/15- 8/6/15- follow-up 10/22/13 8/19/13 5/29/14 10/22/14 5/5/15 7/24/15 9/13/15 6/23/14 26 Table B. 2. Data used to address implementation research questions Implementation Element Types of data used to Frequency/sampling of data Party responsible for data assess whether the collection collection element of the intervention was implemented as intended Adherence (1) How many and how Facilitator Fidelity Logs often were sessions (form developed by offered HART):  Number of sessions  In the fall, once a week; in the  Two facilitators at each program completed per intervention spring, twice a week; in the site were responsible. period summer, every session.  Duration of session in minutes  Type and number of activities completed per session Program Developer’s Manual:  Recommended number of  At every intervention period, Program coordinator was sessions per intervention recommended number of responsible for documenting session period sessions per intervention schedule data. period, type and number of  Recommended type and activities per session, as well Program coordinator was number of activities per as session duration were responsible for documenting session provided to facilitators. recommended type and number of activities per session as well as  Duration of session session duration. 27 Implementation Element Types of data used to Frequency/sampling of data Party responsible for data assess whether the collection collection element of the intervention was implemented as intended Adherence Program schedule:  At every intervention period  Program coordinator was  Session name program schedule with session responsible for documenting  Session delivery dates by names was developed. schedule. program site  At every intervention period  Program coordinator was program schedule with detailed responsible. delivery dates and sites was developed. (2) What and how much . . . was received . Number participants .  Two facilitators at each program Class size site were responsible for collecting class level attendance data . HART Session .  Two facilitators at each program Attendance Log site and the Program Coordinator Individual-level data, were responsible for collecting including name, time in individual level student attendance and time out, and aggregate data. number of students. . Enrollment data-base:  Class size at randomization  Independent evaluator and  Class size at was collected once by program program coordinator were randomization site for every intervention responsible for collecting class size period before session started data. 28 Implementation Element Types of data used to Frequency/sampling of data Party responsible for data assess whether the collection collection element of the intervention was implemented as intended Adherence (3) What content was Developer’s manual and  Recommended total number  The program principal investigator delivered to youth: e.g., lesson plans: of topics was collected once (PI) and collaboration partners were total number of topics  Recommended total during pre-program design responsible for collecting data. covered, proportion of number of topic per session phase. material that was ultimately discussed in sessions . Facilitator logs activity  Once a week in fall at every  Two facilitators at each program completion check list session; Twice a week in spring site were responsible for collecting  Topic-linked activities and summer at every session topic-linked activity covered covered per session were collected in form of numeric codes of completion status and in nominal count of number of activities completed 29 Implementation Element Types of data used to Frequency/sampling of data Party responsible for data assess whether the collection collection element of the intervention was implemented as intended Adherence 4) Who delivered material  Implementation schedule  Implementation schedule was  Program co-coordinator and to youth: and Facilitator Logs collected twice (once before evaluator were responsible for provided evidence of who program starts and once after collecting implementation schedule actually delivered the the program has been data. session implemented.  Program Coordinator and  Biannual and annual  Facilitator logs containing Facilitators were responsible for reports provided staff initials were collected at completing the Facilitator Logs. information on number of every session. staff hired, number who  These were marked with the were trained, and number  Information about staff hiring Facilitators’ initials. who participated in and training was collected ongoing training biannually.  Program coordinator was responsible for collecting  Data include # who information on number of staff hired, completed the initial training number who were trained, and and participated in follow-up number who participated in ongoing training. training Class quality Observer Class quality  Direct observation of class  Program Evaluator and Research form quality was conducted and Assistants were trained as observer  Duration of session in documented during every and responsible for collecting direct minutes intervention period for 10% of observation data.  Type and number of session across program sites. activities completed 30 APPENDIX C: STUDY SAMPLE Table C.1: Youth sample sizes by intervention status . . Sample size Response Rates (%) Number of youth Time Period Total Intervention Comparison Total Intervention Comparison Completed a baseline survey Pre-test 637 326 311 100 100 100 Assigned to . 637 326 311 100 100 100 condition Completed a Immediately post- follow-up survey programming 570 290 280 89 89 90 Completed a 6-months post- follow-up survey programming 552 275 277 87 84 89 Contributed a 6-months post- follow-up survey programming 496 249 247 78 76 79 Completed a 12-months post- follow-up survey programming 548 272 276 86 83 89 Notes: † Planned Target sample was 780. There were 1,086 Haitian teens in selected schools, 794 expressed interest, and 637 were eligible to participate, provided consent, and completed a baseline survey. 31 APPENDIX D: IMPLEMENTATION EVALUATION METHODS Table D.1. Methods used to address implementation research questions Implementation Methods Used To Operationalize Each Implementation Element Element Adherence . (1) How many Variable measures and how often 1a) How many sessions offered were sessions  Total number sessions covered (nominal count) offered?  Average number of session per intervention period  Average duration of session in minutes  Average length of intervention period in weeks 1b) How often sessions were offered  Average number of session delivery times per week  Total number sessions covered were calculated as sum of sessions delivered over a 3-year intervention period.  Average number of session per intervention period was calculated as total number of sessions offered over the three-year period when program was offered divided by number of intervention times.  Average session duration was calculated as total number of delivery minutes used over the three-year program implementation period divided by number of sessions delivered.  Average length of intervention period was calculated as total number of weeks when programming was offered divided by number of intervention periods.  Average number of times sessions were delivered was calculated as sum of number of times per week sessions were delivered over a 3-year implementation period divided by total delivery time in weeks over a 3- year period Measure of adherence a. Session number adherence  Percent of sessions delivered over an intervention period relative to recommended number.  Percent of delivery minutes used per session relative to recommended delivery minutes. b. Session frequency adherence  Percent intervention periods with average number of session delivery times of 1 to 2 per week 32 Implementation Methods Used To Operationalize Each Implementation Element Element Adherence . (2) What was Variable measures received? 2a) What was received was calculated at the individual and class level:  Total number of sessions attended per student  Average attendance number of sessions attended  Percent of students who never attended  Percent of sessions was calculated as the total number of sessions attended divided by the total number of sessions offered 2b) Measures of adherence  Percent of sessions with 75 percent or more students in attendance  Percent of youth who never attended class after randomization (3) What Analysis content was We used univariate analysis to summarize number of activities completed per session. delivered to Output measures youth?  Average number of activities completed per topic, per site over intervention period.  We expected 57 activities to be completed in each intervention period: 6 activities per session in sessions 2,4,5 and 10; 5 per session in session 5 and 8, 3 per session in session 5 and 9 and 9 in session 1 Adherence Measure  Average number of activities completed per session/per program site.  Percent of planned activities completed is a measure of adherence Limitation of this measure is that session 10 was not routinely reported in facilitator logs in year 2 and 3. We relied on the few observations that were conducted on this session and year 4 facilitator logs to assess the degree of implementation adherence (4) Who  Implementation schedule and facilitator logs provided evidence of who actually delivered the session delivered  Number of staff hired, number who were trained and number who participated in ongoing training material to youth? 33 Implementation Methods Used To Operationalize Each Implementation Element Element Adherence . Quality of staff- Summary measures participant  Total number of observation over a 3-year period interactions  Average score per session Adherence  Percent of sessions with facilitator rapport and communication with participants scoring 4 to 5 (good to excellent)  Percent of sessions with facilitator effectively addressed questions/concerns scoring 4 to 5 Quality of Summary measures youth  Total number of observation over a 3-year period engagement  Average score per session with program Adherence  Percent of sessions with facilitator rapport and communication with participants scoring 4 to 5 (good to excellent)  Percent of sessions with facilitator effectively addressed questions/concerns scoring 4 to 5 Percent of sessions with student group members actively participating in discussions and activities scoring 4 to 5 (good to excellent Counterfactual Experiences of  The facilitators for the counterfactual group also submitted fidelity and attendance logs. counterfactual  We compared the control and treatment groups on attendance. External events (1) School structural changes affecting (2) Length of time from introduction to buy-in by School administrators implementation (3) School closures (for instance (4) Length of time of IRB approval in the first year school turnover, budget cuts, etc.) 34 APPENDIX E: SENSITIVITY ANALYSES A. Bench Mark B. Covariate Sensitivity C. Missing Data Sensitivity Outcomes Casewise deletion with regression Casewise deletion with regression Casewise deletion of missing outcome, adjustment for pre-test value of the adjustment for only the pre-test value of missing pre-test coded at constant value, outcome variable plus demographics the outcome variable plus missing data indicator variable with regression adjustment for missing data indicator plus pre-test value of the outcome variable and demographics Variables Treatment P- 95.0%CI 95.0%CI Treatment P- 95.0% 95.0%CI Treatment P- 95.0%CI 95.0%CI minus value LB UB minus control value CI. L UB minus value LB UB control difference B control difference difference Ever had -3.68 .306 -10.72 3.37 -3.58 .316 -10.59 3.43 -3.30 .352 -10.27 3.66 sex Sample 496 500 505 . . . . . . . . . N(n1,n2) (249,247) (250,250) (253.252) Sex in last 2.46 .451 -3.96 8.88 2.12 .514 -4.27 8.51 3.01 .351 -3.32 9.34 3 months Sample 488 492 503 . . . . . . . . . N(n1,n2) (244, 244) (246,246) (253,250) Sex without condom -2.69 .171 -6.54 1.17 -2.63 .175 -6.43 1.17 -2.29 .240 -6.11 1.54 use in the last 3 months Sample 488 492 503 . . . . . . . . . N(n1,n2) (244,244) (246,246) (252,251) 35 A. Bench Mark B. Covariate Sensitivity C. Missing Data Sensitivity Outcomes Casewise deletion with regression Casewise deletion with regression Casewise deletion of missing outcome, adjustment for pre-test value of the adjustment for only the pre-test value of missing pre-test coded at constant value, outcome variable plus demographics the outcome variable plus missing data indicator variable with regression adjustment for missing data indicator plus pre-test value of the outcome variable and demographics Variables Treatment P- 95.0%CI 95.0%CI Treatment P- 95.0% 95.0%CI Treatment P- 95.0%CI 95.0%CI minus value LB UB minus control value CI. L UB minus value LB UB control difference B control difference difference Sex without effective 3.25 birth 3.76 .171 -1.63 9.16 4.41 .110 -.010 .098 .238 -2.16 8.67 control use in last 3 months Sample 493 497 501 . . . . . . . . . N(n1,n2) (248,245) (250,247) (250.251) Source: Haitian-American Responsible Teen impact data collected from fall 2012 to spring 2015, 6 months after the program ended. Bench mark has differences in proportions that are linear regression adjusted estimates of treatment effect controlled for pre-test value of the outcome variable plus gender, age-group, 13 to 15 versus.16 to 19 yrs., and length of time participant had lived in the US. The sample retained only one computer- identified sibling of each set of group-randomized siblings and all cases with missing data were excluded. A Bonferroni adjustment for multiple comparisons was used, which resulted in a critical p-value of 0.0125. 36