Findings from the Replication of an Evidence-Based Teen Pregnancy Prevention Program Evaluation of the Teen Outreach Program® in Louisiana Final Impact Report for Louisiana Department of Health and Hospitals Office of Public Health January, 21, 2016 Prepared by William T. Robinson, Ph.D. Randi Kaufman, DrPH Laura Cahill, MA, MPH Louisiana Health Sciences Center at New Orleans School of Public Health 2020 Gravier Street, 3rd Floor New Orleans, LA 70112 Recommended Citation: Robinson, W.T., Kaufman, R. & Cahill, L. (2016). Evaluation of the Teen Outreach Program in Louisiana. New Orleans, LA. Louisiana State University Health Sciences Center at New Orleans, School of Public Health, Initiative for Evaluation and Capacity Building. Acknowledgements: This publication was prepared under Grant Number TP1AH000076-05-00 from the Office of Adolescent Health, U. S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health This study was conducted by a team of independent evaluators contracted from the Louisiana State University Health Sciences Center at New Orleans, School of Public Health’s Initiative for Evaluation and Capacity Building. No authors have any financial or other conflict of interests to report. The authors would like to thank the program staff at Louisiana Department of Health and Hospitals Office of Public Health Bureau of Family Health as well as the contracted staff at the Institute for Women and Ethnic Studies, Family Road of Greater Baton Rouge, Southwest Louisiana Area Health Educators, Southern Louisiana AIDS Council, Volunteers of America Central Louisiana, Volunteers of America North Louisiana, and the Louisiana Center Against Poverty. . EVALUATION OF THE TEEN OUTREACH PROGRAM ® IN LOUISIANA: FINDINGS FROM THE REPLICATION OF AN EVIDENCE-BASED TEEN PREGNANCY PREVENTION PROGRAM I. Introduction Although teen birth rates are declining nationally, Louisiana’s rates consistently rank among the highest in the nation. In 2014, Louisiana had the seventh highest teen birth rate in the nation with 35.8 per 1,000 females ages 15-19 (Centers for Disease Control (CDC), 2014). Rates also vary across the state with some parishes (counties) as high as 62.8 per 1,000 females. Louisiana also ranks poorly in other reproductive health indicators. For example, Louisiana is fifth in the nation for statewide HIV case rates (Louisiana Office of Public Health, 2015). Furthermore, Louisiana has been ranked among the top three states for gonorrhea, chlamydia, and syphilis for many years, and the incidence of STDs is high among adolescents and youth (CDC, Sexually Transmitted Disease Surveillance Report, 2013). These problems illustrate a clear need to provide enhanced services and evidence-based programs designed to reduce rates of teen pregnancy and unsafe sexual behaviors. There currently exists no single standard of pregnancy prevention programming that has been thoroughly vetted and identified as evidence based. This is in contrast to resources available for other sexual risk prevention programs, such as the Centers for Disease Control and Prevention’s Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. Typically the creation of such a compendium for evidence-based practice requires: 1) an initial review of the literature to identify programs that have shown promising results in studies that were conducted with rigor, and 2) a sufficient number of studies to establish the effectiveness of each program. In 2010, the Office of Adolescent Health’s Teen Pregnancy Prevention Program, under the U.S. Department of Health and Human Services, awarded funds to the Louisiana Department of Health and Hospital/Office of Public Health/Bureau of Family Health and approximately 100 other agencies and investigators to provide the additional studies needed to compile the compendium. The Louisiana Office of Public Health along with the contracted evaluator from Louisiana State University Health Sciences Center at New Orleans - School of Public Health, elected to conduct an evaluation of the Wyman Teen Outreach Program (TOP®) using a randomized control trial design. The goal of this evaluation was to replicate the findings of Allen et al. (1997), which demonstrated a significant reduction in problem behaviors, including teen pregnancy, following the TOP® intervention relative to a no-treatment control group within a randomized control trial design. Previous evaluations conducted between 1990 and 2001 demonstrated the effectiveness of TOP® (Allen et al., 1990; Allen et al., 1997; Allen and Philliber, 2001). Further, the Brookings Institution recognized TOP® in its 2007 report to the U.S. Congress, Cost Effective Investments in Children, as one of the best programs in the country, stating that every $1 invested in TOP® produced a $1.29 return on investment (Isaacs, 2007). 3 A. Introduction and study overview This report describes the methods and findings of the evaluation of TOP® as implemented in Louisiana. The evaluation examined the impact of the offer to participate in TOP® clubs on participants’ reports of sex with no effective form of birth control immediately after the end of the intervention and after a 12-month follow-up period, as well as self-reported incidence of pregnancy. Each of these outcomes was compared to a no-treatment control. Description of the study implementation analysis is also included. B. Primary research question The primary research question explored in this study was the impact of the offer to participate in TOP® on participants’ immediate, post-intervention survey response to the question, “In the past 3 months, have you had sexual intercourse without using an effective method of birth control, even once?” It was hypothesized that after participating in TOP® participants would have significantly lower levels of sexual activity without use of an effective method of birth control relative to a no treatment control group after controlling for relevant covariates. C. Secondary research question(s) Should the intervention have a positive impact on youth’s sexual activity, its utility would be minimal if the effect is not long lasting. Therefore, a secondary research question explored the impact of TOP® on whether youth engaged in sex with no effective means of birth control relative to a no-treatment control group one year after TOP® ended. Finally, another secondary research question addressed whether TOP® would have a long-term effect on reduction in pregnancies. This outcome was operationalized as the change in number of pregnancies from baseline to 12-month follow-up. II. Program and comparison programming A. Description of program as intended TOP® is designed to provide several components that together lead to a decrease in teen pregnancy among participants along with other decreases in other undesired behaviors. These components are delivered in club settings. TOP® is a nine-month youth development program designed for youth aged 12-17 to promote healthy behaviors, teach life skills, and help teens to develop a sense of purpose. The program consists of lessons from Wyman’s Changing Scenes™ curriculum, community service learning (CSL), and positive adult mentorship. The TOP® Changing Scenes™ curriculum is separated into four age/stage-appropriate levels and focuses on values, communication and assertiveness, influence, goal-setting, decision-making, adolescent development, relationships, and sexuality, including information on sexually transmitted infections. Youth participate in TOP® clubs, comprised of 10 to 25 youth, which meet weekly for between one and two hours. Trained adult facilitators guide and encourage teen participation during club sessions. Based on the needs of the teens, facilitators can choose to implement any of 140 lessons. Each lesson is designed around a set of core activities including a starting activity, main activities, and wrap-up/reflection. Lessons can be repeated, and no specific lessons are 4 required for programmatic fidelity. Programmatic fidelity does specify that a minimum of 25 sessions take place over a consecutive nine-month period. Teens also complete a minimum of 20 community service learning (CSL) hours with their peer group or individually. The intent of CSL is that youth learn to engage with their community while connecting service with classroom learning experiences, thereby enhancing their ability to create goals and building self-efficacy. Volunteering and community service learning activities have been shown to be associated with healthy outcomes during the teen years (Child Trends, 2014), and teens who volunteer may be less likely to become pregnant and show improved academic performance (Oesterle et al., 2004). TOP® participants are expected to choose their service work and to feel that the work is engaging. Therefore, the CSL process ideally begins with discussion of community needs and areas of teen interest. Subsequently, teens are engaged in the planning and implementation of service, followed by reflection on and celebration of their CSL. Some examples of CSL projects conducted by teens in this study included community cleanup, landscaping projects, peer education, and letter-writing campaigns. Seven community-based organizations (CBOs) were selected to replicate TOP® with fidelity through a request for proposal submission process. Agencies were scored on the following: agency history with serving Louisiana youth; experience reaching and recruiting youth participants; experience implementing evidence-based interventions with fidelity; community partnerships; community service experience; key staff; experience in accurate and timely reporting; and experience with similar projects. The highest ranking application within a specific public health region of the state (see Figure 1) was selected for funding, thus ensuring adequate geographic spread within the areas of need with highest prevalence. Following selection, one agency chose not to participate due to their unwillingness to participate in an individual-level, randomized control trial. This agency was replaced during the third cohort. All TOP® clubs were implemented in community-based settings, including community centers, libraries, churches, city administrative offices, or the agencies’ offices. 5 Figure 1. Location of TOP® Clubs and Louisiana Parish Teen Pregnancy Rates B. Description of counterfactual condition The control group received neither TOP® nor any other programming. No alternative program was provided for the control group. III. Study design A. Sample recruitment A convenience sample of school-age teenagers was recruited into the LA TOP® replication study during each of three annual cohorts. Six agencies participated in the first two cohorts and a seventh agency was added in the third cohort. Each of the six original agencies had a total recruitment goal of 1,000 youth over the three cohorts (300, 350 and 350, respectively).The seventh agency had a one-year recruitment goal of 300. Program staff from the contracted agencies conducted all recruitment, which occurred in a variety of settings, including schools, local outreach programs, and at existing youth programs. The agencies identified and recruited youth through a variety of means including direct outreach, existing contacts within their agency, from partner organizations, by providing information to local schools, holding information fairs at various locations, advertising through print and media, and by participating in other community events. Table A.1 presents timing of cohort recruitment and data collection. 6 All youth participants were enrolled in the study using the enrollment protocols set forth by the Louisiana State University Health Sciences Center’s evaluation team. Those protocols provided guidance to agencies on enrollment processes, consent forms, all surveys, and data collection. Youth eligibility criteria for inclusion in the evaluation were: a) 12 and 17 years of age, b) resident of the recruitment target area, and c) able to participate in activities and complete all study materials in English. Recruitment and enrollment processes occurred in the following order: 1) recruitment, 2) informed consent, 3) baseline survey, and 4) randomization to condition. Consent Process After explaining the study and the intervention, staff obtained written parental consent and youth assent from all study participants. Incentives Participants received a $50 gift card for completing the baseline survey and a $50 gift card for completing the immediate post-intervention survey. Participants who completed the 12- month follow-up survey were entered into a raffle for a prize worth approximately $200; examples of prizes included a $200 gift card or a small tablet computer. The odds of each raffle were approximately 1/50. B. Study design A randomized controlled trial design was used to assess the impact of TOP® on sexual risk behaviors and pregnancy. Following recruitment, consent, and the baseline survey, the evaluation staff created a unique identifier for each youth. Each youth was then randomized individually to either receive the TOP® program or be part of the no-treatment control group; youth had an equal chance of being assigned to each group. The results of randomization were uploaded by evaluation staff to an online data portal viewable by project staff who subsequently were able to place youth into TOP® clubs. All staff were therefore blind to condition until after youth completed enrollment and the baseline survey. Sibling Randomization While the recruitment process was standardized across all agencies, in some cases factors external to the study precluded randomization of all youth. For example, prior to cohort 2, there was no procedure for sibling randomization. Consequently, some parents were unable or unwilling to allow the separation of siblings into different conditions and declined to allow their teens to participate. In response to parental concerns from the first cohort that siblings and other members of the same household be given the same assignment, a sibling randomization protocol was instituted beginning with the Cohort 2. One sibling from the household was randomly selected to be the primary participant and assigned to treatment or control. The secondary sibling(s) was then automatically assigned to the same condition, but only the one primary sibling was included in the analytic sample. C. Data collection Outcome data were collected at three time points: a) baseline, no more than one month prior to the TOP® club’s initiation; b) immediately after intervention (within one month of the TOP® 7 club’s end); and c) 12 month follow-up after the end of the TOP® club. To protect youth’s privacy, they did not write their name on their survey. Instead, surveys were identified by a unique identifier assigned to the youth at enrollment and used for all data provided by the youth. Each identifier corresponded with one youth enrolled in the study. Baseline surveys were administered in paper and pencil format both individually and in larger group administration formats by CBO staff. 1. Impact evaluation In order to reduce potential social desirability bias, evaluation staff conducted all post- intervention and follow-up surveys in large group settings. These surveys were administered at scheduled sites and at time periods different than regular TOP® club meetings; sites included community centers, libraries, and the CBO offices. Data collection periods for TOP® clubs ranged from one day to two weeks. Program staff contacted all TOP® and control youth to inform them of scheduled data collection events. Program staff attempted at least three contacts with youth by phone, email, text, or in person and recorded contacts in a locally developed tracking log. Each survey administration event occurred over a period of approximately two to four hours. Youth could attend any event regardless of club assignment or condition. 2. Implementation evaluation Measures of adherence, dosage and quality were used to assess fidelity to the program model. Evaluators collected the following adherence and dosage data from program records: the number and frequency of sessions offered, the content delivered to youth (activities planned vs. activities completed, attendance at sessions, and CSL hours completed. To assess quality evaluators observed 10% of sessions offered by TOP® clubs (approximately three visits spaced throughout the year.) Observers utilized a standardized observation form to document and assess the quality of implementation. After observations were completed, evaluators debriefed program facilitators and agency staff on adherence to intervention fidelity and discussed recommendations or additional technical assistance, as needed. Formal written feedback was provided to agency and program staff approximately two weeks after observations in order to support improvement in implementation fidelity and quality. D. Outcomes for impact analyses This study explored the impact of the offer to participate in TOP® on participants’ reports of sex with no effective form of birth control immediately after the end of the intervention and after a 12-month follow-up period, as well as self-reported incidence of pregnancy. The primary research question was answered with a single dichotomous item asking “In the past 3 months, have you had sexual intercourse without using an effective method of birth control, even once?” One secondary research question was answered using responses to the same question from the 12-month post-intervention, follow-up survey. Another secondary research question evaluated the effect of TOP® on recent pregnancies. This required the evaluation team to construct a composite variable to assess incidence of pregnancy since the baseline survey. This was computed based on the two administrations of the 8 questions “Have you ever been pregnant or to your knowledge caused a pregnancy?” and if so, “How many times?” which were recorded at baseline and at 12-month follow-up. Participants who had never been pregnant at baseline but had at least one pregnancy at follow-up or had been pregnant but reported a number of pregnancies greater than those reported at baseline were coded at one. While others who had never been pregnant at follow-up or had a number of pregnancies equal to those reported at baseline were recoded as zero. Table III.1 presents the outcome measures in tabular form. Table III.1 Behavioral outcomes used for primary and secondary impact analyses research questions Timing of measure Outcome name Description of outcome relative to program . Primary Outcome . The variable is a dichotomous yes/no measure of whether a Recent Sexual person has had sexual intercourse without an effective means Immediate post Activity of birth control in the 3 months prior to the survey. The measure intervention is taken directly from the following item on the survey: • “In the past 3 months, have you had sexual intercourse without using an effective method of birth control, even once?” The variable is constructed as a dummy variable where respondents who respond yes they have had sex are coded as 1 and all others are coded as 0. Secondary . Outcomes . Recent Sexual The variable is a dichotomous yes/no measure of whether a 12 month follow-up Activity person has had sexual intercourse without an effective means of birth control in the 3 months prior to the survey. The measure is taken directly from the following item on the survey: • “In the past 3 months, have you had sexual intercourse without using an effective method of birth control, even once?” The variable is constructed as a dummy variable where respondents who respond yes they have had sex are coded as 1 and all others are coded as 0. Recent Pregnancy The variable is constructed as a dichotomous variable with 12 month follow-up values ranging from 0 (not pregnant during the 21 month period) to 1 (pregnancies during the 21 month period reported) and is comprised of the following questions: • “Have you ever been pregnant even if no child was born?” • “How many times have you been pregnant?” E. Study sample The analytic sample was defined as all eligible youth who consented both to the intervention and the evaluation study, completed the baseline measures, and completed the post-intervention and/or 12-month follow-up measures. Some participants with logically inconsistent responses were removed from the analysis. 9 As seen in Appendix B.1, 2,428 youth who completed a baseline survey completed an immediate post survey for an overall response rate of 50.9%. The differential in attrition rates for this survey administration was 2.4% (attrition rate for the treatment group was 51.1% and the control group was 48.7%). At the 12-month follow-up time point 580 teens (treatment group: 318; control group: 262) completed a survey for a response rate of 12.2%. The differential attrition rate for this survey administration was 2.3%. F. Baseline equivalence All demographic and outcome variables of interest were assessed for baseline equivalence. A t-test was used for continuous variables (age) and chi-square was used for categorical variables (including race) to assess potential differences in demographic or behavioral covariates at baseline by condition assignment. There were no differences between groups at baseline for any of the tested covariates or outcomes. Table III.2 summarizes these results. Identical tests for baseline equivalence were conducted for the subset of the analytic sample that participated in the post survey and those that participated in the 12 month follow up. No significant effects were found on any demographic or behavioral covariate in those additional tests. Table III.2 Summary statistics of key baseline measures for youth completing TOP Intervention mean Comparison mean Post intervention sample or % (standard or % (standard Chi square Baseline measure deviation) deviation) or t test p Value Age 13.8 (1.55) 13.9 (1.60) 1.56 .211 Gender (female) 59.5% 57.8% 0.21 .882 Race/ethnicity: Black 89.8% 90.1% 0.58 .831 Race/ethnicity: Other 10.2% 9.9% . . Ever had sex 12.7% 12.4% 0.05 .825 Ever been pregnant 0.9% 1.6% 2.16 .142 Sex in past 3 months 5.7% 6.3% 0.39 .533 Sex NC past 3 months 3.0% 3.6% 0.70 .401 Sex NB in past 3 months 2.9% 2.7% 1.03 .311 Sample size 1248 1180 . . * Intervention mean Comparison mean Follow up sample or % (standard or % (standard Chi square Baseline measure deviation) deviation) or t test p Value Age 13.7 (1.47) 13.7 (1.47) 0.00 .945 Gender (female) 56.6% 58.8% 0.28 .598 Race/ethnicity: Black 88.7% 87.8% 0.11 .739 Race/ethnicity: Other 11.2% 12.2% . . Ever had sex 10.1% 7.4% 1.31 .252 Ever been pregnant 0.3% 1.2% 1.49 .222 Sex in past 3 months 2.8% 2.7% 0.01 .908 Sex NC past 3 months 1.9% 1.5% 0.10 .749 Sex NB in past 3 months 0.6% 1.5% 1.13 .288 Sample size 318 262 . . Notes: NC= no condom, NB = no birth control 10 G. Methods The analytic sample includes only youth who were randomized, and all youth were analyzed according to their initial treatment status. As previously stated, baseline data collection was completed prior to randomization, therefore there were no missing baseline surveys. No imputation methods were employed for missing data. For the main analyses some instances of data with logically inconsistent values were observed. Logical inconsistencies took two forms: those within a single survey administration, and those between administrations. A decision rule for each type of logically inconsistent data was developed. Omission of responses following a confirmed negative response for a precursory behavior (e.g. missing sexual behavior questions, such as number of times a teen has had sex, for teens who report never having had sex) were recoded as zero to reflect logical consistency. Values that positively indicated the presence of a behavior following an inconsistent or negative report of a precursory behavior (e.g. reporting having been pregnant but never having had sex) resulted in exclusion of the entire case from the analytic sample. Similar inconsistent reports across survey administrations (e.g. reporting never having had sex on a follow-up or post survey after reporting sexual behavior on the baseline survey) resulted in the record being discarded. 1. Impact evaluation A linear probability model was constructed in a SAS environment using PROC GENMOD to test the following: where the probability of outcome (P(Y=1)) is predicted by the effect of the intervention T plus the additional k covariates. Models examining the primary and secondary research question of likelihood of sex with no effective birth control were adjusted for baseline demographics of age, race, gender, agency, and study cohort as well a single behavioral covariate for the baseline value of the outcome variable. All effects will be assessed for significance at the .05 level (two tailed test). 2. Implementation evaluation Facilitators submitted data on the lessons they were completing, and activities planned/completed for each club session via an online data portal. They also reported data on additional lesson specific fidelity monitoring logs at the end of each lesson, which were submitted via email or fax. Adherence was calculated as the percentage of treatment teens attending 75% of the intervention. Facilitators reported data for each session via an online data portal. Level, lesson number, number of activities planned, and number of activities completed were recorded in the facilitator fidelity monitoring tool. Lesson number and activities were also captured in the online data portal and the lesson-specific fidelity monitoring log. Ten percent (10%) of all sessions were directly observed using the Louisiana observation form and entered into an online data portal. 11 Each session was conducted by at least one TOP® certified facilitator. All facilitators were TOP®-certified and received ongoing support. Two facilitators from each of the seven agencies received training of trainers (TOT) training. OPH monitored all facilitator training. Staff- participant interaction was monitored through direct observation and was documented in the Louisiana TOP® observation form and entered into the online data portal. Quality of staff- participant interactions was assessed using scores on the observation forms. Ten percent (10%) of all club sessions were observed. The quality of youth engagement was monitored through direct observation and was documented in the Louisiana TOP® observation form and entered into the online data portal. Quality of youth engagements was assessed using scores on the observation forms. The potential confounding variable of participation in another youth development program outside of the one being evaluated was captured in all three participant survey administrations. While evaluation participants were not offered any other teen pregnancy prevention or youth development programming by the agencies, both treatment and control group participants may have had access to other programming from other sources. External events affecting implementation were captured via reports from program monitor. During the club cycle, any minor changes to implementation were approved by the OPH Project Coordinator with input from the Evaluation Director. OPH program monitors and evaluators looked for changes during observation and evaluation review data submissions to identify possible unapproved implementation changes. IV. Study findings A. Implementation study findings Adherence Adherence to the TOP® model for session facilitation was high. According to reporting by facilitators, all planned activities were completed for 98% of sessions. In addition, there was 100% agreement on number of activities planned and completed between observers and facilitators for club sessions observed. The TOP® program requires that at least one trained facilitator be present for every 25 participants and that the trained facilitator attend the full club meeting. All TOP® clubs met this requirement. Dosage Fidelity to the TOP® model in terms of dosage received by youth was low. According to TOP® standards intended program dosage is that youth attend a minimum of 25 weekly sessions (one per week at 40-50 minutes each) and a minimum of 20 hours of CSL over the nine-month period. The dosage received by youth assigned to TOP® clubs did not meet expectations of the program model. Only 21% of teens completed 75% of the recommended 25 sessions, and 6% completed the full 25 sessions. Youth assigned to TOP® clubs attended a mean of 8.2 sessions, a median of 4.5 sessions, and a mode of 0 sessions. On average, youth assigned to TOP® clubs completed four hours of CSL, far below the minimum of 20 hours required by the program. Qualitative reports and technical assistance provision revealed that attendance was affected by a number of issues even among those who were regularly participating. These included participant transportation issues, staff turnover, inclement weather, and inconsistent meeting times. 12 Quality of Implementation Evaluators observed 10% of sessions throughout the study. During those observations evaluators scored facilitators high for both facilitator-participant relationships and participant engagement. In both these areas the median score was 4.6 out of 5, with 5 being excellent. Of treatment participants that responded to the survey items regarding participant perception of TOP® facilitators almost two-thirds (64%) strongly agreed that the facilitator was caring, and 94% strongly agreed that they were understanding. In addition, 79% strongly agreed that the class was a safe, values-neutral environment. Context In order to assess possible contamination from participation in other youth development programs that could have had similar content, all youth were assessed with a single survey item at the pre-intervention or baseline, post-intervention and 12-month follow-up time points to determine participation in other pregnancy prevention or sexual risk behavior reduction programming. As reported at the immediate post-intervention survey administration 18% of youth assigned to the control group participated in a similar community service or teen youth group during the previous 12 months. Some youth may have participated in the TOP® intervention. B. Impact study findings No effect for condition was found for the primary research question of sex with no effective method of birth control in the three months immediately following the end of TOP® . After adjusting for relevant covariates 4.4% of TOP® youth reported having had sex with no effective form of birth control in the three months prior to the end of the intervention while 3.8% of control youth reported the same. No effect of any of the demographic covariates other than age and sex with no birth control was found to be significant at the level. Similarly the intervention had no effect on sex with no effective method of birth control at the 12-month follow-up, nor did TOP® affect the likelihood of having been pregnant since TOP® initiation measured at the 12-month follow-up . No behavioral or demographic covariates were significantly related to the secondary outcomes. Tables IV.1 and IV.2 present the results of the analyses to address the primary and secondary research questions. 13 Table IV.1 Estimated effects using data from immediate post to address primary research question Intervention compared with comparison Mean Intervention % Comparison % difference (p-value Outcome measure (standard error) (standard error) of difference) Recent Sex no birth control .0444 (.0062) .0381 (.0066) .0063 (p=.4332) Sample Size 1248 1180 . Table IV.2 Estimated effects using data from 12-month follow-up to address secondary research questions Intervention compared with comparison Mean Intervention % Comparison % difference (p-value Outcome measure (standard error) (standard error) of difference) Recent Sex no birth control .0387 (.0156) .0418 (.0174) -.0030 (p=.8784) Pregnancy since baseline .0181 (.0078) .0188 (.0084) -.0007 (p=.9478) Sample Size 318 262 . 14 V. Conclusion No evidence was found to support the short or long term impact of the offer of attending TOP® on engagement in sexual intercourse without effective birth control, or teen pregnancy. It should be noted that these findings are not consistent with the previous studies by Philliber and others or with work cited by the developer, which found a more than 50% reduction in teen pregnancy as well as other problem behaviors. It is entirely possible that the TOP® program model may not be effective or not effective in the population or the settings in which it was implemented during this study. Under the standard null hypothesis testing paradigm, a failure to reject the null, as seen in these three analyses, does not entail a valid inference of no effect of treatment. We did not detect any statistically significant differences between the treatment and control groups on any of our primary or secondary outcomes. According to Platt’s method of strong inference examination of possible alternate hypotheses, which are often embodied in study weaknesses, additional analyses are warranted in this case (Platt, 1964). Chief among these alternate hypotheses is that the TOP® intervention was not delivered as intended or in a way that was not impactful on these outcomes. As noted previously, adherence to program fidelity in terms of design and quality was found to be strong, however, dosage was quite low in terms of the number of sessions attended and community service learning hours completed. The current study was conducted exclusively in community-based settings while previous work was done in schools or mixed settings, some of which may have had more captive populations. Additional contextual factors surrounding the current study could explain the inconsistent findings. The original evaluation studies were completed and published more than 15 years ago and some in high risk populations. Many school settings now require or otherwise incorporate activities similar to community service learning into their curricula, which could be responsible for smaller observed differences between the treatment and control groups. While these results did not support the hypothesis that TOP® causes a reduction in risk for teen pregnancy and related risk behaviors, they do add to the growing knowledge base surrounding teen pregnancy programs and ultimately should lead to continued improvements to the declining teen pregnancy rate. Additional work is needed and should focus on the more distal mechanisms of teen pregnancy prevention, including psychosocial outcomes. 15 VI. References Allen, J. P., Kuperminc, G. P., Philliber, S., & Herre, K. (1994). Programmatic prevention of adolescent problem behaviors: the role of autonomy, relatedness, and volunteer service in the Teen Outreach Program. American Journal of Community Psychology, 22(5), 617–638. Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly targeted prevention program? Differential efficacy across populations in the Teen Outreach Program. Journal of Community Psychology, 29(6), 637–655. Allen, J. P., Philliber, S., Herrling, S., & Kuperminc, G. P. (1997). Preventing teen pregnancy and academic failure: Experimental evaluation of a developmentally based approach. Child Development, 68(4), 729-742. Allen, J.P., Philliber, S., & Hoggson, N. (1990). School-based prevention of teenage pregnancy and school dropout: Process evaluation of the national replication of the Teen Outreach Program. American Journal of Community Psychology, 8, 505-524. Centers for Disease Control and Prevention. Sexually Transmitted Disease Surveillance 2013. Atlanta: U.S. Department of Health and Human Services; 2014. Centers for Disease Control and Prevention. Teen Birthrate by State 2014. Atlanta: U.S. Department of Health and Human Services; 2014. Retrieved December 18, 2015, from http://www.cdc.gov/nchs/pressroom/sosmap/teenbirths.htm Goesling, B., Colman, S., Trenholm, C., Terzian, M., & Moore, K. (2014). Programs to reduce teen pregnancy, sexually transmitted infections, and associated sexual risk behaviors: A systematic review. Journal of Adolescent Health, 54(5), 499 – 507. Isaacs, J. B. (2007). Cost-effective investments in children. Washington, D.C.: The Brookings Institution. Martin, J. A., Hamilton, B. E., Ventura, S. J., Osterman, M. J. K., Kirmeyer, S., Mathews, T. J., Wilson, E. (2011). Births: Final data for 2009. Table B. National vital statistics reports 60(1). Hyattsville, MD: National Center for Health Statistics. Retrieved October 23, 2015, from http://www.cdc.gov/nchs/data/nvsr/nvsr60/nvsr60_01.pdf Platt, J. R. (1964). Strong Inference. Science, 146(3642), 347-353. Oesterle, S., Kirkpatrick, M., & Mortimer, J. (2004). Volunteerism during the transition to adulthood: A life course perspective. Social Forces, 48(3), 1123. 16 Appendix A: Data collection efforts Table A.1 Data collection efforts used in the impact analysis of TOP® and timing Data collection effort Cohort 1 Cohort 2 Cohort 3 Baseline survey 09/15/11– 08/15/12– 08/21/13– 03/15/12 01/4/13 12/12/13 Start date of programming 10/10/11- 09/17/12- 09/23//13- 03/21/12 01-04-13 12/12/13 Immediate post-survey 06/12/12– 05/07/13– 05/27/14– 12/12/12 09/03/13 08/21/14 End date of programming 06/25/12- 05/07/13- 05/27/14- 12/12/12 09/03/13 07/10/14 12 month follow-up survey 04/05/13- 05/27/14- 05/04/15- 12/24/13 09/30/14 06/30/15 17 Appendix B: Study sample Table B.1 Recruitment and response rates by condition Total Total sample response Intervention Comparison Number of youth Intervention Comparison size rate response rate response rate Assigned to a 2,397 2,372 4,769 n/a n/a n/a condition Contributed a . . . . . . baseline survey Contributed 1,248 1,180 2,428 50.9% 52.1% 49.7% immediate post survey Contributed 12- 318 262 580 12.2% 13.3% 11.0% month follow-up survey 18 Appendix C: Implementation evaluation methods Table C.1 Methods used to address implementation research questions Implementation element Methods used to address each implementation element Adherence: How often were sessions Facilitators submitted data on the lesson(s) they were completing, and offered? How many were offered? activities planned/completed for each club session via an online data portal. They also reported data on additional lesson specific fidelity monitoring logs at the end of each lesson, which were submitted via email or fax. Adherence: What and how much was This was calculated as the percentage of treatment teens attending received? 75% of the intervention. Facilitators reported data for each session via an online data portal. Adherence: What content was delivered to Level, lesson number, number of activities planned, and number of youth? activities completed were recorded in the facilitator fidelity monitoring tool. Lesson number and activities were also captured in the online data portal and the lesson-specific fidelity monitoring log. 10% of all sessions were directly observed using the Louisiana TOP® observation form and entered into an online data portal. Adherence: Who delivered material to Each session was conducted by one TOP® certified facilitator. 100% youth? of facilitators were TOP® certified and received ongoing support. Two facilitators from each of the seven agencies received TOT training. OPH monitored and provided all facilitator training. Quality: Quality of staff-participant Staff-participant interaction was monitored through direct observation interactions and was documented in the Louisiana TOP® observation form and entered into the online data portal. Quality of staff-participant interactions was assessed using scores on the observation forms. 10% of all club sessions were observed. Quality: Quality of youth engagement with The quality of youth engagement was monitored through direct program observation and was documented in the Louisiana TOP® observation form and entered into the online data portal. Quality of youth engagements was assessed using scores on the observation forms. 10% of all club sessions were observed. Counterfactual: Experiences of The experiences of the control group and confounding variables were counterfactual condition captured using pre-posttest where control youth self-reported participation in other teen development programs. The agencies maintained logs documenting all contact with control youth. Context: Other TPP programming available or offered to study participants (both intervention and counterfactual) none Context: External events affecting implementation Reports from program monitor. Context: Substantial unplanned During the club cycle, changes to implementation were approved by adaptation(s) OPH Project Coordinator input from the Evaluation Director. OPH program monitors and evaluators looked for changes during observation and evaluation review data submissions to identify possible unapproved implementation changes. TPP = Teen Pregnancy Prevention. 19