The goal of proposed project is to strengthen the public health impact of psycho-social interventions by identifying fidelity measurement methods that can be used for research and practice. To date, fidelity has posed a thorny measurement quandary because few reliable, valid, and efficient fidelity measurement methods exist. The gold standard for measuring fidelity to psycho-social interventions, direct observation of therapist behavior, requires extensive resources. When fidelity is measured in the community, the most commonly used and least resource intensive method is self-report. Unfortunately, concordance between observation and self-report is low.

There is a critical need to identify and evaluate methods of fidelity measurement that are accurate (i.e., measure what they intend) and cost-effective. Our objective in this measurement proposal is to compare the accuracy and cost-effectiveness of three methods: self-report, chart stimulated recall and behavioral rehearsal in assessing fidelity to cognitive-behavioral therapy, an established evidence-based practice. We will randomize 135 therapists, trained in cognitive-behavioral therapy through an initiative to implement evidence-based practices in the City of Philadelphia, to 3 conditions: self-report (N = 45), chart stimulated recall (N = 45), and behavioral rehearsal (N = 45). All conditions will include direct observation using the Therapy Process Observational Coding System as the gold-standard comparison. In Aim 1, we will identify the most accurate fidelity measurement method. In Aim 2, we will estimate the economic costs and cost-effectiveness of the proposed fidelity measurement methods. In Aim 3, we will compare stakeholders’ willingness to use each method, as well as identify their perceived barriers and facilitators to use of each method. The proposed work is consistent with the NIMH strategic plan, specifically Objective 4, to strengthen the public health impact of NIMH-supported research.

This study will have a significant positive impact by producing fidelity measurement methods that can then be used by implementation scientists for research and community mental health clinics to monitor therapist fidelity, an indicator of therapy quality.

Role: Principle Investigator
Funder: NIH
Mechanism: R01

Suicide is a leading cause of death in children and adolescents and a critical public health concern. One promising suicide prevention strategy that is under-utilized includes reducing access to lethal means, or means restriction. Firearms are an important target for means restriction given that 1 in 3 homes possess a firearm and firearms are the most lethal manner by which suicide is attempted.

Primary care is an ideal setting in which to provide firearms means restriction given that almost half of youth who die by suicide do not access specialty mental health services in the 12 months preceding their attempt. The Safety Check intervention is an evidence-based practice for implementing firearms means restriction in pediatric primary care (including pediatrics, family medicine, and adolescent medicine). Despite the existence of this intervention and recommendations from the American Academy of Pediatrics, providers rarely discuss means restriction or firearms during visits, suggesting the need for a better understanding of the barriers and facilitators to implementing means restriction in pediatric primary care.

Implementation science frameworks, including the Consolidated Framework for Implementation Research (CFIR), suggest the importance of attending to multiple levels of context during implementation, including provider (e.g., self-efficacy), organizational (e.g., expectations about provider behavior), system (e.g., prompts in the electronic health records), and intervention (e.g., acceptability) factors. The NIMH-funded Mental Health Research Network (MHRN), a consortium of 13 healthcare systems across the United States, affords a unique opportunity to better understand how to implement means restriction in pediatric primary care from a system-level perspective. Our objective in this application is to collaboratively develop implementation strategies in partnership with MHRN stakeholders to increase the use of means restriction in pediatric primary care.

In Aim 1, we will survey leadership and primary care physicians in 96 primary care practices within 2 MHRN systems (Henry Ford Health System and Baylor, Scott, & White Health) to understand acceptability and use of the three components of the Safety Check intervention (i.e., screening, brief counseling around gun safety, and provision of gunlocks).

In Aim 2, in collaboration with MHRN stakeholders in these 2 systems, we will use intervention mapping and the CFIR for to systematically develop and evaluate a multi-level menu of implementation strategies for firearm means restriction in pediatric primary care. The proposed work is consistent with the NIMH strategic plan, specifically Objective 4, to strengthen the public health impact of NIMH-supported research, and will lead to a hybrid effectiveness-implementation R01 proposal. The long-term goal of this line of research is to reduce death by suicide by increasing the use of evidence-based strategies in pediatric primary care while also promoting multi-level implementation strategies informed by a systematic and rigorous development approach.

Role: Principle Investigator
Funder: NIMH
Mechanism: R21

The Philadelphia Department of Behavioral Health and Intellectual Disability Services (DBHIDS) in collaboration with the Penn Center for Youth and Family Trauma Response and Recovery (CYFTRR) at Pennsylvania Hospital’s Hall-Mercer Community Mental Health Center was awarded a Substance Abuse and Mental Health Services (SAMHSA), Community Treatment and services (CTS) grant (SM61087) to develop a trauma-informed and trauma-focused system for young people and their families in Philadelphia.

The Center, named the Philadelphia Alliance for Child Trauma Services (PACTS) , will function as a consortium of child and adolescent behavioral health providers, pediatric and other child serving agencies that see young people who may be traumatized. PACTS will be responsible for the implementation of trauma-focused evidence-based practices (EBP) and for increasing screening and assessment of children in a variety of physical and behavioral health care settings. The impact of PACTS will be evaluated in collaboration with an evaluation team at the University of Pennsylvania’s Center for Mental Health Policy and Services Research (CMHPSR) to collect and provide data and outcomes to providers expeditiously as required by SAMHSA. This mixed-methods study uses a pre-post design. We will investigate the effectiveness of providing trauma-focused cognitive-behavioral therapy (TF-CBT). The study will include community therapists who provide mental health treatment services to uninsured and Medicaid-eligible children and families as well as children and families receiving those services in the city of Philadelphia. Within 13 agencies, we will gather data from all participating therapists (N = 100), and youth and families receiving treatment (N = 100). Data will be collected prior to treatment, at discharge, and 6-months after baseline. Quantitative data will include measures of general child functioning and symptoms. We will also gather information about clinician attitudes towards evidence-based practices. Qualitative data will include semi-structured interviews with a subset of the sample around the implementation experience. Findings from the proposed research will inform national roll-outs of trauma-informed care and provide information on whether the implementation of these evidence-based practices improves youth outcomes.

Role: Lead Evaluator
Funder: SAMHSA

Evidence-based treatments (EBTs), treatments that have been evaluated scientifically and show evidence of efficacy, are not widely available in community settings. It takes up to 17 years after proof of efficacy for these treatments to make their way into community settings. In response to the call for implementation of EBTs, states and counties have mandated behavioral health reform. Evaluations of the impact of these policies and other initiatives on implementation are rare.

A new center to support the implementation of EBTs, the Evidence Based Practices Innovation Center (EPIC), has been created in Philadelphia as part of the Department of Behavioral Health. We have operationalized the activities of EPIC as a policy mandate. EPIC will encourage and incentivize clinics to implement EBTs for youth by providing an infrastructure to support these services (e.g., training, consultation) and potentially through an enhanced payment rate for provision of EBTs.

The formation of EPIC offers a rare and important opportunity to prospectively study implementation in response to a policy mandate. This mixed-methods study is prospective, longitudinal, and observational. We will investigate the response of 30 community mental health clinics to EPIC and the resulting policy from the Department of Behavioral Health encouraging and incentivizing clinics to implement EBTs for youth. Study participants will be approximately 210 therapists and 30 executive directors. Data will be collected prior to the policy, and then 2 and 4 years following the implementation of the policy. Quantitative data will include clinician-level measures of intervention implementation and potential moderators of implementation (organizational and leader-level variables). Clinician-level measures include self-reported therapist fidelity to EBT techniques as measured by the Therapist Procedures Checklist-Revised, observed therapist fidelity as measured by the Therapist Process Observational Coding System- Strategies Scale, organization variables as measured by the Organizational Social Context

Role: Principle Investigator
Funder: NIMH
Mechanism: K23

In 2017, the National Institute of Mental Health (NIMH) awarded The Penn Center for Mental Health (CMH)and the Center for Health Incentives and Behavioral Economics (CHIBE) a P50 Center grant, “Transforming Mental Health Delivery through Behavioral Economics and Implementation Science.” The Center leverages principles of behavioral economics and implementation science to improve mental health service delivery. Three Principal Investigators lead the Center: Rinad Beidas, PhD, David Mandell, ScD, and Kevin Volpp, MD, PhD.

The Center was funded as part of the NIMH Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness (ALACRITY) to support the rapid development, testing, and refinement of novel and integrative approaches for (1) optimizing the effectiveness of treatments for and prevention of mental disorders; and (2) organizing and delivering mental health services in community settings. The ALACRITY Centers program is intended to support research that demonstrates an extraordinary level of synergy across disciplines and has a high potential for increasing the public health impact of existing and emerging mental health interventions and service delivery strategies.

The Penn ALACRITY Center’s research activities comprises three projects, each designed to improve the delivery of mental health care by changing processes at the organizational, practitioner, or patient level:

• Project 1 (Project Directors: Steve Marcus, PhD, Mark Olfson, MD, and Kevin Volpp, MD, PhD) compares ways to incentivize adherence to antidepressant medications in the first six weeks of treatment among adults newly diagnosed with depression.
• Project 2 (Project Directors: David Mandell, ScD, and Melanie Pellecchia, PhD) tests whether leveraging normative pressure and social status can increase data collection among community mental health workers treating children with autism.
• Project 3 (Project Directors: Rinad Beidas, PhD, Nate Williams, PhD, and Rebecca Stewart, PhD) explores how to design organizational strategies to incentivize therapists’ use of evidence-based practices.

In addition to specific research projects, the Center supports the development of novel statistical methods and study designs to increase knowledge of what contributes to successful implementation of evidence-based mental health treatments, and how best to leverage this knowledge to increase quality of mental health care and outcomes.

Role: MPI
Funder: NIMH
Mechanism: P50