Mixed methods research—i.e., research that draws on both qualitative and quantitative methods in varying configurations—is well suited to address the increasing complexity of public health problems and their solutions. This review focuses specifically on innovations in mixed methods evaluations of intervention, program or policy (i.e., practice) effectiveness, and implementation. The article begins with an overview of the structure, function, and process of different mixed methods designs and then provides illustrations of their use in effectiveness studies, implementation studies, and combined effectiveness–implementation hybrid studies. The article then examines four specific innovations: procedures for transforming (or “quantitizing”) qualitative data, application of rapid assessment and analysis procedures in the context of mixed methods studies, development of measures to assess implementation outcomes, and strategies for conducting both random and purposive sampling, particularly in implementation-focused evaluation research. The article concludes with an assessment of challenges to integrating qualitative and quantitative data in evaluation research.

Background: The use of rapid qualitative methods has increased substantially over the past decade in quality improvement and health services research. These methods have gained traction in implementation research and practice, wherein real-time adjustments are often made to optimize processes and outcomes. This brisk increase begs the questions: what does rigor entail in projects that use rapid qualitative analysis (RQA)? How do we define a pragmatic framework to help research teams design and conduct rigorous and valid rapid qualitative projects? How can authors articulate rigor in their methods descriptions? Lastly, how can reviewers evaluate the rigor of rapid qualitative projects?. Methods: A team of seven interdisciplinary qualitative methods experts developed a framework for ensuring rigor and validity in RQA and methods suitable for this analytic approach. We conducted a qualitative evidence synthesis to identify gaps in the literature and then drew upon literature, standard procedures within our teams, and a repository of rapid qualitative training materials to create a planning and reporting framework. We iteratively refined this framework through 11 group working meetings (60-90 minutes each) over the course of one year and invited feedback on items to ensure their completeness, clarity, and comprehensibility. Results: The Planning for and Assessing Rigor in Rapid Qualitative Analysis (PARRQA) framework is organized progressively across phases from design to dissemination, as follows: 1) rigorous design (rationale and staffing), 2) semi-structured data collection (pilot and planning), 3) RQA: summary template development (accuracy and calibration), 4) RQA: matrix analysis (matrices), and 5) rapid qualitative data synthesis. Eighteen recommendations across these sections specify best practices for rigor and validity. Conclusions: Rapid qualitative methods play a central role in implementation evaluations, with the potential to yield prompt information and insights about context, processes, and relationships. However, guidance on how to assess rigor is nascent. The PARRQA framework enhances the literature by offering criteria to ensure appropriate planning for and assessment of rigor in projects that involve RQA. This framework provides a consensus-based resource to support high-level qualitative methodological rigor in implementation science.

Ethnography has been proposed as a valuable method for understanding how implementation occurs within dynamic healthcare contexts, yet this method can be time-intensive and challenging to operationalize in pragmatic implementation. The current study describes an ethnographically-informed method of guided discussions developed for use by a multi-project national implementation program. The EMPOWER QUERI is conducting three projects to implement innovative care models in VA women’s health for high-priority health concerns – prediabetes, cardiovascular risk, and mental health – utilizing the Replicating Effective Programs (REP) implementation strategy enhanced with stakeholder engagement and complexity science. Drawing on tenets of ethnographic research, we developed a lightly-structured method of guided “periodic reflections” to aid in documenting implementation phenomena over time. Reflections are completed as 30–60 min telephone discussions with implementation team members at monthly or bi-monthly intervals, led by a member of the implementation core. Discussion notes are coded to reflect key domains of interest and emergent themes, and can be analyzed singly or in triangulation with other qualitative and quantitative assessments to inform evaluation and implementation activities. Thirty structured reflections were completed across the three projects during a 15-month period spanning pre-implementation, implementation, and sustainment activities. Reflections provide detailed, near-real-time information on projects’ dynamic implementation context, including characteristics of implementation settings and changes in the local or national environment, adaptations to the intervention and implementation plan, and implementation team sensemaking and learning. Reflections also provide an opportunity for implementation teams to engage in recurring reflection and problem-solving. To implement new, complex interventions into dynamic organizations, we must better understand the implementation process as it unfolds in real time. Ethnography is well suited to this task, but few approaches exist to aid in integrating ethnographic insights into implementation research. Periodic reflections show potential as a straightforward and low-burden method for documenting events across the life cycle of an implementation effort. They offer an effective means for capturing information on context, unfolding process and sensemaking, unexpected events, and diverse viewpoints, illustrating their value for use as part of an ethnographically-minded implementation approach. The two implementation research studies described in this article have been registered as required: Facilitating Cardiovascular Risk Screening and Risk Reduction in Women Veterans (NCT02991534); and Implementation of Tailored Collaborative Care for Women Veterans (NCT02950961).

Policy Points: Community‐engaged research (CEnR) engenders meaningful academic‐community partnerships to improve research quality and health outcomes. CEnR has increasingly been adopted by health care systems, funders, and communities looking for solutions to intractable problems. It has been difficult to systematically measure CEnR's impact, as most evaluations focus on project‐specific outcomes. Similarly, partners have struggled with identifying appropriate measures to assess outcomes of interest. To make a case for CEnR's value, we must demonstrate the impacts of CEnR over time. We compiled recent measures and developed an interactive data visualization to facilitate more consistent measurement of CEnR's theoretical domains. Context: Community‐engaged research (CEnR) aims to engender meaningful academic‐community partnerships to increase research quality and impact, improve individual and community health, and build capacity for uptake of evidence‐based practices. Given the urgency to solve society's pressing public health problems and increasing competition for funding, it is important to demonstrate CEnR's value. Most evaluations focus on project‐specific outcomes, making it difficult to demonstrate CEnR's broader impact. Moreover, it is challenging for partnerships to identify assessments of interest beyond process measures. We conducted a mapping review to help partnerships find and select measures to evaluate CEnR projects and to characterize areas where further development of measures is needed.Methods: We searched electronic bibliographic databases using relevant search terms from 2009 to 2018 and scanned CEnR projects to identify unpublished measures. Through review and reduction, we found 69 measures of CEnR's context, process, or outcomes that are potentially generalizable beyond a specific health condition or population. We abstracted data from descriptions of each measure to catalog purpose, aim (context, process, or outcome), and specific domains being measured.Findings: We identified 28 measures of the conditions under which CEnR is conducted and factors to support effective academic‐community collaboration (context); 43 measures evaluating constructs such as group dynamics and trust (process); and 43 measures of impacts such as benefits and challenges of CEnR participation and system and capacity changes (outcomes).Conclusions: We found substantial variation in how academic‐community partnerships conceptualize and define even similar domains. Achieving more consistency in how partnerships evaluate key constructs could reduce measurement confusion apparent in the literature. A hybrid approach whereby partnerships discuss common metrics and develop locally important measures can address CEnR's multiple goals. Our accessible data visualization serves as a convenient resource to support partnerships’ evaluation goals and may help to build the evidence base for CEnR through the use of common measures across studies.