By Kurt C. Stange, MD, PhD, Co-Investigator, ESCALATES
Why is it that research findings from different studies often seem to contradict each other? Why do summaries of research on a particular topic usually come to the unhelpful conclusion that ‘more research is needed’?1 Why is it so difficult to take what was learned in one time and place and apply it in another?
In “Bridging the Two Worlds of Evidence for Best Patient Care,” Laura Damschroder, MS, MPH, explains how the disconnect between the world of researchers and world of clinicians contributes to the long time it takes to get research’s best care recommendations to patients. Another reason for the apparent inconsistency and lack of transportability of research is that the standards for rigorous science focus on internal validity – the relationships between the factors that are the particular emphasis of a study. This is fine when the focus is something that isn’t influenced by outside factors.
For example, it is fine, even helpful, to focus narrowly when you are trying to isolate a single risk factor. The early observational studies that identified the risk factors for cardiovascular disease focused on one thing at a time – smoking, cholesterol, blood pressure. These studies tried to isolate the phenomenon of study from all the confounding factors. They did this in different ways – by restricting the study to people with only one risk factor, or by statistically “controlling” for other factors. And this research has worked fabulously well to identify individual causes of cardiovascular disease.
If we want to go beyond identifying risk factors, to intervene to reduce risk, the ideal study design goes even further in isolating a single factor for study. The experiment, the clinical trial, does this by using a coin flip to assign people to get the intervention or not, and then looks over time for differences in outcomes. For example, clinical trials have shown the benefit of aspirin for preventing heart attacks and strokes by randomly assigning people to receive aspirin or not. Following people over time has shown that certain groups have lower rates of cardiovascular problems on aspirin. The magic of randomization is that it washes out all the messy differences between people that might influence the effect of aspirin. The effect of people’s differences fade away when we look at what works on average.
But what if we want to actually make a difference in reducing cardiovascular disease risk in the real world, where everyone isn’t above average? What if we are interested in how different interventions translate the ABCS measures (aspirin use, blood pressure or cholesterol control, smoking cessation) of heart health from the lab or research setting to the places where most people get most of their care most of the time? In that situation, we don’t want to just isolate the ABCS services from the messy differences in people’s lives and the lives of those who provide their health care. To understand this messy world and to make a difference in it, we need to embrace and shine a light on its many confounding factors.
For me, contextual factors are one of the important behind-the-scenes details that help you make sense of what’s really going on when you’re trying to make a difference in real practices and patients. It’s hard to make sense of the practice’s lack of change in ABCS, for instance, and their lack of engagement with their practice facilitator or coach until you understand that they are getting pressure from their health system to work on cancer prevention, not ABCS. Or, that the practice is working on another project related to ABCS outreach that has asked that wait to implement a hypertension protocol because for that study the practice is in another wave. If you don’t know this information, you could badly misinterpret what’s happening.
It turns out that for research about health care and health, context matters. It matters a lot. In 2013, 14 AHRQ-sponsored study teams did an experiment with reporting context. They submitted manuscripts from their studies of Transforming Primary Care Practice to the Annals of Family Medicine for publication in a journal supplement. And then they did something remarkable. They illuminated the black box contextual factors and described what someone else might need to know to really make sense of their work – things that fall outside the usual internal validity, the minimal reporting requirements. They reported those extra things in an appendix.
I was one of the editors for that supplement. I thought I understood their studies-until I read the appendices. There I learned about policy initiatives in the state where one of the studies was done. That information suggested there was probably no way their effective intervention would work in my state, where we didn’t have those policies and the infrastructure the research team had generated. I learned about long relationships and pivotal events – outside of the internal validity of the study – that affected how the research team was seen by the participating practices in ways that likely affected subject enrollment, intervention uptake, and acceptance by patients. I wouldn’t have known any of this information if the investigators hadn’t reported them in the appendices. You can see for yourself if you want to look at the supplement. With context like this, you can understand the lessons so much better and adapt the lessons to your situation.
To understand this messy world and to make a difference in it, we need to embrace and shine a light on its many confounding factors.
Our context flashlight includes both statistics and stories, quantitative measures and qualitative data. We are focusing this light to answer two questions:
- What contextual factors do we need to know to understand what happened within the Cooperatives and why?
- What would someone else need to know to transport or implement what we learned here into their setting?
The ESCALATES team is looking at context across multiple levels, from differences in patients and practices, to local networks and health care systems, communities, and larger policies and politics. We are gathering the different perspectives of diverse stakeholders on what contextual factors might be important at the beginning, middle and end of the project. Shining light into the black box of context is challenging because different contextual factors matter at different times and situations. Context is not amenable to a simple checklist approach. But it is open to specification that focuses attention on what matters at multiple levels, perspectives and time points.
By shining a light on contextual factors and bringing assessment of those contextual elements into its evaluation, ESCALATES is advancing the science of what works in what situation into the real world where people live their lives and receive their health care.
Study recruitment is typically displayed in Table 1 of research papers, showing a few basic characteristics of participants, and if we are lucky, comparing those to the population or to those who were approached but declined. In ESCALATES, we see that recruitment is affected by the challenges of a constantly shifting health care environment – affiliations and de-affiliations, practices and systems closing, moving, consolidating. In each region, there have been local partners such as non-profits or hospital networks that were important to recruitment, while history and previous relationships mattered a lot. For some Cooperatives, recruiting has been extremely time-intensive, requiring multiple touches and in some cases 45 hours of outreach before a practice was successfully enrolled.
Recruiting the health care systems, which bought up many of the previously independent practices in different regions, seemed like a solution, but relationships still need to be developed with the individual practices. Working with large systems that have their own culture, priorities, and often already organized quality improvement departments with identified goals, has its own challenges. Many practices are overwhelmed and struggling to keep their doors open and could not afford to spend time or money on anything but patient care. Sometimes contextual factors can be the early indicator for things even more important than the primary focus of the study.
For some of the most important and complex problems in improving health care and health, context matters. By shining a light on contextual factors and bringing assessment of those contextual elements into its evaluation, ESCALATES is advancing the science of what works in what situation into the real world where people live their lives and receive their health care.
- Those used to reading the scientific literature will recognize the frequently used phrase; “Due to great heterogeneity of treatment effects, no meta-analysis was possible. More research is needed.” ↩