The Origins of AHRQ’s EvidenceNOW Evaluation: ESCALATES

Deb Cohen, PhD Principal Investigator, ESCALATES


I had the opportunity to talk with David Meyers, MD, the Chief Medical Officer at the Agency for Healthcare Research and Quality (AHRQ) about the Agency’s EvidenceNOW initiative. He recently wrote about the origins of EvidenceNOW: Advancing Heart Health in Primary Care Initiative. And in an AHRQ Views blog post, he highlighted two goals for EvidenceNOW: “to measurably improve the health of Americans through delivery of evidence-based primary care” and “to build a blueprint of how to provide small primary care practices with quality improvement support.”

David talked with me about why AHRQ funded an independent overall evaluation of EvidenceNOW. The transcript has been lightly edited.


Deb: Can you explain, from AHRQ’s perspective, why it was important to have an independent national evaluation for EvidenceNOW?

MeyersDavid: That’s a great question. We recognize that this is a very large investment. AHRQ is committed to its success, and we believe in the project enormously. But, we recognize that some folks might point out that AHRQ was biased if there are positive findings. So we wanted to be very clear from the beginning that AHRQ is dedicated to learning from EvidenceNOW whether the results are positive or negative because those findings are critical to improving primary care practice.

Needless to say, we are hopeful that there will be positive outcomes, and we want people to have real confidence in them. For this reason, we were committed from the very beginning to having a national evaluation that was external and independent – both external from AHRQ to avoid any perceived bias, and also independent, which is why we used the grant mechanism and not a contract. Your evaluation – ESCALATES – will be truly independent of AHRQ in both its conclusions and the dissemination of those results.

Although we knew there would need to be cooperation between the evaluator and the Cooperatives, we wanted the evaluator to be independent and external from the Cooperatives in addition to AHRQ and the government.

 

Deb: That makes sense. I’m wondering what you all were looking for in an evaluation of this program?

David: We thought carefully about what was necessary in evaluating this initiative, and we put those ideas into the FOA [funding opportunity announcement]. This included the idea that the evaluation had to have a quantitative analysis of the effect on the delivery of the ABCS that would look across not just one Cooperative but all of the Cooperatives, as well as an evaluation of the concept of practice capacity. Did the practices that received external quality improvement support increase their capacity, and were they better equipped to use evidence in the future?

The national evaluation and the work ESCALATES is doing builds on the tremendous efforts of the individual Cooperatives and is the next step in translating, teaching, and guiding this whole field.

Thus, we built in a request for both quantitative and qualitative analyses. The aim was to have the evaluation try to get a handle on not just did it work, but what was it that worked, and are there ways to look across Cooperatives to see patterns about what works in which situations. One of the things we believe is that this isn’t just going to be a simple yes or no. We aren’t looking for a grantee to just do a high-level analysis, but to go deeper into it, and try to build on what we eventually hope to use as a blueprint for how others might use practice facilitation and the other quality improvement supports moving forward.

This is implementation research. It just comes with the territory that these interventions are going to be complex. The evaluation needs to get in there and really poke around to understand what is happening, to see the nuances. That’s what we thought would produce a really rich, valuable evaluation. We wanted to be able to look under the hood of these interventions, and that’s why we were really interested in the partnership developing between the evaluator and the Cooperatives for the purpose of evaluation. We recognized that we are asking the evaluator to do something very difficult. The Cooperatives are all independent grants, and we needed an evaluator that had experience building a sense of a larger whole and that could build cooperation.

 

Deb: What role do you think the evaluation has in field-building? That might not be the right word.

David: I’d probably use slightly different words, but I think it’s the same idea. We set this up not to become a, “Do all the evaluation process, collect all your data, go away into your ivory tower and spend two years analyzing the data, and then produce the book of answers” evaluation. When we wrote the FOA, we were looking for people with a more timely approach, willing to share lessons learned starting as soon as they had them, because the field is moving so fast. AHRQ really saw this as an investment to inform our public and private partners by helping them understand what I often call ‘externally-provided quality improvement support’ for primary care. That’s a mouthful. We often shorten it to ‘practice facilitation’ as a catch-all for that larger concept. We wanted our partners to understand: Does this work? How does it work? Who does it work for? Under what circumstances? What kind of results might you expect?

We recognized that we are asking the evaluator to do something very difficult.

The national evaluation and the work [ESCALATES is] doing builds on the tremendous efforts of the individual Cooperatives and is the next step in translating, teaching, and guiding this whole field. We are excited by the fact that the seven Cooperatives are different from each other. They’re not testing the exact same way to do this, and we’re hoping that will produce a richness that will inform the field moving forward. As evaluators, you’re going deeper to look at what about the Cooperatives’ delivery helped or didn’t help, or in what situations, and what intensity of intervention helped or didn’t help practices. Through your work and the work of the Cooperatives, we are building the instruction guide or’ how-to-manual’ for how other programs such as CMS [Centers for Medicare & Medicaid Services] and CMMI [Center for Medicare & Medicaid Innovation], or the QIO/QIN [Quality Improvement Organization/Quality Innovation Network] Program, or a private insurer like Blue Cross / Blue Shield, or a large health system, can help primary care practices thrive.

 

Deb: That’s a great point to end on I think. Thank you so much for your time, David. It was really interesting to hear first-hand about the genesis of the evaluation piece of EvidenceNOW.

David: Of course, Deb. I’m very proud of this initiative and can’t wait to see all that you discover through ESCALATES.

 

 

SEE AHRQ’s Story on the Genesis of EvidenceNOW