Q&A with Jon Baron

In his most recent Straight Talk on Evidence post, Arnold Ventures’ Vice President of Evidence-Based Policy Jon Baron argues that – just like with medical research – we must strengthen the way we build and use evidence in the social sector in order to develop a body of demonstrably effective strategies for meeting our nation’s pressing challenges. We recently sat down with Jon to ask him about what steps practitioners, funders, and policymakers can take to create a more disciplined process for learning, testing, and improving social sector programs.

In your piece, you write that “social policy needs a major innovation and testing strategy aimed at building a body of proven-effective strategies” with “large RCTs as the final determinant of success.” In order to expand the pipeline of programs ready for an RCT, what are some steps earlier-stage organizations can take to lay a foundation for more rigorous evaluation?

Jon: Here are some examples of earlier steps programs can take to lay a foundation for an RCT:

• Conducting feasibility studies to establish whether the program can be successfully implemented – e.g., can enroll and retain participants, and be delivered in adherence to its key elements, in settings where the program would typically operate;

• Assessing whether program participants understand, and begin to apply, the skills taught by the program;
• Using results of the above activities to refine and improve the program;
• Developing a manual documenting the content of the program, and a training and supervision program for those who will deliver it;
• Creating a data system to support successful implementation of the program – i.e., to track whether its key elements are being faithfully delivered, and identify any substantial deviations that need correcting or indicate a need to refine the program; and
• Creating or identifying a data system to support an impact evaluation (should one eventually be undertaken) – i.e., a system to track outcomes of program participants as well as outcomes of a control or comparison group of nonparticipants.

Why is it important to build practitioners’ capacity to adopt a disciplined process for learning, testing, and improving their programs? 

Jon: If our country hopes to make progress on important social problems, it is essential for practitioners to engage in a disciplined evidence-building process, with the goal of moving their programs along the evidence continuum toward strong evidence of meaningful effects (and ultimately strong, replicated evidence of meaningful effects). It is also important for government and philanthropic funders of social programs to incentivize and assist the practitioners they fund to engage in such evidence-building activities.   

What criteria should practitioners and funders use to determine if a program is ready for an RCT? 

Jon: To be considered ready for an RCT evaluation, a program generally should be able to show that it: 

1. Can be successfully implemented, with close adherence to its key elements, in the settings where it would normally be delivered (e.g., public schools, community health clinics); and 
2. Has promising initial evidence to suggest it may have an impact on an important outcome.

Funding RCTs of programs that do not yet meet the above conditions is likely to be an inefficient use of resources, leading to findings of weak or no impact for programs that need further development to be effective, or might never be able to establish initial evidence of promise.

Our organization’s experience as a funder of RCTs is that too few programs currently meet these conditions to be ready for an RCT evaluation, underscoring the need for earlier-stage pipeline-building activities.