Over the past 9 years, I have spent a lot of time with leaders who are trying to learn from work they care about deeply. The settings have varied widely. One engagement centered on rural churches in North Carolina trying to help young children maintain their reading gains over the summer. Another focused on teacher coaches helping educators use outdoor classrooms to make learning more experiential. Others involved health centers expanding access to contraceptive care and a workforce training organization in Silicon Valley preparing to carry its model into new communities.

What connected those efforts was not a shared sector or program model. It was a shared pressure. Leaders needed to know whether their work was making a difference, and they needed evidence to help them decide what to do next in service of better outcomes for those they served.

That pressure often manifests as a pursuit of a rigorous third-party evaluation. It is a reasonable request. Boards and funders want evidence. Teams want to know whether the work they do each day is having the effect they hope for. While true, Strategic Evidence Planning has taught me that the most useful evidence rarely begins with the study itself. It begins with the opportunities that better evidence unlocks: clarity about the decisions ahead, a model ready to be learned from, an honest understanding of what participants would experience otherwise, the people and systems to carry the work, and a culture that can use evidence without turning it into blame. The contexts vary, but the pattern is remarkably consistent: good evidence building is not a single evaluation decision. It is a journey.

Evidence Starts With Decisions, Not Measurement

Measurement is easy to overproduce. Useful learning is harder. One of the most important shifts in our work is moving the first question from ‘What should we measure?’ to ‘What decisions are we trying to make, and what information would help us make them better?’

That shift might sound small, but it changes the work. It keeps resource-constrained organizations from building data systems around information no one has the time, authority, or appetite to use. The goal is not to ask them to collect more data, but to help them collect the data that will matter.

When we partnered with Out Teach, an organization that coaches teachers in outdoor experiential learning, the team brought a serious commitment to rigorous evaluation. But the first step was not to design a study; it was to build a learning agenda, a tool that asks what decision a piece of information will inform before anyone collects it. For Out Teach, that meant organizing questions around what leaders and coaches actually needed to answer: how school partnerships were progressing, whether coaching was changing teacher practice, and where the model needed strengthening before larger impact questions could be answered.

We saw the same pattern with Upstream USA, which works with states and health centers so that women have same-day access to the full range of contraceptive methods. Upstream had no shortage of important evaluation questions. The value of the evidence plan was in organizing those questions into a sequence: which ones would help the team optimize the model now, and which ones would help validate impact later. The plan became a decision-making tool, not just a research agenda.

Rigor Has to Meet the Program Where It Is

The instinct to reach for the most rigorous design is often a sign of seriousness. Leaders want to know whether the work is causing the outcomes they care about, and funders want to support programs that can withstand scrutiny. However, rigor before readiness can consume scarce resources and produce a finding that says more about uneven implementation than about whether a program has promise. A rigorous study is only as useful as the learning agenda that informs it.

Our work with The Duke Endowment’s Rural Church Summer Literacy Program — a six-week, church-based effort to help young children in rural North Carolina maintain reading gains over the summer — made this lesson concrete. When we looked closely at how the program was operating, we saw meaningful variation across sites in transportation, recruitment, and enrichment activities. The issue was not that the program lacked value. The issue was that the program was still developing, and the Endowment was preparing to expand.

Our recommendation was to wait for a large impact study. Not forever, and not because rigor did not matter. We recommended a staged path: define what quality implementation looks like, strengthen data collection, run smaller tests of change, and then move toward the most rigorous feasible design once the program could show it was being delivered consistently. Launching a large trial too early would likely have produced a result that was difficult to interpret and even harder to use.

The level of evidence should match the decision before the team. A high-stakes, difficult-to-reverse decision deserves stronger evidence. A lower-stakes, reversible decision can often be made with more readily available evidence, a quick test, and a clear plan to keep learning.

Every Causal Question Has a Hidden Assumption

Causal evidence always asks a comparison question, even when we do not say that part out loud. That context can change the entire evidence strategy before a study is ever designed. In technical terms, this is the counterfactual: what participants would have experienced if they had not received the program. In practical terms, it is the world a student, family, teacher, patient, or job seeker is already navigating.

The Rural Church Summer Literacy Program taught us this vividly. North Carolina law requires schools to offer voluntary summer reading camps to students through a policy called Read to Achieve. That meant a comparison group drawn from the same communities would very likely include students receiving another summer reading intervention. The real question was not, ‘Does this program work compared to nothing?’ It was, ‘Does this program improve outcomes compared to what families can already access?’

That distinction changed which designs were worth considering. It also pointed toward the value of existing administrative data rather than building an expensive new trial in an already crowded field. And it raised an interpretive challenge: if the comparison condition were itself uneven or weak, any future result would need to be understood in that light.

This is why the comparison condition has to be mapped early. A good causal question is not ‘compared to nothing’ unless nothing is the real alternative. It is compared to what people would otherwise experience. That is where credible evidence begins.

Evidence Plans Are Only As Durable As Their Underlying Infrastructure

Early in our history, we set out to provide actionable roadmaps as part of our Strategic Evidence Plans. The roadmaps mattered. They still do. But the work taught us that a plan is only as durable as the people and systems underneath it. The real deliverable is the capability that remains after the engagement ends.

Upstream made this lesson especially visible. The organization was scaling quickly, from its first statewide initiative in Delaware to additional work in Washington, Massachusetts, and North Carolina by the time we began our engagement. Its monitoring, evaluation, and learning team was talented and deeply committed — but also stretched. Team members were carrying data engineering and management responsibilities in addition to their learning and evaluation work.

We described the challenge using a term borrowed from technology: technical debt. When growth outpaces systems, an organization accumulates hidden costs, and the bill comes due at the worst possible moment.

For Upstream, the evidence questions could not be answered well until the foundation could carry them. Our recommendations centered on people and systems: senior data and evaluation leadership, clearer role specialization, stronger data governance, and the technology capacity needed to meet the expectations of sophisticated health partners.

JobTrain, a Silicon Valley workforce training organization serving low-income adults, many of them formerly incarcerated or long-term unemployed, reinforced the same lesson from a different starting point. JobTrain wanted to codify and replicate its career training model across new sites and partner organizations. Without thoughtfully codified programs and a clear understanding of which elements made them work, it could not responsibly expand. Nearly two-thirds of the investments in JobTrain’s evidence plan were directed toward internal talent and learning culture rather than evaluation. The capability questions had to come before the proof questions. That capacity building work gave JobTrain the internal muscle to document and codify its core program components, laying the groundwork for its first replication pilots with partner organizations, including Goodwill Industries and Destination Home.

Culture is What Turns Evidence Into Improvement

A learning agenda can be well designed. A data system can be well structured and reliable. A study can be methodologically strong. None of that guarantees the evidence will be used.

Evidence infrastructure only works when people trust what will happen with the information they provide. If data are experienced mainly as a tool for judgment, staff learn to protect themselves. They may avoid the system or quietly stop believing that the information is meant to help them improve. The model then bends toward looking good rather than getting better.

This was a crucial consideration in our work with Out Teach. Data collection had to be low-burden, woven into the work people were already doing, and visibly useful to the staff providing it. The organization needed transparency about which data would inform which decisions. It also needed a consistent tone from leadership: improvement over blame.

We saw the same dynamic at Upstream, where the most valuable evaluation work depended on the program and measurement teams trusting one another enough to collaborate honestly. Technical quality mattered, but trust determined whether the work could become part of how the organization learned.

Systems that practitioners experience as imposed get worked around. Systems they help shape, understand, and trust are much more likely to be used. The human side is not a soft addition to the technical work. It is the condition that makes the technical work matter.

Where This Leaves Us

Taken together, these lessons are less about any one evaluation method than about sequencing. 

Each step lays the groundwork for the next. That is what makes causal learning possible. Organizations do not get to a credible understanding of impact by jumping straight to the most sophisticated study. They get there by building the conditions that make the question answerable and the answer useful.

In Delaware, Child Trends estimated that Upstream’s work coincided with an increase in the use of long-acting contraception among Title X clients from 13.7 percent to 31.5 percent, along with a simulated drop in unintended pregnancy of close to 25 percent. Better evidence, better decisions, and ultimately, better outcomes for the people organizations serve.

Technology is changing quickly. AI-enabled approaches to causal learning are making it possible to do more with data organizations already collect, at lower cost and with faster feedback loops. But better tools do not replace the discipline underneath the work. A model can help analyze data. It cannot decide which question matters most, earn practitioners’ trust, or build the routines that make evidence useful. Those are still organizational choices — and they are the harder ones.

People move on. Programs evolve. Funders change priorities. Good evidence infrastructure stays, and it keeps an organization learning long after any one plan is finished.

Matt Hillard is the Senior Director of the Evidence for Outcomes practice at Project Evident.