A company introduces a new sales process and revenue rises. A school changes its timetable and test scores improve. A manager reorganizes a team and output increases. In each case, the result may be encouraging. It does not yet establish what caused the improvement.
The essential question is what would have happened without the change. That unobserved alternative is the counterfactual. Impact evaluation uses comparisons and explicit assumptions to estimate it. [1]
A before-and-after comparison has a missing piece
Imagine two comparable branches of a hypothetical business. Branch A adopts a new process and monthly sales rise from 100 units to 120. Branch B keeps its existing process and sales rise from 100 to 115.
Looking only at Branch A suggests a gain of 20 units. Comparing the changes suggests a different estimate: A improved by 20 while B improved by 15, leaving a difference of five units. The method is called difference-in-differences because it compares changes over time between groups. [2] [1]
That arithmetic is illustrative, not proof that the new process caused five additional sales. The comparison is credible only if Branch B provides a reasonable estimate of how A would have changed without the intervention.
Examine the comparison before trusting the result
Perhaps Branch A hired an experienced salesperson, faced a new competitor, or served customers with different seasonal purchasing patterns. Any of those changes could affect the interpretation.
The Magenta Book’s discussion of difference-in-differences examines pre-intervention trends when constructing a comparison. Similar earlier trends can support the design, but cannot guarantee that the groups would have continued moving together afterward. [2]
It is also possible for the new process to affect the comparison group. If customers shift from B to A because of the intervention, the apparent relative gain may include redistribution between branches. The IDB identifies such spillovers as a challenge for interpreting program effects. [1]
Design the evaluation before the rollout
Evaluation becomes harder when the only available evidence is a selected success story. Before implementation, define the outcome, measurement period, eligible population, and comparison method. Preserve the initial measurements and record other changes that could influence the result.
Where practical and appropriate, a phased or randomized rollout can create a stronger comparison. The purpose is not to make routine management resemble a laboratory. It is to avoid a situation in which every outcome can be explained afterward as confirmation of the original belief.
For a modest operational change, a limited comparison may be enough to guide the next step. A costly or difficult-to-reverse commitment deserves stronger evidence. The method should fit the decision and the consequences of being wrong.
Separate effect from value
Even a credible positive effect does not establish that an intervention was worthwhile. The additional outcome must be considered alongside cost, disruption, and other consequences. Five additional units of sales mean something different if the process costs $50 than if it costs $50,000.
Similarly, an average effect can hide meaningful differences. A process may help one group while adding work for another. That is a reason to inspect the mechanism and the distribution of results, rather than stopping at a single favorable number.
Counterfactual thinking encourages a useful habit: compare the observed outcome with a defensible alternative. It allows decisions to be evaluated without assuming that every improvement was earned by the most recent intervention or that every setback was caused by it.
Sources
Research, reflection, and a wider perspective.
