Assessing the external validity of an experimental wage subsidy

"In Canada, a policy aiming at helping single parents on social assistance become self-reliant was implemented on an experimental basis. The Self-Sufficiency Entry Effects Demonstration randomly selected a sample of 4,134 single parents who had applied for welfare between January 1994 and March 1995. It turned out only 3,315 took part in the experiment despite a 50% chance of receiving a generous, time-limited, earnings supplement conditional on finding a full-time job and leaving income assistance within a year. The purpose of this paper is to determine whether a non-response rate of 20% is likely to harm the external validity of the experiment. We compare the estimated impact of the program using experimental data only to that obtained using additional data on individuals not taking part in the experiment. We find strong evidence of non-response bias in the data. When we correct for the bias, we find that estimates that rely on experimental data only significantly underestimate the true impact of the program"--Forschungsinstitut zur Zukunft der Arbeit web site.