Of Power and the Prayerful

 

A Critique of Randomised Control Trials in Poverty Alleviation

Last week, Michael Kremer, Abhijit Banerjee and Esther Duflo received the Nobel for Economics for "their experimental approach to alleviating global poverty” and for addressing “smaller, more manageable questions,” rather than big ideas.

This experimental approach is based on so called Radomised Control Trials. Simply put, in such experiments, a randomly selected group of individuals (randomization is a method of removing bias) receive an intervention whose efficacy is being tested. Changes that result in the conditions of this random experiment group is compared with those in another ‘similar’ group of individuals (referred to as a ‘control group’) that was not provided the intervention. The difference in outcomes is directly attributed to the intervention.

The RCT as a scientific research method is primarily widely practiced in clinical research to test the efficacy and safety of new pharmaceutical products/treatments. RCT is a pre-requisite for the regulatory approval of a new drug or vaccine. Evidence from such experiments have to confirm both internal validity (are the results of the study reliable?) as well as external validity (are the conclusions universally applicable to other population groups/locations?). Once a treatment/product is so approved, it is made available for general use.  

However, even in clinical research, which have an established level of design rigour and oversight, RCTs are not beyond question. The extent to which their results can be generalized to a wider patient population (external validity) is often interrogated, because carefully controlled study conditions may be far from reality and patients selected for a study may not necessarily be representative.

Given this background of RCTs in clinical research, the application of RCT for testing solutions for social change has been critiqued on two major grounds part from the fact that fully random sampling with blinded subjects is almost impossible in among human participants.

One, that RCTs of this kind rarely establish external validity of their conclusions. For example - if a solution, say to teacher absenteeism, succeeds in Udaipur, will the same solution, in toto, work elsewhere? Will it work in schools in Uttarakhand where teachers often don’t attend school because of the harsh terrains of the northern reaches. Or in the conflict prone West African country of Liberia? So instead of universally applicable causality between intervention and effect, as is expected from an RCT, we only get what has been called ‘circumstantial causality’. What makes it worse, is that while the results of an RCT are true at the time the experiment was undertaken, there is no guarantee that the results hold true ever after. So while RCTs can provide useful insights into what the Nobel press release called ‘manageable questions’, extrapolating such RCT results as having any impact on national or global poverty seems exaggerated.

Two, that RCTs answer technical, intervention-based research questions rather than structural issues that lead to unequal development, deprivation and unequal access to basic goods. People with less access include men, women and children who are poor, but more broadly citizens who find themselves unable - for a variety of class, caste, gender and geographical barriers - to exercise their basic rights to development. Say in the example of teacher absenteeism, should not issues of political economy be considered in addition to technical fixes such as incentives to teachers?

That said, the development sector which has been facing questions about ‘measuring change’, has been taken in by the forceful lure of quantitative data as the best kind of evidence. Donor agendas have pushed for measuring change and showcasing success, even though projects have been funded for a measly twelve or eighteen months. This combination of brushing aside the macro dimensions of poverty in favour of micro interventions and short termism and worse, a fear of system-wide change can have lethal impact on how we tackle development and global poverty. More broadly, the question of what constitutes ‘hard’ evidence is worth pondering upon.

Let us not forget that some of the most effective programmes, such as the school mid-day meal and the rural guarantee programme germinated as ideas which were tested at a smaller scale before they became national programmes. Instead of the RCT, these processes involved the participation of people,  an application of reason, observation and intuition, and a process of iterative improvements based on a monitoring and feedback.

If RCTs are being widely considered the new “gold standard” in development economics, then it must be said that the test by fire has only just begun. The grounds for critique of RCTs for social policy interventions go beyond the simple binary of whether it is an effective approach or not – to include methodological, philosophical, ethical and political questions. But having won the Nobel, there will be a lazy tendency among development practitioners to bestow RCTs their uncritical devotion, deification, and universal applicability.  RCT may thus become the nail, dangerously so even, in the law of the hammer.