Proving impact

DMI and LSHTM are running a five-year cluster-randomised controlled trial in Burkina Faso, funded by the Wellcome Trust and Planet Wheeler Foundation, to test the predictions of our model in a real-life setting.

The randomised controlled trial

The trial uses radio broadcasts to change behaviours in order to improve child survival, covering a range of health issues, and is the largest, most rigorous evaluation ever conducted of a mass media intervention. It will generate a substantial body of further evidence for the impact of mass media campaigns on a range of behaviours, including those linked to demand for healthcare services, and on child mortality.

The trial involves fourteen geographical areas that were randomised and equally divided into an intervention group and a control group. Messages are being broadcast for 2.5 years in the seven geographical areas of the intervention group, but not in the control group. 

A controlled trial using radio would not normally work, due to the risk that people in 'control' areas would listen to radio stations broadcasting from 'intervention' areas. However, Burkina Faso has a very localised, radio-dominated media environment, so we can use local FM radio stations to broadcast our messages to intervention areas (blue) without 'leaking' into control areas (red).


The radio campaign

For 30 months we are broadcasting 60-second advertisements at least 10 times per day on seven radio stations (one in each intervention zone), in six languages. In addition, we are broadcasting two hours per night, five nights per week on each station. This represents a total of 70 hours per week of live radio.

This would be logistically almost impossible to do using a soap opera format, for example, given the six languages involved. We needed to devise a format that is cheap, that can be broadcast daily, that can be produced ‘live’ (which costs a fraction of pre-produced radio), and yet can be controlled centrally. We have created a system of self-contained drama modules that are written in French in the capital city, emailed to our partner radio stations, and improvised live by actors on location in their own language within their two-hour shows. This works well in a fragmented media environment, which is becoming the norm in most developing countries.

We are not paying airtime fees to our partner radio stations in Burkina Faso; instead, we provide on-the-job training and produce the live programmes together. The stations’ incentive is that they thrive in a competitive media market.

We implemented a system of grassroots recruitment, leafleting university campuses, meeting places and bars, inside and outside the capital. We invited interested people to public meetings and challenged them to write a script. We received over 600 scripts, interviewed 80 people and hired 13 as scriptwriters – who came from a diverse range of previous employment, ranging from teachers to a security guard. Every week we conduct creative training workshops for all staff. We have an in-house research team which conducts monthly formative, pretesting and feedback research, with one scriptwriter participating in every research trip.

Read an article in Global Health, Science & Practice about the implementation of the RCT


Example radio outputs

Malaria radio spot Hygiene radio drama

One-minute spots

Interactive dramas


The results so far

The evaluation, led by LSHTM and Centre Muraz, includes a baseline mortality survey of 50,000 children under 5 years (with a two-year recall period), and a baseline behaviour survey of 5,000 households, before radio broadcasts began in March 2012. Broadcasting finished in January 2015, and full results, based on an endline mortality survey of 100,000 children (with a one-year recall period), will be available in 2016.

We have published our midline results on behaviour change (based on a survey of 5,000 households). They show that behaviours in the intervention zones have all improved (by between 4.7pp and 25.3pp). When changes in the control zones are subtracted from these results, the adjusted difference in difference (DiD) is substantial (at least 5pp) for 7 out of 10 key behaviours, and the average adjusted DiD is 7.9pp. This is the first randomised controlled trial to demonstrate that mass media can cause behaviour change.

In addition, an economic evaluation, led by Jo Borghi and Anne Mills at LSHTM, will assess whether the intervention represents 'value for money', measuring the cost per child death averted and per DALY averted. We will also model the likely cost-effectiveness of implementing the campaign at national scale in Burkina Faso and elsewhere in Africa.


Implications for future campaigns

If similar five-year campaigns are implemented in 10 African countries, our model predicts that one million lives should be saved, more cost-effectively than any other health intervention. This process would, for the first time, place mass media campaigns in the mainstream of public health interventions, based on robust scientific evidence. We are currently designing health behaviour change campaigns in these countries and are looking for partners and funders through our Media Million Lives initiative.


The ongoing evidence debate

There is a lively discussion taking place within the behaviour change community about the most effective (and feasible) ways to produce robust evidence of impact. Most organisations agree that it is a pressing priority to generate more high-quality evidence of impact, because too much of the existing evidence base is drawn from the 'grey literature' (internal reports produced by the implementing organisations). However, there is a wide range of views about the best strategies for generating this evidence.

One school of thought, exemplified by a recent BBC Media Action working paper on mass media evidence, is that randomised controlled trials are not feasible or practical when it comes to evaluating media campaigns. While we agree that RCTs are not feasible or affordable in every situation, they remain the 'gold standard' evaluation design for demonstrating and attributing the effects of interventions. We strongly believe that the behaviour change community needs to produce evidence of the same scientific standard and rigour as that which is produced for other public health interventions. Only this will allow policymakers to make informed decisions about how best to spend limited healthcare budgets in order to deliver the maximum return on investment. At present, two fundamental questions remain unanswered: firstly, are media campaigns effective in changing behaviours, and secondly, do they represent value for money? It is our hope that our RCT in Burkina Faso will provide robust answers to both of these questions.


Film: The Science (Part Two)

This films describes how we are testing our model through the randomised controlled trial in Burkina Faso, interviewing some of the key players.