UPDATED: Fact-Checking an Article on Saving Mothers Giving Life in the Christian Science Monitor

1/22 Update: I received a response from the author, Howard LaFranchi, late last week, but haven’t had a working computer until now. He made a number of updates to the article, as he notes in his email to me (which I’m posting in full, with his blessing):

Dear Michael: Please find below a link to the corrected version of my maternal mortality story with Uganda now featured in the lede instead of Zambia.

You are indeed correct that Zambia did not implement a transport voucher program under the Saving Mothers program. My error was to conflate information I was provided on the challenges and programs in the two countries. That has now been corrected.
Your other criticisms, however, pertain to information and statistics provided by USAID, CDC, and the ministries of the two countries involved. I don’t agree with your observations, but you might want to take them up with the institutions you question.
I appreciate that a reader with your knowledge took the time to call my attention to the errors in the piece. On the other hand, I have to say that I find it a bit disheartening that someone involved in addressing the issue of maternal mortality would advocate “removal” from a general-interest site of an article that deliverers some good news on the issue.
In any case, thanks for your interest.
Best wishes, Howard

 I have a few thoughts on this, but will likely write a separate post (or update this post) later this week outlining them rather than doing so now. I very much appreciate Mr. LaFranchi allowing me to post his response.

It’s rare that I find an article (published in a prominent outlet) that is so inaccurate I feel compelled to spend some time researching and writing a rebuttal. But it happens.

This article is from the Christian Science Monitor, a great publication. Unfortunately, almost everything – the title, the statistics, and lede – is inaccurate or misleading. It needs to be modified or removed.

A bit of background: a USAID-backed partnership, Saving Mothers Giving Life (SMGL), is working to reduce maternal mortality in western Uganda and parts of Zambia, with possible scale-up districts/countries to follow. It’s a big project – $280 million over ten years (at least), with USAID, the Norwegion Ministry of Foreign Affairs, and Merck for Mothers working in tandem with the Ugandan and Zambian Ministries of Health. The partnership is implementing a number of health-systems-strengthening-related interventions that it hopes will ultimately lead to fewer maternal deaths in the area.

Let’s go through the article, paragraph by paragraph. Starting with the title (likely from the editor, not the author):

How a simple travel voucher is saving the lives of pregnant African women

This is a misleading title, for a few reasons. Even a travel voucher scheme is really complex to implement in low-resource settings like Uganda; getting the amounts right, sensitizing the community, monitoring to ensure it’s working properly – all difficult. Furthermore, the title gives the impression that SMGL is a travel voucher scheme only, when vouchers are one small piece.

While we’re on the subject, the author later notes that the project:

…has a lot more to it than rides on the motorized cycles used in those rural areas. But the availability of transportation is one key reason the program has shown impressive results in a short time, experts say

That’s quite the claim for an anonymous “experts say,” and neither the SMGL 2013 Annual Report nor the Columbia University external evaluation of SMGL analyzed specific interventions for efficacy.

Continuing, the lede:

In rural Zambia in southern Africa, a pregnant woman faces on average a five-hour walk to reach prenatal care – or a facility equipped to handle complications in birth

This could very well be true, but, like the rest of the article, there’s no linked source. I did a quick search and didn’t find any time study or survey on pregnant women in Zambia, though that obviously doesn’t mean this statement wrong – just not simple to source.

It is is a useful statistic to have; it’s visceral and emotional, and can help readers empathize with women going through pregnancy in difficult settings. But it’s useful only if it’s accurate.

Compounding the above criticism, here’s the next paragraph:

Enter a very simple idea – transport vouchers for pregnant women. Over the past year and a half of a new US-initiated program for addressing maternal mortality, Zambia began providing thousands of vouchers for women to be able to access pre-natal care and properly equipped birth facilities

This is flat-out wrong; SMGL doesn’t provide transportation vouchers as part of its program in Zambia. From the external evaluation of SMGL (p. 25):

“…over half the women (58%) who had heard of SMGL in Uganda reported familiarity with transportation vouchers compared to only 3% of women in Zambia. The contrast is unsurprising, as the SMGL program in Zambia did not provide transportation vouchers” (emphasis mine)

Sloppy journalism, plain and simple.

Here’s the next paragraph (I’ve also included a bit about Uganda from a later paragraph for clarity):

Zambia records about 2,600 maternal deaths every year, giving it a maternal mortality ratio of 440 per 100,000 live births. (In the US, the number is closer to 12 per 100,000)… [later, regarding Uganda – MM] … the maternal mortality ratio in the four districts fell from 452 to 316 per 100,000 births. Uganda’s national average is 438.”

The Zambian figure, 440 per 100,000 live births, is for 2010, and comes from the World Health Organization (WHO). For the same period and from the same source, the Maternal Mortality Ratio is 21 per 100,000 in the United States, not 12. (This is probably just a typo that was not corrected by the editor.)

Uganda’s national average is not 438, as best I can tell; according to the same source linked above from the WHO, it’s 310 per 100,000 live births in 2010. The author may have a separate source for his figure, but unless he has some reason to believe the WHO analysis is wrong for Uganda, he should be comparing apples to apples.

The author continues:

But in the maternal care facilities in the four Zambia districts participating in the Saving Mothers Giving Life (SMGL) program, the maternal mortality ratio fell by 35 percent.

This isn’t technically wrong, according to the SMGL 2013 Annual Report. But this number tells us a lot less than we would think – something that the average reader may not immediately intuit. It’s the job of a journalist to help walk the reader through statistics like this.

At the most basic level, this figure fails to tell us what would have happened in that facility were it not for SMGL; it doesn’t give a counterfactual. Maybe a similar, non-SMGL district also saw a 35% reduction in the Maternal Mortality Ratio; perhaps a similar district saw its Maternal Mortality Ratio double. We just don’t have the context, and insinuating this reduction is both good and a result of SMGL isn’t exactly appropriate.

Even if the SMGL Annual Report listed results from comparison districts, we still need to take the figure with some skepticism. SMGL wasn’t set up as a randomized controlled trial – the leadership hand-picked which districts it planned to work in – so we can’t say for sure that SMGL’s results in these districts could be expected elsewhere, nor can we say that SMGL caused the MMR to drop by 35%.

There were comparison districts in both countries for the external evaluation, but they were not random and not selected at the start of the intervention, so, in the authors’ words, “we can compare performance of SMGL versus comparison districts as of May 2013, but we cannot assess whether there was change in the SMGL districts during the year of program implementation.”

The bottom line is this: readers need more information before they can understand what “fell by 35%” actually means, and it’s a journalists job to find that information and walk them through it.

Finally, after a quote from Dr. Rajiv Shah, the administrator for USAID, the article continues:

The Saving Mothers initial program cost $20 million – or about $7 per birth under the program in Zambia, and about $13 per birth in Uganda.

This is going to get wonky, so I won’t bury the lede: the above figures make no sense to me, and I have no idea where they came from. To start, $20 million is the cost of the pilot program for both Uganda ($10,505,255), and Zambia ($8,144,510), some of which was long-term capital investment (e.g., equipment, construction) and some of which was short-term/admin expense (e.g., salaries, transportation vouchers) over a 17- and 20-month period, respectively.

Let’s look at figures for Uganda:

  • According to the SMGL 2013 Annual Report, $10,505,255 was allocated to Uganda, of which 29% was for capital expenditures which were expected to last for five years. So, taking .71*$spent+ (.20*capex) should give us a very conservative figure (i.e., erring low) to start with: $8,068,036 ($7,458,731 + .2*(.29*10,505,255))). At $13 per birth, there would be 620,619 births, according to the statistics the CSM author reports
  • Quick gut check: is 620,619 births reasonable? The same report lists Uganda as having 1.5 million births per year, or about 2.125 million every 17 months (the time period listed for the disbursement in Uganda). Does it make sense to have 29% of the country’s births come from four western districts (Uganda has 111 districts total)? Probably not
  • Furthermore, the external evaluation lists  78,400 “deliveries, annual” in the four Uganda districts in 2011. Correcting for number of months and assuming this only is facility deliveries (to err on the side of caution), the estimated total number of births in the four districts during the period would be about 198,333 – less than one third the figure I got by reverse-engineering the author’s data.

I could do the same with the Zambian figures, but it’s beside the point. The author should back up his assertion with data and sources; right now, it’s as if these figures were just pulled from the ether.

But. Let’s assume, for the sake of argument, that the author’s figures hold up. What do they tell us? With the right additional information, it’s possible we could use them to best prioritize the next marginal dollar of investment; it would be better, one could argue, to invest that dollar in an area with a high birth rate than an one with a low birth rate, all else equal.

Or maybe not. Why not ask if its possible to invest in family planning services that will drive down the number of births in an area? That wouldn’t be captured in this metric (it would actually make the cost per birth higher), but it could be a valuable way to reduce the number of maternal and child deaths.

All of which is to say: this metric doesn’t tell us anything meaningful.

The author then spends the remainder of the article discussing maternal mortality more generally.

The point of this isn’t to castigate one specific writer, or even one specific publication – it’s to highlight the need for quality journalism on global health/development topics. If it’s worth writing about (and it is!) it’s worth writing about well.

I’ve emailed the author and will post a response if I receive one.

 

Leave a Reply

Your email address will not be published. Required fields are marked *