Sep 17, 2018

The Bad Science of Estimating Hurricane Deaths

How many residents of Puerto Rico died because of Hurricane Maria?

It seems like a simple question, right? Not in the world of the Opposition Media vs. President Trump, "fake news" and alternative facts. In that world, you get to pick from a range of 6 dead to … 8,498.

Below is a list of estimates I gathered from various news sources (in descending order):

8,498-793 according to the New England Journal of Medicine (NEJM)
4,465 according to activists using the NEJM mid-point estimate
3,290-2,658 according to George Washington University
2,975 according to the Governor of Puerto Rico
1,427 according to a December 2017 report to Congress
1,272-1,006 according to the Journal of the American Medical Association (JAMA)
1,085 according to a December 2017 analysis by two scientists for Vox
1,052 according to a December 2017 analysis by The New York Times
499 according to a November 2017 CNN investigation
81 according to an October 2017 Vox investigation
64 according to the original official death toll
57 based on the number of names that have been released
18-6 when President Trump visited the island

I gathered these figures because of a September 14 CNN article headlined: "Trump falsely claims nearly 3,000 Americans in Puerto Rico 'did not die.'" Since the anniversary of 9/11 had just passed, I needed to understand how Hurricane Maria could possibly be as big of a tragedy. I read the article and immediately noticed a few issues. For instance, the "nearly 3,000" figure was said to account for "Puerto Ricans who succumbed to the stifling heat and other after effects of the storm and had not been previously counted in official figures."

As I did my research and found the other estimates, I noticed similar caveats. For instance, the NEJM report disclaims:
"In the United States, death certificates are the primary source of mortality statistics, and in most jurisdictions, death can be attributed to disasters only by medical examiners. Survey-based studies can therefore provide important complementary population-level metrics in the wake of natural disasters, despite inherent limitations associated with the nature of participant-reported data, recall bias, nonresponse bias, and survivor bias."
That's a lot of biases. More important, the source of mortality statistics turns out to be the real key to understanding why these wildly varying death-toll estimates exist.

Around the same time as the CNN article came out, The New York Times updated a June 2, 2018 article titled, "Puerto Rico: How Do We Know 3,000 People Died as a Result of Hurricane Maria?" In trying to defend all of the varying estimates the paper had published and convince readers it wasn't "bad science," the newspaper ended up demonstrating how little hurricane mortality estimates have to do with anything like science. The article offers a fascinating look at how numbers can change without context and be manipulated to fit narratives.

Some highlights:
"George Washington researchers said they found that doctors in Puerto Rico at the time of the storm were not aware of new guidelines from the federal Centers for Disease Control and Prevention, released the month after the hurricane, which recommend that doctors also consider a natural disaster’s indirect impacts in assessing how to tally deaths."
A month before Hurricane Maria, the CDC apparently changed the standard for measuring hurricane deaths. The standard went from counting death certificates certified by medical examiners (see above) to also counting “indirect impacts.” How does one define and measure such a vague phrase? Here's how the Times did it:
"To obtain our figure of 1,052, we compared the number of deaths for each day in 2017 with the average of the number of deaths for the same days in 2015 and 2016. The figures came from the Puerto Rican government, which provided us with tables showing the number of deaths per day and deaths broken down by cause. The 2017 numbers were preliminary, so we limited our analysis to September and October." 
"In September and October of 2017, 197 people died of sepsis — a complication of severe infection. That was a 55 percent increase from the average for the same months in 2015 and 2016. Those changes could be explained by delayed medical treatment or poor conditions in homes and hospitals.
"The number of diabetes deaths in September and October 2017, at 666, was 46 percent higher than the average for the same period in the two previous years. Many people with diabetes had difficulty keeping insulin refrigerated, and some had trouble maintaining special diets.
"Deaths from chronic respiratory diseases and Alzheimer’s also appeared to be increased. As for suicide deaths, 49 people took their lives in September and October of 2017, whereas in the same months of 2015 and 2016, an average of 33 people died by suicide."
At first, I was tempted to dismiss this as a typical abuse of statistics by journalists. But the NEJM method sounds even worse:
"Researchers visited more than 3,000 residences across the island and interviewed their occupants, asking whether anyone in their households had died, and whether the storm and its aftermath might have contributed. Residents reported that 38 people living in their households had died between Sept. 20, 2017, when Hurricane Maria struck, and the end of that year.
"That toll, converted into a mortality rate, was extrapolated to the larger population and compared with official statistics from the same period in 2016. Researchers arrived at an estimate of roughly 4,600."
You read that right: To get their estimate of 4,600 Puerto Ricans killed by Hurricane Maria (a number that incidentally became a blaring headline and activist rallying cry), researches used a sample size of 38 that they then “converted” “extrapolated” and “compared” to the previous year.

"Was it bad science?" the Times asks, and then it answers: "Experts who study the health impacts of natural disasters say no." These are probably the same journalists who wonder why the public has lost faith in science and no longer listens to experts.