Friday, October 29, 2004

Pick a number, any number

The Lancet stepped in with its bid for "October Surprise" in the US elections with a study that breathlessly announces that "100000 excess deaths, or more have happened since the 2003 invasion of Iraq" and "most individuals reportedly killed by coalition forces were women and children". Damning claims, if true. But is it?

The original study is available here and here. Major newspapers are already running stories based on it, for example the Washington Post.

A quick summary of the data:

The sample is of ~7500 residents in 33 clusters of 30 households in 11 out of the 18 provinces. There are 21 recorded violent deaths over the 18 months after the invasion: 4 children, 13 men, 2 women, and 2 elderly people. Of these, 9 are due to the coalition military, 7 are criminal murders, 2 terrorist attacks, 2 unknown and 1 by the former regime. Of the deaths due to the coalition military, 2 were known accidents. In this summary the data from Falluja are excluded for reasons which are described in the study itself.

Analysis:

The sex ratio suggests that the excess men and possibly some of the under-15 children were combatants. The study makes no effort to count the number of combatants. The mortality for men would be higher in any case because they travel more and engage in more dangerous activities. If the risk from all non-combat-related causes is assumed to be twice higher for males (comparable to the ratio of accident-related deaths between males and females), out of the 21 violent deaths in this study, approximately 6 were combatant men and 1 was a combatant child under 15. This would put the countrywide total of combatants killed at 25,000 - not an unreasonable number, considering the number of military personnel killed during the invasion itself has been estimated to be 9,000 (PDA) and that number would be included in this study. Under these assumptions, the number of noncombatants killed by the coalition military in this sample would be 1 or 2, in other words too small to be meaningfully measured (but note that the study mentions two known accidental deaths caused by coalition military). The corresponding number of noncombatants killed countrywide would be under six thousand. If the gender risk ratio estimate above is correct, 80% of those killed by the coalition military are combatant men.

As measured by this study, the criminal murder rate is 60 per 100,000 per year, which is in the same ballpark as other estimates (like this, or this). That should be compared to 40 per 100,000 in Washington, DC and 80 in Compton, Los Angeles. The combatant death rate is the same as the murder rate, 60 per 100,000 per year.

Each death recorded in the study is a proxy for roughly 3500 deaths countrywide. Since the numbers we are trying to measure are on the same order of magnitude, it is obvious that the sample is too small. A decent sample size would be five to ten times larger.

Violent deaths were recorded in only 15 out of the 33 clusters. That means that as seen by this study, the violent death rate in most of the country (containing a population of 15 million) is effectively zero. Of course, it is impossible to know which half, since the study does not give us the geographic distribution of violent deaths by governorate. It is possible that they are concentrated in only a couple of governorates (perhaps Baghdad and two or three others?).

The geographic distribution of the data in general is very suspect. A full geographical breakdown by cause and province is not provided, only overall mortality (all causes) by province. Some of the more peaceful provinces (Irbil, Basra, Al Muthanna) were not sampled at all. Some of the provinces which saw heavy fighting show no change in the mortality rate (Ninawa, Karbala) while other provinces which saw little fighting show large increases (Wasit, Missan). This discrepancy casts doubt on the reliability of the sampling technique. One province (Al Sulaymaniya) shows a 3-fold drop. It is impossible to know what the extent of the sampling bias was without knowing which communities were sampled, and comparing them to known areas of heavy fighting.

Conclusion:

Based on the data in the study, we can reasonably estimate (in other words guess) that 23000 +- 9000 combatants and 7000 +- 9000 non-combatant civilians were killed by coalition military forces. This is comparable to other estimates, within the large margin of error.

Crime is obviously a huge problem in Iraq, but criminals and insurgents in many cases work together (for example here and here), and of course insurgents help criminals by specifically targetting police and creating an environment in which law enforcement is impossible.

What about those claims of "mostly women and children"? Excluding the data from Falluja (which is suspect, a subject on which I'll write more in a bit), the truth is the exact opposite: most of those killed (and almost all of those killed by coalition forces) are men of fighting age.

What about the 100,000 overall deaths number? Let's quote the study itself...

"The crude mortality rate [pre-war] was 5.0 per 1000 people per year (95% CI [confidence interval] 3.7-6.3) ... the post-attack mortality is 7.9 per 1000 people per year (95% CI [confidence interval] 5.6-10.2)" and "the risk of death is 1.5-fold (1.1 [to] 2.3) higher after the invasion".

In other words, anywhere between a 10% and 130% increase. This is a damned large margin of error. The headline claim of 100,000 would have you believe that mortality doubled in the entire country. The study certainly proves no such thing.

Next: Why did this study get released right now? Are the Falluja data reliable? Is this scientific research or a psy-op? What are the political goals of the people behind the study - in their own words?


UPDATE: Readers have pointed out a statistical error in the next-to-last paragraph which I have now corrected. A rushed response to a rushed study... let this be a lesson to anyone else who rushes into publication with little or no peer review.