- SURVEYS THAT CHANGED THE WORLD It’s not unusual for political opinion polls to cause some controversy. But that controversy is usually local, or at best national.
SUPER SIMPLE SURVEY BLOG - SURVEYS THAT CHANGED THE WORLD It’s not unusual for research surveys, particularly political opinion polls, to cause some controversy. Every now and then a survey conducted at the height of a political campaign will raise a stir. But the controversy is usually local, or at best national. Which is why the British medical journal The Lancet deserves a place in our series of Surveys That Changed the World. They pulled off the rare feat of publishing a survey that provoked international debate and heated denials - and a flood of accusations - from the US Government, the British Government, the United Nations, and a whole bunch of statisticians, journalists, and doctors. What’s more, this didn’t happen just once. It happened twice. Putting the US and British Governments on the Defensive The year is 2004. It’s been a year since a US-led coalition invaded Iraq. Saddam Hussein, the Iraqi dictator, had already been deposed, but lots of Iraqi nationalists and Islamic militants were putting up quite the resistance. Saddam’s Weapons of Mass Destruction, the official reason why America had invaded, were starting to feel like a mirage, but the Americans were now in it too deep to just walk away. It was already looking like it was going to be a long war. So far casualty figures had been modest, but they were increasing steadily, and each death was quickly broadcasted to the world by the international media. Still, no one could tell exactly how many Iraqis had been killed as a victim of the war. The invading forces had their own estimates. So too did the United Nations, various international NGOs, and what was left of the Iraqi Health Ministry. But all those figures varied widely. So, The Lancet, one of the oldest professional journals in the world, contracted John Hopkins University and the Iraqi Al-Mustansiriya University to jointly conduct a survey on the Iraqi death toll. The survey’s team leader, American Les Roberts, was a veteran in this sort of work, having previously done a similar survey in Congo. The results of that particular survey had been adopted by the UN Security Council and the US State Department. How the Lancet Survey Was Conducted The Lancet survey on Iraq was soon conducted. The survey team used the cluster sampling method: they divided the country into 33 clusters covering 11 of Iraq’s 18 provinces, and took a sample of 30 households in each cluster. The data was collected by Iraqi medical doctors who had experience in surveys, while the oversight and data analysis was done by the John Hopkins University team. The final results were then compared to the mortality rate in the country in the period of 14.6 months before the invasion, so they could calculate the increase of the death rate since the invasion. This was the same survey technique that they had used in Congo, as well as in other similar mortality surveys in Darfur and Bosnia. This method was also preferred because it sidestepped the risk of undercounting that would have occurred if only official hospital records were used. The survey results quickly caused an international uproar. The Lancet reported that in the 18 months since the invasion, about 98,000 more Iraqis had died than would otherwise have died if the USA had stayed home. Violence was widespread in 15 of the 33 clusters, and the violence was mainly attributed to coalition forces, particularly air strikes. All in all, the war had raised the Iraqi death rate by 58% - and this was only because the deaths in Fallujah City, a cluster where fighting had caused more casualties than anywhere else in Iraq, had been excluded from the final results since it was considered to be an outlier. With Fallujah factored in, the death rate was a staggering 150% above pre-invasion rates. The Controversy Surrounding the Lancet Survey Results American and British governments were outraged. So too were the United Nations and the Iraqi Body Count Project, a British based international research NGO that was at the time considered to be the authority on the Iraqi death toll. A large cross section of journalists, statisticians, and epidemiologist opposed The Lancet and, increasingly, a surprised Roberts. The problem was that the Lancet figures were a lot higher than the estimates of the US and British government, as well as those of the United Nations and the Iraqi Ministry of Health. The new figures were also at odds with the results of other household surveys on Iraq, such as the Iraqi Family Health Survey and the Iraqi Living Conditions Survey. Officially, the US and British government disputed the figures because they were higher than those of the Iraqi Ministry and the UN. Of course, their real angst was with the fact that the figures made their increasingly unpopular war more politically and militarily indefensible. But some journalist, statisticians, and epidemiologist had more specific issues with the survey. Some claimed the survey teams did not visit some designated clusters or chose households that were easier to reach. The survey team flatly denied that claim. Others claimed the Lancet survey had used misleadingly low pre-invasion death rates, to which others retorted that the rates were actually misleadingly too high since the country had been under international sanction years before the invasion. Others felt that the survey’s reported response rate of 99.5% was suspicious. Also, some even accused Roberts of rushing out the survey results so as to influence the American presidential campaigns, to which he pointed out that all of his 20 previous surveys had in fact been released faster than this one. The Impact of the Lancet Survey on Iraq The Lancet not only stood by its survey results, but also – just when the controversy was fading away two years later – went on to conduct a follow-up survey. This time, they reported that over half a million more Iraqis had died since the previous survey, bringing the total to 654.965. About 500 Iraqis were killed by violence every day, and about 2.5% of the entire Iraqi population had already been killed by the effects of the war. These new figures were more than 20 time the estimates given by President Bush in a speech a few months before, and more than 10 times the estimates of the Iraqi Body Count Project. Predictably, again there was an uproar. Even today, the results of the two surveys are controversial. Even though The Lancet’s surveys are only one of two surveys on that topic that have been subjected to peer-review before publication, there are still both opponents and proponents. Some journalists, governments, and statisticians have reacted with criticism and disbelief, whereas other journalists, governments, and statisticians are supporting the outcomes. So what was the impact of the Lancet surveys? We will probably never know for sure. But what we do know for sure, is that the two surveys had a major part in further puncturing the public perception of the war in Iraq by starting international discussions about its true cost in human lives. And that has most likely affected American politics and saved lives by hastening the end of the invasion.