Harry Shannon
We all want to know how the pandemic will play out. Sorry, I don’t.
But some groups are doing mathematical modelling to try to predict what will happen. The statistician George Box famously noted that all models are wrong but some are useful. His point was that a model is an attempt to describe some aspect of the world in mathematical terms, and inevitably is not perfect. But if it’s close to reality, it can be useful. Models have to make various assumptions, and especially early on in the pandemic, much of the data being used was preliminary and needed to be interpreted very cautiously.
One model from Imperial College in the U.K. that got a lot of attention tried to estimate how many deaths would occur in the U.S. and the U.K. The numbers were huge. Over 2 million in the U.S. and over half a million in the U.K. But the model was estimating deaths if no measures were taken – no social/physical distancing, no shutdowns of gathering places, no school closures, etc. That simply wasn’t going to happen; even if governments did nothing, enough people would have changed their activities to make a difference. So when politicians claim credit for lower numbers than predicted, use a pinch of salt.
The Imperial College model also estimated how many lives would be saved with various combinations of interventions. With all of the measures most countries have in place, the reductions were dramatic. It was connected with ‘flattening the curve,’ trying to spread out the severe infections that need hospitalization so the health care system isn’t overwhelmed. I’m not going to cite any specific numbers. They wrote the report in mid-March when a lot of the data they used was uncertain, so they gave multiple forecasts based on different assumptions.
Another quote you have probably heard: ‘Insanity is doing the same thing over and over again and expecting different results’. It’s usually attributed to Albert Einstein, but that apparently is a myth. That doesn’t change the idea being expressed, and a good example of it is one of the models projecting the number of deaths predicted in the U.S. It’s important, because it’s the one that Donald Trump uses.
This model comes from the Institute for Health Metrics and Evaluation at the University of Washington in Seattle. They do the Global Burden of Diseases project, with many papers published in the prominent journal The Lancet. But their COVID-19 model has been criticized, both because it doesn’t do the modelling that epidemiologists use and because it keeps revising its numbers. Of course, revisions as new data come in are expected; but if the changes are substantial, it doesn’t speak well of the model.
And there have been plenty of changes. On April 5, IHME were predicting ‘81,766 deaths, with a range between 49,431 and 136,401.’ But just three days later, they revised that down: ‘A key forecasting model used by the White House has revised its prediction of COVID-19 deaths in the U.S., now estimating a peak of 60,415 by early August.’ In case you’re wondering, that number was exceeded on April 29 – more than three months ahead of ‘schedule’! Their latest forecast is 72,433 deaths by August 4.
So I thought I’d take a look at their short-term prediction of the number of deaths in the U.S. You can find IHME’s latest prediction here. (The numbers may well have been updated by the time you read this.) They show actual data up to April 27. For the next five days, the daily predicted deaths are 2,170, 1,382, 1,266, 1,155, and 1,049 for a total of 7,022. In fact, the numbers were 2,470, 2,390, 2,201, 1,897, and 1,691 – which add to 10,649. That’s 52% more than IHME expected and it’s only looking at the first few days of their forecast.
Now IHME could argue that they provide a range, which shows the margin of error. But the range is so wide that it’s essentially useless for policy. It also doesn’t make sense on the face of it. The lower limit of their prediction was that the total number of deaths would stay below 60,000. Yet the actual data reported by IHME to April 27 showed there were already 55,891 deaths. Only a precipitous drop in the mortality rate could possibly have led to less than 60,000, which simply wasn’t credible.
What model can you trust then? Well, I don’t have a simple answer. You can read about what to look for in a model here. The article points out how tricky it is to get it (close to) right. So be careful which models and predictions you pay attention to. I certainly don’t trust the IHME forecasts. If they keep getting it wrong over and over again, you’d think they’d admit it’s insane to keep going with their seriously flawed model.