Coronavirus exposes the problems and pitfalls of modelling
Models based on assumptions in the absence of data can be over-speculative and ‘open to gross over-interpretation’
The lessons to be learned from the coronavirus pandemic will keep scholars and university lecturers busy for decades to come. Chief among them is the value of modelling, and the fact that an uncritical reliance on their findings can lead you badly astray.
Take a recent model from Oxford University. It assessed how well different outbreak scenarios fitted the rise in coronavirus deaths in the UK and Italy. The most extreme UK scenario assumed that only a tiny fraction of people were at risk of serious illness. It also estimated that, as of last week, 68% of the population had been exposed to the virus. The study, which has not been published or peer-reviewed, unleashed a flurry of headlines declaring that coronavirus may have infected half of the people in Britain. That is 34 million people.
But as infectious disease modellers and public health experts, including the Oxford team themselves, have pointed out, the model used assumptions because there was no hard data.
No one knows what fraction of the public is at risk of serious illness. The study merely demonstrates how wildly different scenarios can produce the same tragic pattern of deaths, and emphasises that we urgently need serological testing for antibodies against the virus, to discover which world we are in.
Paul Klenerman, one of the Oxford researchers, called the 68% figure the most extreme result and explained that “there is another extreme which is that only a tiny proportion have been exposed”. The true figure, which is unknown, was likely somewhere in between, he said.
In other words, the number of people infected in Britain is either very large, very small, or middling. This may sound unhelpful, but that is precisely the point. “We need much more data about who has been exposed to inform policy,” Klenerman said.
The Oxford model was useful in emphasising the need for serological testing, but not for grasping the scale of the pandemic in the UK. As Prof James Wood, a researcher in infection dynamics at Cambridge University, put it: “The paper does substantially over-speculate and is open to gross over-interpretation by others.”
Devi Sridhar, a professor of global public health at Edinburgh University, said the Oxford study set out a hypothesis and nothing more. “It’s like me sitting here and putting into very fancy equations what would change if we had a vaccine tomorrow. I could model how a vaccine would save lives and you would see headlines reading ‘New vaccine is going to save lives.’ But we don’t have a vaccine.”
The modelling from Imperial College that underpinned the government’s belief that the nation could ride out the epidemic by letting the infection sweep through, creating “herd immunity” on the way, was more troubling.
The model, based on 13-year-old code for a long-feared influenza pandemic, assumed that the demand for intensive care units would be the same for both infections. Data from China soon showed this to be dangerously wrong, but the model was only updated when more data poured out of Italy, where intensive care was swiftly overwhelmed and deaths shot up.
Nor was that the only shortcoming of the Imperial model. It did not consider the impact of widespread rapid testing, contact tracing and isolation, which can be used in the early stages of an epidemic or in lockdown conditions to keep infections down to such an extent that when restrictions are lifted the virus should not rebound.
It is not a question of whether models are flawed but in which ways are they flawed, and models can still be enormously valuable if their shortcomings are appreciated. As with other sources of information, however, they should never be used alone.
“Models are a useful input among many when you are doing public policy, but you have to use triangulation. You have to look across different sources of information and not just rely on one. It’s more messy and complex than just saying ‘OK here’s a number’, but you get to a more accurate answer for our world,” Sridhar said.
Never have the words of the British statistician George Box rung truer than in this pandemic: “All models are wrong, but some are useful.”
