Models that predict NC COVID-19 deaths are imperfect. Should we pay attention to them?
Early in the coronavirus outbreak in the United States people wanted someone to tell them how bad it would get. Along came the Institute for Health Metrics and Evaluation at the University of Washington with answers.
IHME began making state-by-state forecasts in March that provided seemingly precise predictions of how many hospital beds would be needed for COVID-19 patients and how many people would ultimately die of the disease. News outlets across the country offered up these forecasts with the certainty of a weather report, even if they couldn’t explain the calculations that went into them.
Since then, those forecasts have lurched up and down, as the researchers tinkered with their models and new information about the virus and the efforts to curb its spread emerged.
In late March, IHME predicted 2,411 North Carolinians would succumb to COVID-19 by August, then revised the figure to less than 500 by the end of the first week of April. On May 10, it was back up to 1,190 deaths, then nearly quadrupled to 4,413 on May 12.
Now IHME puts the death toll in North Carolina by August at about 2,500, more than triple the current number.
The IHME highlights both the promise and the limitations of computer models that attempt to predict something as complex and ever-changing as the spread of a new infectious disease. Hospitals and public health officials use models to craft strategies to try to curb the outbreak or to decide how many hospital beds to make available for the sick.
“People are going to decide that based on their gut or their Twitter feed or their sister’s boyfriend’s blog,” says Aaron McKethan, CEO of health data science company NoviSci and a member of a team doing coronavirus modeling in North Carolina. “Models are better than those things, if handled with humility.”
By humility, McKethan means acknowledging that models are imperfect, and the farther out they reach the less certain they become. As with predicting a hurricane, he says, it’s hard to say what might happen with COVID-19 two months from now.
“Americans like certainty, and we just don’t have it right now,” he said. “It’s very easy to be disappointed by what models can tell you.”
More models better than one
IHME is now among more than a dozen research groups at government institutes and universities such as Columbia, MIT and UCLA predicting how many people will die from COVID-19. The Centers for Disease Control and Prevention gathers the forecasts and creates composite figures for each state.
As of Thursday, the “ensemble” prediction was that 1,250 people will die of COVID-19 in North Carolina by June 13. That’s more than 23 people a day on average between now and then, up from an average of 16 a day over the last month.
The 14 models that went into that forecast produced a range of predictions, but the consensus is that the death rate in North Carolina will rise, though not necessarily spike, as the state reopens for business in coming weeks.
Such short-term predictions drawn from several models are better than one, said Kim Powers, an epidemiologist at UNC Chapel Hill’s Gillings School of Global Public Health.
“I think the CDC ensemble modeling approach is a really nice step in the right direction, where we’re not relying on one model that could be quite flawed,” Powers said. “We’re looking at a range of different models put together by a range of different research groups, most of which have a lot of experience with this.”
Gov. Roy Cooper and the state Department of Health and Human Services didn’t use the models cited by the CDC in further easing restrictions on businesses and public gatherings in North Carolina starting this weekend. Instead, they set several benchmarks based on actual trends, such as the number of people hospitalized with COVID-19 and the percentage of people tested who have the virus.
All models are subject to limitations and uncertainties, said DHHS spokeswoman Kelly Haight Connor.
“Although the magnitudes of these forecasts are different, all suggest that the number of deaths will continue to rise over the coming weeks,” Connor wrote in an email. “This is not directly used in our decision-making but is an important reminder of the importance of staying vigilant and taking actions to control spread of the virus.”
Coronavirus models built on assumptions
The models that seek to predict the future of the coronavirus outbreak essentially ask a series of if/then questions, Powers said. For example, if a state orders businesses closed and people to stay at home, then how many people will contract the virus, how many will require hospitalization and how many will die? If stores reopen but restaurants don’t, then how will those numbers change?
The questions and assumptions vary from model to model, and then must be adjusted as conditions change. A stay-at-home order limited physical contact, but that’s changing as businesses reopen and people begin mixing again.
The mathematics may be precise, but the assumptions depend on human behavior, which is anything but. What if everyone wears masks and stays six feet from each other? What if only 75% do, or 50%? How will that change the outcome? Modelers must also make assumptions about the availability of testing and the effectiveness of tracing and tracking people who have come in contact with someone who tests positive, Powers said.
“It’s a lot of moving parts,” she said.
All of this is layered on top of a disease that scientists still don’t fully understand. Early in the outbreak, scientists assumed coronavirus would behave like SARS, a similar respiratory illness where people were only contagious if they were sick. We’ve since learned that people can carry coronavirus and pass it along to others even if they don’t show any symptoms. That changes the models.
“We still do not understand some very basic things about this stuff — the disease, the transmission, the virus,” said Pia MacDonald, senior epidemiologist at RTI International and an adjunct professor at UNC. “As long as that’s the case, there’s a lot of uncertainty here.”
MacDonald said people don’t really need models to see the direction the outbreak is taking in North Carolina. The numbers reported by the state Division of Public Health show the daily number of new cases increasing.
“That means there’s more people in North Carolina getting this disease, and the more people who get the disease, the more people will have bad outcomes,” MacDonald said. “That’s regardless of the models. That’s me just looking at the data right now and where the trends are going.”
Modelers and epidemiologists say people shouldn’t get hung up on specific numbers produced by the models but should look at the signs and directions the numbers suggest. Right now, MacDonald says, they point to the need to continue wearing masks in public, maintaining physical distance, staying home if you feel sick and washing your hands a lot.
“It’s basic public health, but it’s a lot to ask of people and it’s hard to do,” she said. “But doing all of those things, we will surely see that we will start getting these trends to go in the right direction.”
