Building Archimedes - a Q&A with 'Dr. Data' - ' Testing for Accuracy '

Testing for Accuracy

How do you know the model is accurate?

We test the accuracy of the model by simulating a wide range of epidemiological studies and clinical trials. The studies are chosen by independent advisory committees. For each validation exercise, we create a "virtual trial" by repeating the steps taken in the real trial, and then compare the outcomes seen in the virtual trial with those that occurred in the real trial.

More specifically, we begin by having the model create a large virtual population that contains a broad spectrum of ages, sexes, race/ethnicities, characteristics, behaviors and diseases. Then, to simulate a particular clinical trial we begin with the initial description of the trial, focusing in particular on the inclusion and exclusion criteria, the treatment protocols and the follow-up protocols. We have the model search the large simulated population to identify people who meet the entry criteria for the trial. After confirming that their characteristics (e.g., age, sex, history of previous conditions and/or treatments, lab results) match the distribution of characteristics published for the real trial, we randomize the simulated people into the number of groups used in the real trial. We have simulated providers give the people in each group the designated treatments, using the same protocols described for the trial. We include any breaches in either provider or patient compliance that are described for the trial. We then let the people's physiologies function, subject to whatever treatments they are receiving. As the simulated time passes in the model, the simulated providers follow each patient with simulated appointments and tests at the same intervals used in the real trial. Between scheduled visits, simulated patients can also develop symptoms, seek care, make appointments, have visits, be tested, diagnosed and treated, just as might happen in a real trial. We then record the results at the time intervals used in the real trials, and process them using the methods described for the real trial. The results of the virtual trial can then be compared with the results of the actual trial.

We've done this 74 times now, and the results from the virtual world of the model match the results seen in the real trial well within sampling margins for 70 of those exercises. The mismatches were either very close misses (e.g., p = 0.04), which you would expect from statistical theory, or have good explanations that do not call for any changes in the model. Including the mismatches, the correlation between the results that occurred in the model and the results that occurred in reality is 0.995, on a scale of 0 to 1, where 1 is perfect. If we consider only trials that were never used to develop the model – which is about half the 74 exercises – the correlation is still 0.99. Frankly it's almost spooky. These equations really seem to be representing what Mother Nature is doing.

Give me an example of a match.

The Diabetes Prevention Program was published in February of last year. It compared two treatments (a drug called Metformin and an intensive lifestyle program) and a placebo, in preventing or postponing the onset of diabetes in a group of people who were at very high risk of developing diabetes. Before the results were published we were asked to simulate that trial. Our predictions of the proportion in each group who would develop diabetes in four years were either right on the mark (18% vs. 18% for the lifestyle group), within one percentage point (27% vs. 28% for the Metformin group), or within two percentage points (36% vs. 34% for the control group). Needless to say, this is well within random variation.

How will medical practice and prevention change as a result of Archimedes in five years?

Our objective is to systematically walk through all the critical clinical questions – such as prevention, screening, initial treatment, later treatment and support care – for a variety of diseases, and determine the magnitudes of the benefits, harms and costs of all the important options that patients and physicians face. From this we will work with the most important professional societies and disease-based organizations such as the American Diabetes Association (ADA) to develop a variety of tools for improving the quality and cost of care – such things as guidelines and performance measures. For the first time there will be recommendations to patients and physicians that are based not only on clinical research, but also on knowledge of the magnitudes of the expected health and economic outcomes as estimated from a validated model.

The big unknown is the extent to which physicians will change behavior. This is tricky, because physicians don't always respond even when there is a large, convincing clinical trial. Given that physicians' behaviors and responses to new information, even from very credible sources, vary so widely, it is difficult to predict the extent to which things will change. Because the credibility of a source is such an important factor in the adoption of an innovation, we are spending as much time validating our models as we are building them. The fact that we are working with top clinical organizations such as the ADA should also increase the credibility of the model.

Is there anything Archimedes doesn't do? Any holes or gaps?

The model is no more complete and accurate than the existing research. Fortunately, there is a lot of good basic and clinical research on the most important diseases such as diabetes and coronary artery disease. However, for some other diseases there is not as much research. Asthma is an example. That will be a limitation of the model. But even there, there is a silver lining. There is nothing like building a detailed model like this to expose what we do and do not know. Furthermore, Archimedes can help determine exactly what studies are needed to fill the gaps and just how they should be designed. Over the longer run, we should be able to improve the research base needed for the future.