It is quite an achievement for a diagnostics company to endure for 13 years and attain a valuation of $9bn without releasing any clinical data, but perhaps not a commendable one. Theranos (THER) has repeatedly promised to provide scientific evidence for its range of direct-to-consumer blood tests, but a group of academic researchers has got there first – and the reported results are not good.
Several of Theranos’ finger-prick tests gave less reliable results when compared with commercially available blood draw products, with cholesterol readings in particular being consistently low. Theranos has decried the study as “flawed and inaccurate”, but the group must recognize that the only real way to counter the researchers’ conclusions is to release clinical data of its own.
The researchers from New York’s Mount Sinai hospital pitted several of Theranos’ tests, which used the company’s much-vaunted Nanotainer technology, head to head with similar products from Laboratory Corporation of America and Quest Diagnostics. The trial was conducted in July 2015; in October, the FDA declared the Nanotainer an unapproved medical device for all uses except the company’s herpes HSV-1 test. The Nanotainer is no longer used in the US for the purposes detailed in the study – and the results will not help it regain its former place on the market.
Outside the range
The study saw 60 healthy adults seek consumer testing services totaling 22 separate lab tests, including red and white blood cell counts, lipid and cholesterol tests and biomarkers of liver and kidney disease.
The tests were sent off to the three companies’ Clia-certified facilities, and while that means the researchers had no control over the actual testing procedures, this is at least a reflection of how these services work in practice. The researchers collected 14 samples per subject for each lab measurement, giving a total of 18,480 measurements.
They found that Theranos tests had higher sample rejection rates, with 2.2% of test data missing compared with 0.2% for Labcorp and 0% for Quest (NYSE:DGX). Theranos’ tests also flagged results as outside the normal range – each service provides test-specific reference ranges, in accordance with guidelines – 60% more often than tests from the two other providers. The proportion of measurements outside their normal range were 8.3%, 7.5% and 12.2% for LabCorp, Quest and Theranos, respectively.
It was cholesterol testing that showed the most dramatic differences between Theranos’ tests and those from the other companies. Analyses showed that, when compared with both Labcorp’s and Quest’s tests, the Theranos tests reported "systematic biases" toward lower test values – around five to ten points too low – for total cholesterol, LDL and HDL. There were no significant differences on these measures when Labcorp’s tests were compared with Quest’s.
More data please
The Mount Sinai researchers were the first to admit that this study is not the most robust assessment: most obviously, they have no knowledge of the actual testing methods Theranos employs. Theranos has a proprietary instrument called Edison but also uses other companies’ devices to run its tests. The differences found in the trial could have arisen during collection or processing or owing to the instrumentation used, or a combination of these factors, and there is no way to know.
But arguably that is the point. Theranos’ methodology and procedures are a mystery not just to the researchers but to everyone outside the company.
Theranos offers nearly 250 separate tests directly to the consumer, ranging in price from $1.53 to $117.96. Customers will stay away in droves if they get the message that these tests, far from being as reliable as traditional blood draws and easier to do, appear to be neither.
Theranos has written to the Journal of Clinical Investigation, in which the study appears, to repudiate the findings. The company said that sample collection procedures were "faulty" and do not represent normal practice, and pointed out that the researchers did not use any reference testing methods. Theranos also alleged that the lead author failed to identify a potential conflict of interest.
But there is only one way for the company to be sure of restoring consumer confidence. Theranos must keep its promises and publish a robust study in a peer-reviewed journal showing how – and that – its tests work.