In the past few months I have read a lot of negative comments and opinions on Northwest Biotherapeutics (NWBO) regarding their company history, data, CEO and the potentially ground breaking new product DCVax -L. One particular article where one author, Adam Fuerstein, was particularly critical of the data from NWBO's DCVax-L Phase II results, claiming that "Northwest Bio has the chutzpah to claim the benefit conjured up in this ginned-up survival analysis is statistically significant." As such, I decided to pull out the old Statistical handbook, and delve into this a bit more.
First off I used the data provided on the recent company presentation on the Phase II results for 20 patients using DCVax -L compared to 119 patients using the Standard of Care (SOC) for treating Gliobastoma Multiforme (GBM). Now here is where much of the criticism begins in that there is no control to properly compare the DCVax treatments with. Furthermore, Adam F. criticizes that comparing the DCVax results to a different sample set is a "grinned up ... analysis". Well, one rather intuitive statistician, Frank Wilcoxon would vehemently argue that fact, mainly because he developed a method to compare two independent populations, the Wilcoxon Rank Sum Test. More importantly, it was used for Non-normal Distributions, which is exactly what we see with both the 119 SOC treated patients as well as the 20 DCVax-L treated patients. So let's start at the beginning, with the data:
From this graph, we have all data points, (and yes it took me a long time to interpolate all these values). From both sets of data we get mean (average) values for both populations (Surviving DCVax patients were given the final survival rate of ~123.48 months to be conservative).
μSOC = 24.33 months, σDCV = 18.15, nsoc = 119 patients
μDCVax = 47.63 months, σDCV = 35.13, nDCV = 20 patients
The difference between the median values listed on the graph and the mean values calculated above is that the mean is the average, where the median value is the middle survival length, MSOC =17 and MDCV = 36.4 months. The average survival rates are actually greater than the median values, which also provides a good indication that the distribution is not normal. Another side note of information, is that the National Brain Tumor Society states a median survival rate of ~15 months; 2 months less than the data provided for the comparison SOC treated patients (which adds more conservatism to NWBO's comparison case). So now, let's prove that these populations are not normally distributed by taking the frequency of the data to produce a Histogram of the Standard of Care data below.
As we can see from the histogram, the mound is concentrated between 7 to 13 months, with a long tail of values in the upper region. We know this is the case for the DCVax patients as well, with patients still living. More so this is commonly observed on distributions for treatment and reaction studies. Now we must use a nonparametric test to compare these two samples, so we will use the famous Wilcoxon Rank Sum Test (famous in terms of Statisticians).
As such, a nonparametric statistical test must be used to evaluate and compare these two populations. The Wilcoxon Rank Sum Test for Large populations (n1 > 10, n2 > 10) would provide a robust method to compare these two mean values.
Setting up the hypothesis test we get:
Ho: D1 & D2 are identical
Ha: D1 is shifted to the right of D2
Where D1 and D2 are the DCVax Treated and Standard of Care Distributions respectively.
Ranking each of the samples of both sample sets and summing the rankings of each set, we get the following:
T1 = 1964 = Summed Rank for DCVax Samples
T2 = 7766 = Summed Rank for Standard of Care Samples
Now since n1 > 10 and n2 > 10, we are able to use the Wilcoxon rank sum test using the Z-Test and the normal distribution probabilities.
We use the following equation for the test statistic:
Test Statistic z =
Now we can solve for the probability that these two distributions and ultimately their mean values are the same, by converting the Z-value of 3.3846 into a p-value of 0.000356. What does this mean in Adam F. terms? It means that there is a probability of 0.0356% that the 20 patients that received the DCVax vaccine had no increased survival due to the vaccine, that it was just amazing luck, or NWBO was able to pick 20 of the right patients. More so, it would be difficult for NWBO to choose the youngest patients (as Adam F. alleges) as the average age of a GBM patient is 64 years old.
As you can see in the graph that NWBO provided, the p value is 0.0003 rather than the 0.000356 which I calculated, however, I used several conservative assumptions such as the living DCVax survival rates which would easily account of 0.000056 difference. Furthermore this calculation does reproduce the statistical significance of NWBO's Phase II results comparison in layman's terms and provides some more insurance that, as opposed to said author's distrust of this company, the statistical analysis of this data was not grinned up, and pulled out of the air, kind of like many of the accusations certain authors like to produce. Most Phase II trials hope to achieve p values of 0.05 or less, providing enough confidence for further Phase III studies. However, with a p-value as low as 0.000356, DCVax-L is providing some truly significant results. Couple this with the fact that they are in the midst of a Phase III trial, with production facilities in both Europe and in North America, the company seems poised to do what very little Biotechnology companies do, succeed.