Endpoint is a few weeks away.
Could length of trial be explained by better than expected survival in test arm ?
New simulation and discussion.
Cel-Sci, the final countdown
CEL-SCI (NYSE CVM) Phase III clinical trial is about to end soon.
Whilst this clinical trial is reaching its end, there are various claims by both Cel SCI management and some shareholders following the stock, that the experimental compound, Multikine (MK), is necessarily showing some efficacy because recruited patients are surviving longer than they should : Their claim is that the clinical trial should have ended a while ago under standard survival hypothesis for this type of cancer.
Nonetheless, other claims argue that patients survive longer in Clinical Trials whether or not they take MK, and is typically a reason why those trials last longer.
The purpose of this article is to explain, in the simplest possible terms, the pros and cons of each argument by examining in detail all the parameters of the study
A previous article written in July 2019 has exposed an initial analysis on this clinical trial. This new article is both an update and a last critical analysis before the binary event that will either witness the trial success or failure.
I-Quick summary of the trial design
As a reminder, the clinical trial is designed with three arms, two of them being used for the primary endpoint determination. 3/7th of the patients have been randomized to each of the two arms.
As per study design aimed at proving an effect size superior to 10% in the test arm, sample size has been determined in the SAP to be of 298 events. Meaning that the two comparative arms will need to show 298 cumulative deaths for the primary endpoint to be reached.
All Arms receive Standard of Care (SoC). SoC is Surgery + Radiotherapy or Surgery + Radiochemotherapy. There is a third arm (1/7th of the patients) which is not relevant for assessing success of the Clinical Trial.
Efficacy of the drug will be assessed through the logrank test, a calculation that allows to compare two survival curves. Significance (the "famous" p number) is calculated from this test. If p <0.05 then the trial will be considered as successful (meaning that the sample has shown the level of efficacy required with an at least 95% confidence it mimics reality)
II- Survival Analysis could help to assess if this Phase III can be successful
Until late January 2020, CEL-SCI management has claimed that the last expected event had not occurred yet.
As the trial is blinded, there is no way to know if the people surviving longer than expected are in the Test Arm or the Control Arm. However, given the patient population and cancer type resulting from inclusion criteria, we have developed a variety of powerful tools that allows to estimate survival in the group treated by SoC alone. Comparing SoC alone survival to the behavior of current patients in this clinical trial can be a way to assess if MK is efficient.
In survival analysis, you need to estimate Overall Observed Survival for SoC arm in order to compare it to observed survival for this group of Patients from CEL-SCI clinical trial. One important factor in survival analysis is to understand the inclusion criteria in terms of tumor size and lymph nodes.
II-1 Inclusion criteria
As per Clinical Trial site, Inclusion criteria is untreated Squamous Cell Carcinomas (SCC) of Oral Cavity/Soft Palate, categories T1N1-2M0, T2N1-2M0,T3N0-2M0,T4N0-2M0 (T4 allowed only if invasion of mandible is negligible).
More than 90 percent of cancers that occur in the oral cavity are squamous cell carcinomas.
Which can be summarized by the following graphic:
This corresponds to AJCC’s Stages III and IVa.
Another important point to consider, which is often neglected by people running survival analysis for CEL-SCI, is the exact tumor location for the Trial. CEL-SCI’s Scientific Presentation provides clues in p 13. Oral cavity includes lip, oral tongue, floor of mouth, buccal mucosa, upper and lower gum, retromolar trigone and hard palate. In particular, it excludes Base of Tongue (BoT), Oropharynx, Hypopharynx, etc. The study is also about Soft Palate so you need to add Soft Palate in the inclusion criteria.
Please note that it is critical to exclude BoT, because the BoT has a lot of occurrences in the US and many oral HPV cases are located in BoT. BoT cancer is highly curable and if one does not exclude it, the whole overall survival numbers will be exaggerated.
Most of the cumulative statistics shown by doing simple internet searches in publications and websites, like this one, include BoT cancer and provide "relative survival figures". One needs to really dig into the detailed database to remove BoT cancer from the survival analysis and search for "Observed survival" which differs from "Relative survival" by 5-6% at year 5 (see for example in the table below).
II-2 Survival statistic for Standard of Care
We accessed the ultimate tool for the survival analysis: the SEER online database (SEER*Stat Software). The SEER database compiles survival for many sites and spans many years in the US. We applied the inclusion criteria for the Oral Cavity, excluding BoT. We included years from 2000 till when data is not available (2017) to get a flavour of the variance of survival through time, and from 2010 until 2016 which corresponds to the years patients were recruited for CEL-SCI clinical trial.
The SEER database query retrieved the following survival data, that we broke down into two tables : the first one (figure 1) because there is more data which is useful for Confidence Intervals to see how survival evolved in many years and the second one (figure 2) because it corresponds more or less to the years CEl-SCI patients were recruited and as well because it provides staging information from AJCC that was not available in database prior 2010.
Figure 1-Survival Means 2000-2016 n=14,462
Figure 2- Survival means by stage 2010-2016 n=5,635
- Data is pretty homogeneous across the years showing little variance (tight confidence interval) on both tables meaning that survival of this type of cancer has not evolved much from 2000 till 2016 : SoC has not improved much the survival of the patients from 2000 till 2016 despite what drugs manufacturers want you to think that treatments have improved
- For Stage III patients, survival is 56.8% at Year 3 and 48.5% at Year 5. There is a great difference between Stage III and Stage IVa patients but there are also many more patients in Stage IVa than in Stage III. This is a global pattern that will be seen in this international clinical trial as well, meaning that the survival pattern will likely be closer from Stage IVa than from Stage III.
- Basic Calculation will show that with an Overall Survival close to 40% at year 4, the endpoint should have been reached many months ago: at least 400 of the evaluable patients from the original study arms should have passed away by now, but until recently (end of January 2020) Cel SCI communicated that the endpoint had not been reached.
We can already conclude that patients are living longer than they should. We need to analyze what causes this delay in the endpoint.
III- Analysis tool
We have developed a Spreadsheet that takes into account various parameters of the study. This is a predictive tool aiming at simulating survival in a given date in the SoC arm from the enrollment date of the various patients cohorts, then adding survival benefit in the Test arm so that the endpoint can be reached at the desired date. For instance, we know that by end of January endpoint was not reached.
Being very conservative, we have considered a survival curve for SoC that was superior to the expected SoC survival for this study, and eve superior than the Stage III survival known during the five years (NB : patients in this study include Stage IVa).
For the needs of our analysis tool a Logarithmic log regression was performed to calculate the survival numbers at any point in time in the SoC arm.
The curve does stay between the lower and uppers boundaries for Observed Survival Stage III as per the SEER table above.
The curve is superior by a range of 20% to 28% to the survival curve retrieved in SEER database for the patients in this condition (data in Red)
We obtained the following results and survival curve (data in Green) :
Figure 3- Survival models in Standard of Care - SEER db and used Regression
We computed this SoC in our Sheet - CVM Clinical Trial Final Survival Analysis by Fosco
We have considered a cumulative dropouts rate of 3.5% per year in the study (Initial patients should show more dropouts than the latest ones) reaching 18.9% at trial completion calculated as of 15th February 2020 (646 Evaluable patients in the two comparative arms of the study).
The SoC survival curve then computes a number of events for a given date.
This tool has been proven reliable so far as it allowed to predict that the endpoint would not happen in early 2019 (as announced by both Cel SCI and Cel SCI CRO), at best in late 2019 and predicted an end date by Q1 2020.
Under the above assumptions, the output of this tool is compatible with a 20% benefit of a clinical trial MK arm VS SoC.
Furthermore, we tried to predict median survival in the following sheet.The tool predicts that 164 patients out of 328 in the controle arm should have passed away (50%) by October 15th 2019, therefore the median survival in the SoC arm should be of 54 months (vs SEER’s 55 months for STIII see table above) as weighted time since treatment was approx 4.5 years for all evaluable patients.
In the test arm, at the same date, 127 events should have happened, therefore more events and months are needed for the test arm to catch up the number of required events to reach the median.
In the same fashion, the survival curve in the test arm predicts a median survival reached by October 15th 2021, so approx 6.5 year or 78 months since treatment
Survival curves predict a median survival improved by 24 months
Figure 4- 24 months median survival improvement
IV- Confronting the potential causes of trial delay
We have identified 3 main potential causes as reasons for a delay in clinical trial (Not given in importance order) :
CAUSE #1 : Dropouts : the more people dropout from the trial the less events will be counted for and the endpoint will take more time to happen
CAUSE #2 : Increased overall survival of patients treated by SoC, in both arms. This is a recurrent argument given by some clinicians. Patients in Clinical Trial tend to survive better than in real life. Real life being supposed to be measured in SEER we need to confront this potential bias with available documentation from research in an objective way
CAUSE #3 : Increased overall survival of patients treated in MK arm, in test arm only. This would be the "raison d'être" of this clinical trial : show a better survival in the Test arm that explains why this trial is lasting longer than expected
Using our simulator, we have illustrated the impact of the above causes in the following figure, dropouts factor in horizontal axis and better than expected survival in control arm in the vertical axis. Obviously, if patients survive better in the control arm, then the delay would not mean anything and the sample would have a strong probability of showing no significance at endpoint :
We have discussed extensively CAUSE #1 in the following article.
We believe, and this belief is supported by some information available on the web that we gathered from the clinical trials sites, that dropouts are not negligible but still contained around 15% and in any case below 20%. We have used a 18.9% dropout rate in our simulator and still found some efficacy (20.5%) in the Test arm.
CAUSE #2 :
Can SoC patients in this clinical trial be living longer than the national database states? There are contradictory studies comparing survival in Clinical Trials VS Reality.
Let’s examine two of them, Study 2 being the most recent one.
Study 1“Trial participation was associated with better survival in the first year after diagnosis, likely because of eligibility criteria that excluded higher comorbidity patients from trials. Similar survival patterns between trial and nontrial patients after the first year suggest that trial standard arm outcomes are generalizable over the long term and may improve confidence that trial treatment effects will translate to the real-world setting”
Study 2"Median survival for those not enrolled in a trial versus those enrolled was 5.70 (95% CI, 5.70–5.71) versus 5.00 years (95% CI, 4.87–5.12). The 5-year OS for those in trials versus not in trials was 0.57 versus 0.50". (Noteworthy, this study was kindly provided to me by Biotech Beast who defines himself as "Scientist and trader of biotech stock" as a proof Cause #2 might be the reason for trial delay. As a matter of fact this article proves the opposite, as we are going to see below)
While this trial reaches almost the five year mark for all the participants, Study 1 states that their should not be a notable Clinical Trial effect on the survival in control.
Study 2 states that there should be a 14% improvement of survival at year 5 (57% VS 50).
We do not know if this clinical trial effect would apply for the Oral Cavity cancer types, but for the sake of our analysis, we have used a SoC survival at year 5 that is 47.4% VS the 38.1% from SEER database for this inclusion criteria. This is 24% better. Hence, for the sake of our analysis we have been well over the potential “Clinical trial effect” mentioned in Study 2 for SoC improvement in survival being 14% and still have been able to prove efficacy (Test arm showing more than 20% more survivals vs control arm in average, or a hazard ratio around 78% ).
For this trial to fail significance, simulation shows that the observed SoC survival at year 5 should be around 53%,e.g. show a 39% better survival than in real life. Unless there was a huge glitch in the inclusion criteria as screened by the investigators _this would have been highlighted by the IDMC_ we do not see as likely to witness such survival in the control arm for this very bad cancer condition in this clinical trial.
In addition, median improvement has been noted to 24 months, which cannot be explained by the Clinical Trial effect from Study 2 (0.7 year or 8 months improvement).
Let’s also discard the belief that PD1 inhibitors could be the cause of a better survival in SoC. PD1 inhibitors where not available in most countries where the CT took place at the time the trial took place (most events have happened in the past). We have also seen above that increased median is not compatible with PD1 inhibitors which increase life by only a few months.
V- What about survival in other countries ?
We incidentally read this article stating survival in various countries and in the “la crème de la crème” US hospital, namely Memorial Sloan-Kettering, New York, NY. (following link gives the top 20 Cancer Hospitals)
In all countries, Stage IV shows poor survival. To the exception of MSKCC hospital, Stage III patients have also a poor outcome. MSKCC shows surprising good survival for stage III (64.5%), but it looks like the exception. This could be due to the presence of highly curable HPV cancer that shows a better survival rate, but we cannot rule out that excellence in healthcare can improve outcome of patients significantly. In any case this outcome is unfortunately not true for ST IV patients and we don’t expect patients in this clinical trial to survive much longer than those treated in the Number One US hospital.
We have therefore reasonable reasons to believe that the CAUSE #3 is the cause of this clinical trial duration.
VI- This Phase III trial has a fair probability of success
Assessing the risk of this trial has been a challenge for years.
Above all, what closes the door to extreme negativism around the MK treatment is the simple fact that current Phase III patients live much longer than they should if treated with Standard of Care alone by any metrics of Standard of Care survival which we could extract from the US cancer database, a country where patients have better hopes of survival than any of the other regions where this clinical trial was performed.
Therefore, we believe, given all the above and the thorough due diligence we have been carried out, that at this point in time, there is very little risk that Primary Endpoint will not be met by this Trial.
VII- Company is deeply Undervalued
In the last year or so, CEL-SCI stock price has witnessed an impressive come back from less than $1 in July 2018 to more than $10 in February 2020. This is mainly due to the Independent Data Monitoring Committee (“IDMC”) and FDA removing all the holds and the successful end of the arbitration with the previous Contract Research Organization (“CRO”). While the stock has been in bearish territories for years, it looks like the bulls have come back to support the price.
Cel SCI hopes that this drug will be part of Standard of Care. If effective it will allow some patients to live many more years. If Phase II results are replicated in this Phase III _in particular total response rates_ some patients might witness a total disappearance of the tumor, a total cure. If shrunk significantly, other patients will be able to avoid a painful and disfiguring surgery. If survival improvement is proven as good as it seems to be, it will dwarf results obtained by other immunotherapy drugs such as Merck’s Keytruda or Bristol-Myers Squibb’s Opdivo which are very toxic drugs and relatively not so effective (median overall survival being only improved by a few months). Those are billion dollars blockbuster .
Therefore, if survival benefit is confirmed, we firmly believe that this drug will be part of Standard of Care.
There are 165’000 new HNSCC cases each year in the U.S., Canada and Europe. Of these, 110’000 are "advanced" Stage III or IV, and are MK eligible. The average new oncology drug cost is between $200K and $400K. Let us assume a market penetration between 20’000 and 50’000 a year and MK only receives $50K per treatment. Annual sales would be between $1B and $2.5B.
Assume a multiple of 4x sales, which is extremely conservative, value of the company would be between $4B and $7.5B. Fully diluted by all possible shares plus future potential dilutions (50M shares) this equals around $80 to $150 per share.
We are suggesting a fair valuation between $80 and $150 per share, if MK is successful. We believe this drug will be a blockbuster.
As primary endpoint is the open door to approval by the FDA, we believe there is a very high probability of success for this drug
VIII - Trial Risks
While we strongly believe that this trial should be a tremendous success in terms of survival, there are many aspects of a Clinical Trial that we did not factor-in and could be the cause of failure.
Failure could be due to many factors, such as, but not limited to:
- SoC showing an extraordinary capacity to heal patients that we have not enough anticipated. This could happen for instance if the inclusion criteria for patients recruitment has not been well respected or tend to chose patient with the least lethal form of the cancer (lower staging)
- A very high level of dropouts during the study (well above 20%)
- Violations of the protocol not detected or reported properly by the investigator with consequences either on survival or on the whole study
- Considerations due to the fact that the trial happened in many countries outside of the USA.
All those risks do remain and a wise investor will consider them before making an investment decision. If the trial fails, we would expect the share price to be close to $1.
Disclosure: I am/we are long CVM.