Hansen and Picker's HIV/SIV vaccine: New Analysis of the Paper-blunders??

A high profile paper by Picker and colleagues that was publicized widely now has been panned for its lack of proper statistical analysis, cherry picking of data etc etc. The anonymous writer and analyst commented on PubPeer that the Paper has several holes: The link is as follows:

https://pubpeer.com/publications/C8D982830E18E0D289F0CF103876E3#fb21451

I am copying and pasting the entire comment here and wish the readers to follow up on these comments and post their views:


The comment is as follows:
HIV continues to bear a heavy toll on society with a disproportionate number of infections and deaths occurring in developing countries. Development of a safe and effective vaccine is clearly needed and several vaccine approaches are in clinical and preclinical development. Those of us in the HIV field hope to see an effective vaccine be developed that will prevent HIV infection and one of the most promising candidates to date is a T cell vaccine that utilizes recombinant rhesus cytomegalovirus (RhCMV) as an SIV vaccine vector (Hansen et al., Nature 2013;502:100). However, upon closer analysis we found potential errors and omissions in the Hansen et al. study that together indicate that recombinant CMV vaccination may not provide any significant protection against SIV challenge.

In summary, our analysis of the 2013 Hansen study indicate that:
1) Animals that had been given RhCMV vaccination received significantly fewer challenge doses of SIV-Mac239 compared to unvaccinated animals (p = 0.05). With fewer challenge doses administered, there is less chance of a productive infection to occur and this by itself could contribute to the appearance of protection and markedly influence the outcome of the challenge studies.
2) The SIV-Gag-specific RT-PCR approach used mainly to measure protection from SIVMac239 challenge is an unreliable method to quantify SIV vireamia in this model due to SIV Gag-specific PCR primer cross-reactivity to the live vaccine vector itself (RhCMV-SIVGag). This observation is confirmed in SIV-Gag RT-PCR+ animals that appeared to have transient systemic infection with SIVMac239 despite no boosting of pre-existing SIV-specific T cell responses.
3) Independent evaluation of the statistical analysis within the Hansen et al. study shows that RhCMV-SIV vaccination provides no statistically significant protection against intravaginal SIV challenge (p = 0.10) and no significant protection against intravenous SIV challenge (p = 0.33).
Based on these observations, we suggest an independent re-evaluation of this vaccine approach in a researcher-blinded study in which all vaccinated and unvaccinated animals receive the same number of SIVMac239 challenge doses.

Inconsistencies in data presentation and statistical analysis:
We were initially very excited to read the 2013 Hansen et al. publication (the third in a series of AIDS vaccine studies by this group of investigators) but several questions arose, beginning with the disparity between the number of unvaccinated animals shown in Fig. 2A and the total number of animals shown in Supplemental Figure 10. In the left panel of Fig. 2A, all 16 experimental animals (Group A, vaccinated NHP) are shown regardless of their levels of vireamia. In the right panel (Group B+C, representing unvaccinated NHP), only 10/14 animals from Group B and 8/12 animals from Group C are shown. Only the unvaccinated control animals with persistent SIV infection are presented in the figure and the 8 animals that did not become persistently infected with SIV despite multiple challenge doses are absent from the figure. This is visually striking because it appears to show that all of the unvaccinated animals were infected with SIVMac239 whereas 9 of the 16 RhCMV-SIV vaccinated animals in the left side panel were protected from SIV vireamia. On page 101, the main text describing Fig. 2A also appears to indicate that all of the control animals had become chronically infected; “…9 out of 16 RhCMV/SIV vector-vaccinated female RM manifested stringent (MHC class I allele-independent) control of plasma viraemia compared with none of 18 infected female control RM (Fig. 2a and Supplementary Table 2)”. This is not the complete story when one compares the data presented in Fig. 2a to the additional data found in Supplemental Fig. 10. In fact, 31% of the unvaccinated animals in Group B+C showed no vireamia at the end of the challenge study but this information is not mentioned in the main text and is missing from the legend of Fig. 2.

The appearance of Fig. 2 prompted us to question whether the statistical comparisons of protective immunity had failed to include these important unvaccinated control animals that escaped infection after repeated low-dose SIV challenge. For SIV vaccine studies, we look for significant protection against chronic SIV infection or a reduction in plateau vireamia after infection. In the 2013 Hansen et al. paper, the fraction of infected RM that met “controller” criteria in the vaccinated group was 9 out of 16. There were no “controllers” among the animals is Groups B+C (0 of 18) and, as reported by the authors, this indicated that there was a highly significant difference between these groups (p = 0.0002, Fisher’s Exact Test). However, this approach fails to take into account that there were 26 unvaccinated animals instead of 18 unvaccinated animals and that 31% of unvaccinated NHP had not been infected with SIV. When using the straightforward criterion of no persistent SIV infection, we found that the RhCMV-SIV-vaccinated Group A (9/16 = 56% without persistent vireamia) is not statistically significant from the unvaccinated animals in Groups B+C (8/26 = 31% without persistent vireamia) by the same 2-sided Fisher’s Exact Test (p = 0.10).
In other words, when all experimental animals are included in the statistical analysis, there is no statistically significant protection against intravaginal SIV challenge following RhCMV-SIV vaccination.

In this analysis, the authors compared the vaccinated animals in Group A to the combined number of negative control animals in Group B+C. In contrast, in Supplemental Fig. 10 the authors split these groups apart when determining if there was a significant difference in the rate of SIV acquisition as a function of the number of SIV challenge doses administered. When divided into these smaller groups, there is no significant difference in the acquisition of SIV infection. However, if Group A is compared to the extrapolated number of control animals in Group B+C (i.e., similar to how the protection experiments were performed), then RhCMV/SIV vaccinated animals were infected more rapidly based on SIV-Gag RT-PCR (logrank test, p = 0.05). This is an important finding because RM were considered “infected” after detection of plasma SIV RNA of >/=30 copy equiv. per mL and are immediately removed from the challenge pool. These selected animals may therefore not receive the full series of SIV challenge doses required to infect all of the unvaccinated control animals. These details can be found in the methods summary which references the challenge protocol used in Hansen et al., 2011 and 2009 in which an animal will be removed from the challenge pool even after one low-dose SIV exposure if the SIVGag RT-PCR scores above 30 copy equiv. per mL at a single time point. Early removal of animals from the SIV challenge pool may artificially inflate the perceived levels of vaccine-mediated protection because the RhCMV-SIV vaccinated animals would be subjected to low-dose SIV exposure significantly less often then the unvaccinated controls.

The authors also performed repeated limiting dose challenge of RhCMV-SIV vaccinated animals and unvaccinated animals by the intravenous route. As shown in Supplemental Figure 10, two of 6 vaccinated animals met controller criteria whereas 0 of 4 unvaccinated animals controlled SIV infection. However, similar to the intravaginal challenge animals, our analysis indicates that this is also not statistically significant (p = 0.33, Fisher’s Exact Test). This p value was not provided in the publication.

Use of SIVGag RT-PCR that crossreacts with the RhCMV-SIVGag vaccine vector:
One could argue that protection of 2/6 animals from IV challenge with SIVMac239 is biologically significant even if it’s not statistically significant. One critical caveat to this viewpoint however, is that SIV “infection” in these IV challenge studies was not confirmed by live virus isolation but was instead based upon RT-PCR using primers specific for SIV-Gag RNA. SIV-Gag RNA is also expressed by one of the recombinant RhCMV vaccine vectors (RhCMV-SIV-Gag). This means that the SIV-Gag-specific RT-PCR assay used for determining protection is cross-reactive with the SIV-Gag gene cassette expressed during persistent infection with recombinant RhCMV-SIVGag. Indeed, in 2010 the authors showed that live RhCMV-SIVGag virus could be isolated from vaccinated NHP for up to 3 years after vaccination (Hansen et al., Science 2010;328:102) Other groups who use latent or chronic viruses as vaccine vectors realized the need to differentiate challenge virus from vaccine strains of virus that express the same viral RNA (e.g., Byrareddy et al., Plos One 2013;8:e75556) and they specifically use RT-PCR probes for virus sequences that are not found in the vaccine itself.

Use of quantitative PCR for an SIV gene expressed by a persistent virus vector such as recombinant RhCMVSIVGag is particularly problematic because it is not feasible to rule out the possibility that the small “spikes” or “blips” in transient SIV RNAemia that appear to be rapidly controlled after SIVmac239 infection could instead be simply due to intermittent reactivation of RhCMV-SIV in the vaccinated host. This question was partially addressed in which the authors identified low-level RT-PCR+ responses in one or more solid tissues in 3/3 RhCMV-SIV immunized animals that had never been exposed to SIV challenge (see Supplemental Figure 2 and Supplemental Table 1). Although specific RT-PCR analysis of plasma samples was notably omitted from these analyses, this data nonetheless indicates that RhCMV-SIV vaccine-derived or spontaneous SIV-specific RT-PCR+ products occur in the absence of experimental SIV infection. Although the implications are not discussed, the authors acknowledge that false positive RT-PCR results occur and state in the Supplemental Figure 2 legend, “The very low level detection of SIV sequence observed in occasional tissues from these 3 SIV- RM (see also Suppl. Table 1) reflect the rate of background positive amplification unrelated to SIV infection”. Knowing that the target of the SIVGag-specific RT-PCR is found in the RhCMV-SIVGag vaccine vector itself, this could account for many, if not all of the results demonstrating vaccine-mediated protection. Indeed, the authors did not demonstrate actual SIV infection in plasma using any other virological techniques such as live virus isolation - with the possible exception of the vaccinated animal, Rh20363 (an elite Mamu-A*01+Mamu-B*17+ controller with a genetic predisposition for superior control of SIV infection, Supplemental Table 2). This specific animal (Rh20363) is particularly interesting because unlike the other RhCMV-SIV-vaccinated animals, it was the only macaque to mount CD8+ T cell responses to SIV epitopes, CM9 and SL8 (Supplemental Fig. 13) and the only one to demonstrate controlled, low-level vireamia (Fig. 2) that rebounded after anti-CD8 depletion (Supplemental Fig. 14).

Lack of anamnestic T cell responses support the theory of SIVGag RT-PCR artifacts:
Since the SIV-specific RT-PCR assays are cross-reactive with the live persistent rhCMV-SIVGag vaccine vector, a better independent approach to confirming recent antigenic stimulation is to measure the boosting of the SIV-specific T cell response. Live virus vaccines, including RhCMV, induce memory T cells that respond to infection by proliferating and expanding to higher levels. Indeed, this is a defining characteristic of an antigen-specific memory T cell response. The authors state that Effector Memory T cells (TEM) induced by recombinant RhCMV-SIV vaccination are unique in the sense that they do not proliferate and that, “As previously described, protection occurred without anamnestic boosting of vaccine-elicited SIV-specific CD8+ T cell responses in the blood (Fig. 1b)…” (page 100, right column). However, this assumption appears to be refuted by the author’s own data. In Supplemental Figure 7, the data shows that booster vaccination with RhCMV/SIV of female RM results in about a 5-fold increase in SIV-specific memory CD4+ T cells and a 4-fold increase in SIV-specific memory CD8+ T cells in the blood. Similarly, data in male RM indicate that booster vaccination with RhCMV/SIV results in approximately an 8-fold increase in SIV-specific memory CD4+ T cells and a 4-fold increase in SIV-specific memory CD8+ T cells. Further proof of RhCMV-SIV-specific effector memory T cell proliferation can be found in Hansen et al. Nature 2011, Fig. 2a. In these experiments, SIV-specific effector memory T cells from RhCMV-SIV vaccinated animals proliferate by greater than 4-fold after intramuscular booster vaccination with recombinant Ad5-SIV. Since booster vaccination is essentially re-infection with a live virus carrying SIV antigen, this data shows that RhCMV vaccine vectors do indeed elicit readily detectable antigen-specific CD4+ and CD8+ effector memory T cells that show robust proliferation in response to viral infection.

As noted on page 102 of Hansen et al. 2013; “Other characteristics of protection in these intravaginally challenged, RhCMV/SIV vector-vaccinated female RM were identical to those previously reported for RhCMV/SIV vector-mediated protection of male RM against intrarectal challenge, including development of de novo SIV Vif-specific CD4+ and CD8+ T cells, lack of SIV Env seroconversion, and lack of CD4+ T-cell depletion at mucosal sites”. It is difficult to gauge the specificity of the Vif-specific T cell response since the frequency of these T cells prior to challenge is not provided and a low frequency of up to 0.03% Vif-specific T cells can be observed in SIV-negative RhCMV-vaccinated animals prior to challenge (Hansen et al. Nat Med 2009, Figure 4d). In the 2013 Hansen et al., study, systemic plasma viral load (as measured by SIVGag RT-PCR) can reach greater than 10 million genome equivalents/mL of plasma (Fig. 3) and yet no boosting of SIV-Gag-specific CD8+ T cell or SIV-Gag-specific CD4+ T cell responses is observed in “protected” animals. Based on these results, the lack of seroconversion to SIV (after achieving systemic vireamia levels of 10,000 to 10 million genome copies/mL) coupled with the lack of an anamnestic T cell response in the RhCMV-SIV vaccinated animals after purported brief systemic SIV infection cannot be due to an inability for T cells to expand in response to SIV antigens. Indeed, RhCMV-SIV specific effector memory T cells readily proliferate in response to RhCMV-SIV or recombinant Ad5-SIV infection. An alternative hypothesis is that these particular animals simply did not receive an infectious dose of SIV before being removed from the low-dose SIV challenge pool.

There has been considerable interest in the peculiar nature of RhCMV vaccine-mediated immunity. Unlike other vaccines, the RhCMV-SIV vaccine vector elicits essentially complete protection in about half of vaccinated animals and no protection in the other vaccinated animals. The lack of any appreciable level of partial immunity has remained a mystery. However, based on the details discussed here, it seems that approximately half of RhCMV-SIV vaccinated animals exposed to low dose SIV challenge become chronically infected with no reduction in plateau vireamia. The other half of the vaccinated animals are not exposed to an infectious dose of SIV due to premature removal from the challenge pool following a transient low level “blip” of SIV-Gag+ PCR that is an artifact of intermittent SIV-Gag RNA expression by the persistent RhCMV-SIV-Gag vaccine vector. This interpretation is supported by the most recent paper (Hansen et al., 2013) in which RhCMV-SIV vaccinated animals are more rapidly removed from the SIVmac239 challenge pool (p = 0.05) and when independent statistical analysis is applied, there is no statistically significant protection against SIV following intravaginal challenge (p = 0.10) or intravenous challenge (p = 0.33).

Comments

Popular posts from this blog

Coronavirus and Vaccine Development

More interesting discoveries from the RV144 trial (Tweeted by ‏@TweetSosa )