A recent critique from a prominent public health expert, published by Eyal Shahar via The Brownstone Institute, has brought to the forefront significant methodological concerns surrounding the Centers for Disease Control and Prevention’s (CDC) annual estimates of influenza vaccine effectiveness (VE). These annual figures, crucial for public health messaging and policy, are derived primarily from a research approach known as the test-negative design. However, the integrity of this design, and consequently the trustworthiness of the resulting vaccine effectiveness data, has now been challenged by none other than the National Institutes of Health (NIH) Director, Dr. Ashish Jha, adding considerable weight to the ongoing scientific debate. This high-level scrutiny underscores a growing demand for greater transparency and rigor in the assessment of public health interventions.
Background and Context: The CDC’s Role in Flu Surveillance
The CDC plays a pivotal role in monitoring influenza activity and evaluating the efficacy of seasonal flu vaccines in the United States. Each year, as part of its public health mandate, the agency publishes estimates of how well the flu vaccine performed in the preceding flu season. These estimates are vital for informing healthcare providers, policymakers, and the public about the protective benefits of vaccination, guiding decisions on vaccine uptake and future vaccine development strategies. The test-negative design has become a widely adopted methodology for observational studies of vaccine effectiveness, not just for influenza but also for other respiratory viruses like SARS-CoV-2.
The core principle of the test-negative design involves comparing the vaccination status of individuals presenting with acute respiratory illness (ARI) who test positive for influenza (defined as "cases") against those who test negative for influenza but present with similar symptoms (defined as "controls"). The rationale behind this approach is that by restricting the study population to individuals who actively seek medical care for similar respiratory symptoms, potential confounding factors related to differing healthcare-seeking behaviors between vaccinated and unvaccinated groups can be mitigated. This aims to provide real-world effectiveness data, complementing the controlled environment of randomized clinical trials which are rarely feasible for annual vaccine updates due to the constantly evolving nature of flu viruses. While lauded for its practicality and efficiency in generating timely VE estimates, the very premise designed to reduce one type of bias may inadvertently introduce others, as detailed in the recent critiques.
The Core of the Critique: Methodological Flaws Emerge
The NIH Director’s recent public criticism marks a significant moment in the discourse surrounding flu vaccine effectiveness. His primary concern revolves around the potential for "colliding bias" within the test-negative design. While limiting the study population to individuals who seek medical care for respiratory symptoms might seem like a sensible way to control for differences in health-seeking behaviors, it can create a statistical artifact. Colliding bias arises when researchers condition on a common effect (in this case, seeking medical care for ARI) of two independent causes (vaccination status and influenza infection). This can inadvertently create a spurious association or disassociation between the causes, making the vaccine appear either more or less effective than it truly is. The net impact of this specific bias on VE estimates, whether it inflates or deflates them, remains largely undetermined and poorly quantified in many studies. This fundamental structural flaw calls into question the very foundation upon which these critical public health estimates are built, urging a re-evaluation of a long-standing methodology.
Eyal Shahar’s analysis further expands on these concerns, identifying several additional shortcomings inherent in the test-negative case-control studies. His detailed examination specifically targets a large study from the 2022-2023 flu season conducted by the VISION Vaccine Effectiveness Network, a key collaborator with the CDC. This focused critique reveals layers of potential biases that could significantly distort the reported vaccine effectiveness figures.
Confounding by the Background Risk of Infection: A Temporal Misalignment
One of the most compelling arguments raised by Shahar centers on confounding introduced by the fluctuating background risk of influenza infection throughout a typical flu season. Flu activity is rarely constant; it usually experiences a pronounced peak in late fall or winter and subsequently wanes. Vaccine rollout, however, is often staggered, with the majority of the population receiving their vaccinations early in the season, typically preceding or coinciding with the initial surge in cases. This inherent timing difference between vaccine uptake and disease incidence creates a significant challenge for accurate assessment.
In the 2022-2023 season study, Shahar notes a critical temporal misalignment: "vaccinated people accumulated more exposure time when the background risk of infection was low… and unvaccinated people accumulated more exposure time when the background risk was high." Specifically, the data showed that a lower share of the vaccinated population (47%) was exposed during the high-risk period of October-December 2022, compared to the unvaccinated population (61%). Conversely, the vaccinated group had a higher share of exposure (53%) during the low-risk period of January-March 2023, compared to the unvaccinated (39%).
This uneven distribution of exposure time creates an inherent bias. If a larger proportion of vaccinated individuals’ "at-risk" time falls within periods of lower viral circulation, while unvaccinated individuals are disproportionately exposed during peak activity, the vaccine will artificially appear more effective than its true biological impact. Shahar illustrates this with a hypothetical extreme: if a "saline solution" were administered only when infection rates were already declining, it would misleadingly appear to offer protection against a receding threat. This temporal confounding, previously highlighted in analyses of COVID-19 vaccine effectiveness studies, appears to be a significant, yet often unappreciated, factor in flu vaccine evaluations, especially given the typical pattern of flu vaccine campaigns. This bias, if not adequately addressed through sophisticated time-dependent matching or modeling, can inflate VE estimates without reflecting true biological protection, thereby presenting an overly optimistic picture of vaccine efficacy.
Immortal Time Bias: An Artificial Period of Protection
Another critical flaw identified by Shahar is the presence of "immortal time bias," a well-documented source of distortion in observational studies. This bias arises from the common practice of excluding events that occur within a specific period (e.g., 14 days) following vaccination. The rationale behind this exclusion is to allow sufficient time for the vaccine to induce an immune response and to avoid counting infections that occurred before the vaccine could reasonably be expected to be effective.
However, this analytical decision inadvertently creates an "immortal" period for vaccinated individuals—a window of time during which they are counted as "vaccinated" but cannot, by definition, experience a vaccine "failure" (i.e., infection) that would be attributed to the vaccine. If an individual contracts an infection within this 14-day window, they are often excluded from the vaccinated group’s risk calculation or their event is simply not counted towards vaccine failure during this period. Meanwhile, unvaccinated individuals are continuously at risk throughout the entire observation period, with all their infections contributing to the unvaccinated group’s incidence rate. Shahar cites previous research on COVID-19 vaccines where removing this bias drastically altered effectiveness estimates, sometimes cutting them by half. For the flu vaccine, he suggests that current VE estimates, often in the 30-40% range, could drop to 20% or lower if this bias were properly accounted for. The VISION network study, by excluding nearly 2,000 outpatient encounters due to vaccination occurring 1-13 days before the index date, exemplifies this problem. The lack of detailed breakdown of these exclusions further obscures the true impact of this bias, making it difficult for external researchers to quantify its influence on the reported VE.
Detailed Examination of the 2022-2023 VISION Network Study’s Questionable Findings
Beyond the systemic biases inherent in the test-negative design, Shahar delves into specific anomalies within the 2022-2023 VISION Network study’s reported results, further undermining their credibility and consistency.
-
Counter-Intuitive Effectiveness Against Hospitalization: The study reported that adjusted vaccine effectiveness against hospitalization (a more severe outcome requiring intensive medical intervention) was almost always smaller than the effectiveness against an outpatient encounter (a milder outcome). This finding is inherently counter-intuitive. Generally, vaccines are expected to show similar or even stronger effects against more severe disease outcomes, as each step in the progression from mild illness to hospitalization represents an additional layer of risk reduction that a protective intervention should ideally influence. The study authors’ claim of "similar VE within the same health systems across ambulatory and inpatient settings" is challenged by Shahar, who points out a consistent, and sometimes substantial, reduction in VE for hospitalization. Their struggle to provide convincing explanations for this discrepancy further raises questions about the internal consistency and biological plausibility of the reported figures.
-
The Persistent "Healthy Vaccinee Bias": Shahar notes that in the majority of analyses, particularly for outpatient encounters and universally for hospitalizations, the estimated effectiveness was lower after statistical adjustment. This phenomenon is indicative of the "healthy vaccinee bias"—the well-established observation that vaccinated individuals tend to be healthier, more health-conscious, and have fewer underlying comorbidities than their unvaccinated counterparts. This inherent health advantage can make vaccines appear more effective in unadjusted analyses, simply because vaccinated individuals might naturally have better health outcomes regardless of the vaccine. While statistical models aim to adjust for measured confounders (like age, existing health conditions, socioeconomic status), Shahar emphasizes that no matter how sophisticated, these models cannot fully account for all unmeasured aspects of health status and lifestyle. This leads to residual bias in the "adjusted" estimates, meaning that part of the observed benefit attributed to the vaccine might actually be a reflection of the healthier baseline status of the vaccinated population rather than a direct vaccine effect.
-
Opaque and Complex Statistical Models: A significant concern for scientific reproducibility and scrutiny is the description of the statistical models used in the VISION study. The authors employed "unusually complex models" that combined classical covariates (some in a non-linear form, such as natural cubic spline variables for age and calendar time) with inverse-propensity-to-be-vaccinated weights (IPVWs) derived from generalized boosted regression trees. While sophisticated modeling can be beneficial for controlling confounding, the lack of clear explanation regarding how "additional imbalance" was detected and which specific variables were used for computing these complex weights makes replication impossible. This opacity hinders independent verification and robust peer review, which are cornerstones of scientific integrity. Without a detailed methodology that can be reproduced by other researchers, the reliability of the findings remains in question.
-
Anomalous Age-Related Effectiveness: The study presented another perplexing finding: higher effectiveness against hospitalization in the elderly (41%) compared to younger adults (23%), even though effectiveness against outpatient encounters was similar across age groups (41% vs. 45%). This is unexpected, as immune responses tend to wane with age, a phenomenon known as immune senescence, typically leading to lower vaccine effectiveness in older populations. The authors attempted to explain this by suggesting that most older adults received "enhanced products such as high-dose inactivated or adjuvanted vaccines," while younger adults received standard-dose vaccines. However, Shahar points out that this explanation falls short, as the "remarkable benefit of ‘enhanced products’ was only observed for hospitalization," not for outpatient encounters. The inconsistent protective effect across severity levels, despite the enhanced vaccines, renders these findings untrustworthy without further, clearer biological or epidemiological explanation.
The Outcome of Hospitalized Flu Patients: A Troubling Observation
Perhaps one of the most stark and concerning observations highlighted by Shahar is the analysis of in-hospital outcomes for influenza-positive patients, particularly among the elderly. In this nested cohort design, the study compared the outcomes, including death, of hospitalized flu patients based on their vaccination status. Shahar focuses on the elderly population (constituting almost 70% of all deaths in this cohort), noting that while vaccinated individuals were slightly older and sicker, the differences in baseline characteristics were generally small.
Crucially, the fatality rate among vaccinated elderly hospitalized patients was 50% higher than among their unvaccinated counterparts (4.0% vs. 2.7%). If the flu vaccine were genuinely beneficial in preventing severe outcomes and death in this vulnerable population, such an observed difference would imply an "extreme confounding" that would have to shift a true risk ratio of 0.5 or 0.75 (representing 50% or 25% effectiveness against death) to a biased 1.5 (representing a 50% higher risk). This would require a two- to three-fold shift on the ratio scale, a magnitude of confounding that Shahar suggests is perhaps stronger than what could be expected from the reported baseline differences in age and comorbidities.
Remarkably, the study authors dismiss this significant disparity, stating that "The percentage experiencing severe in-hospital clinical outcomes including ICU admission, receipt of IMV, or death, was similar across vaccination groups." Shahar critiques this assessment, suggesting that if one follows their reasoning and assumes no meaningful confounding, then the observed higher fatality rate in vaccinated individuals could imply that "vaccination increased the risk of death in these patients." This specific finding, coupled with the authors’ interpretation, raises profound questions about the protective claims of the flu vaccine in the most critical patient populations and underscores the potential for misinterpretation of observational data.
Broader Implications and Calls for Scientific Reform
The detailed critique of the CDC’s methodology and the specific findings of the VISION Network study have far-reaching implications, extending beyond the realm of influenza vaccination.
-
Erosion of Public Trust: When public health institutions like the CDC rely on methodologies that are questioned by leading scientists, including the NIH Director, it inevitably erodes public trust. In an era where skepticism towards scientific and governmental bodies is already heightened, particularly after the COVID-19 pandemic, maintaining transparent, robust, and reproducible scientific practices is paramount. The perception of questionable data or flawed analyses can undermine public confidence in vaccine recommendations and broader public health initiatives, leading to decreased compliance and poorer public health outcomes.
-
Impact on Public Health Policy: Annual flu vaccine campaigns are a cornerstone of seasonal public health efforts globally. If the effectiveness estimates guiding these campaigns are systematically biased upwards, policymakers may be making decisions based on an inflated understanding of the vaccine’s benefits. This could lead to misallocation of scarce public health resources, inappropriate public health messaging that overstates benefits, or a missed opportunity to develop and promote more effective interventions. Accurate, unbiased data is not merely an academic concern; it is essential for effective, evidence-based public health policy.
-
Urgent Need for Methodological Re-evaluation: The critiques underscore an urgent need for public health agencies to critically re-evaluate their standard methodologies for assessing vaccine effectiveness. Shahar’s call for alternative study designs is not
