?Mathematical formulae have been encoded as MathML and are displayed in this HTML version using MathJax in order to improve their display. Uncheck the box to turn MathJax off. This feature requires Javascript. Click on a formula to zoom.Abstract
New Zealand adopted a policy of mandatory COVID-19 vaccination for workers in many sectors. Existing analysis suggests expected costs of this mandate policy far outweigh benefits. This paper discusses an issue potentially contributing to adoption of this costly vaccine mandate policy. There is a widespread public misunderstanding about the testing the vaccines underwent in the pivotal trials underpinning their approval, with over 95% of New Zealand’s voting-age public believing that the vaccines were tested against more demanding criteria than was actually the case. Consequently, public expectations about performance of these vaccines were likely inflated, and expected benefits of vaccine mandates may have been overstated. The ambiguous evidence on effects of COVID-19 vaccination on mortality risk also highlights the importance of these informational problems. If the public misunderstanding described here persists, a continuation of inefficient vaccine mandates whose costs exceed benefits is likely.
Introduction
In 2021, New Zealand adopted a policy of mandatory COVID-19 vaccination for workers in the health, education, corrections, defence, police, and fire and emergency services sectors. Thousands lost jobs from not complying with this policy. Lally (Citation2021a) finds expected costs of these mandates far outweigh benefits; if sanctions lower quality of life of non-compliers by even just 1% per year mandates would need to prevent at least 5200 deaths from COVID-19 (that are due to the pool of unvaccinated people) to outweigh costs. Given COVID-19’s low infection-fatality rate and on-going transmission amongst the vaccinated, Lally suggests this threshold is very unlikely to be met.
In addition to mandatory vaccination in some sectors, the COVID-19 Protection Framework (traffic light system) prohibited the unvaccinated from using many facilities. Businesses that had to use vaccine passports to operate under the traffic light system (or to operate with fewer restrictions) also needed vaccine mandates for their workers, which extended employment disruptions into the private sector. No cost–benefit analysis of the traffic light system was conducted, although public health commentator Michael Baker of the University of Otago considered it coercion to promote vaccination rather than a useful tool for reducing spread of COVID-19: ‘The traffic light system … was never designed to dampen down transmission, it was only designed to nudge people towards vaccination’ (McKenzie, Citation2022). Given the traffic light system is mandate-like, Lally’s (Citation2021a) cost–benefit calculations should apply, so this policy is also likely to be inefficient.
While a High Court ruling in February 2022 overturned the mandate for police and defence personnel, the Defence Force persisted with an internal mandate, requiring all personnel to be fully vaccinated (including the third, ‘booster’, dose) or face dismissal (Sachdeva, Citation2022). In a subsequent High Court case, vaccination mandates for health and education workers were not overturned. The government eventually relaxed mandates for schools and some other sectors but several employers in these sectors persisted with their own vaccination requirements. Moreover, the traffic light system remains in place (albeit at a more relaxed ‘orange’ setting) and may return to more stringent settings in future in response to a combination of Covid-19 and influenza (Weekes, Citation2022). Hence, the economic evaluation of these novel policy interventions remains important.
A large literature on vaccine mandates pre-dates COVID-19, for things such as vaccination of healthcare workers against seasonal influenza, but many of these studies lack comprehensive comparison of costs and benefits (Imai et al., Citation2018). Instead, vaccine uptake is often the focus of evaluations (Brewer, Citation2021), and this feature of the evaluations has carried over into several studies of COVID-19 vaccine mandates as well (Karaivanov et al., Citation2021; Mills & Rüttenauer, Citation2022). These studies focus on a means to an end – vaccination rates – rather than the end itself, such as reduced mortality risk. In contrast, the Lally (Citation2021a) cost–benefit analysis was in terms of mortality, which is a more fundamental outcome when considering the efficiency of a policy.
In this paper, I take the Lally (Citation2021a) conclusions as given, and provide evidence on a factor related to COVID-19 in New Zealand that potentially contributed to adoption of costly and inefficient vaccine mandates. There is a widespread public misunderstanding about the testing the vaccines faced in pivotal trials underpinning their approval. Over 95% of the voting-age public believe the vaccines were tested against far more demanding criteria than actually is the case. A logical implication is that the public expectations about performance of these vaccines were exaggerated and consequently vaccine mandates may have been expected to yield greater benefits than is truly the case. A further contribution of the paper is to discuss the ambiguous evidence regarding the effects of COVID-19 vaccination on mortality risk.
The information issue studied here may only partially explain adoption of inefficient policies. New Zealand’s entire COVID-19 response could be described, tartly, as taking interventions that haven’t worked overseas – such as lockdowns (Allen, Citation2022; Bjørnskov, Citation2021; Gibson, Citation2022a) – and applying them even harder (Gibson, Citation2022b). Consequently, these interventions clearly fail conventional cost–benefit tests (Heatley, Citation2022; Lally, Citation2021b). A desire to lockdown harder than elsewhere, to vaccinate at a higher rate, and to mandate more widely may indicate something about national character, or the character of current leaders, that invites deeper examination in future research using more psychological approaches.
Pivotal vaccine trials
Pivotal randomized control trials (RCTs) underpinning approval of COVID-19 vaccines did not set out to, and did not, test if the vaccines prevent transmission of the SARS-CoV-2 virus. Nor did the trials test if the vaccines reduce mortality risk. A review of seven phase III trials for COVID-19 vaccines from Moderna, Pfizer/BioNTech, AstraZeneca (separately for U.S. and U.K.), Janssen, Sinopharm and Sinovac found endpoints in each case were just reduced risk of COVID-19 symptoms (Doshi, Citation2020). The RCTs were not designed to inform about protection against infection or death. Helpfully, the review quotes Tal Zaks, chief medical officer of Moderna (and Pfizer had the same setup), so claims about not testing for protection against infection nor testing for reduced mortality risk are straight from the horse’s mouth:
… Our trial will not demonstrate prevention of transmission … because in order to do that you have to swab people twice a week for very long periods and that becomes operationally untenable.
… Would I like to know that this prevents mortality? Sure, because I believe it does. I just don’t think it is feasible within the timeframe [of the trial] – too many people would die waiting for the results before we ever knew that. (Zaks, quoted in Doshi (Citation2020, p. 3))
… we are heading for vaccines that reduce severity of illness rather than protect against infection [and] provide only short-lived immunity, … as well as damaging public confidence and wasting global resources by distributing a poorly effective vaccine, this could change what we understand a vaccine to be. Instead of long-term, effective disease prevention it could become a suboptimal chronic treatment. (Godlee, Citation2020)
The second feature of the RCTs – not testing reduced mortality risk – is from poor statistical power, due to weak external validity. The main Pfizer trial had <5% of participants older than 75 years, yet more than 60% of COVID-19 deaths are in that age group. A young sample has few deaths so lacks statistical power for that outcome; initial reports on the Pfizer trial noted 15 deaths amongst vaccinees (7% more than deaths in the placebo group) – numbers too low to form firm conclusions (Doshi, Citation2021). Pfizer were not fully transparent, as documents subsequently released by the FDA show 21 deaths amongst vaccinated trial participants (as of March, 2021); 25% higher than the number of deaths in the placebo group.Footnote1 The ambiguous evidence on mortality effects of COVID-19 vaccination is discussed more in Appendix A.
In case readers think just the BMJ covered these features of the RCTs, the Journal of the American Medical Association also needed to correct a claim made by health bureaucrats Walensky, Walke and Fauci that ‘clinical trials have shown that the vaccines authorized for use in the U.S. are highly effective against COVID-19 infection, severe illness and death’ (Walensky, Walke, & Fauci, Citation2021). The basis of the correction was that the primary endpoint for the RCTs was symptoms of COVID-19; a less exacting standard than testing to show efficacy against infection, severe illness, and death (Doshi & Kaplan, Citation2021).
What the New Zealand public know
Features of the vaccine trials discussed in medical journals appear to be largely unknown by the New Zealand public. To provide evidence on this, I had the following question added to an omnibus survey (using landlines and mobile phones) of a nationally representative random sample of voting-age New Zealanders:
The vaccine for COVID-19 marketed by Pfizer is the main COVID vaccine available in New Zealand. Based on your own understanding, were the trials that allowed the authorization of this vaccine designed to:
Test if the vaccine prevents infection and transmission of SARS-CoV-2 (the virus that causes COVID-19)?
Test if the vaccine reduces the likelihood of getting symptoms of COVID-19?
Test if the vaccine reduces the likelihood of getting seriously sick and dying?
All of the above
The poll was fielded in the first week of December 2021 when Parliament voted in the traffic light system restricting the unvaccinated. Just two weeks earlier, many education and health workers had lost jobs due to vaccine mandates. The government had also just procured a small batch of the AstraZeneca vaccine for people resisting the Pfizer jab used exclusively until then. Thus, aspects of what the vaccines were designed to do and how they had been tested should have been very salient for the public at the time.
The correct answer to the survey question is option (b). The RCTs only tested if vaccination reduced risks of getting symptomatic COVID-19. Yet there were very high levels of misunderstanding, with only four percent choosing the correct option. Instead, almost all respondents believe there was also testing for protection against infection and/or for lowering risk of death. Figure shows weighted percentage responses for each answer (error bars show 95% confidence intervals).Footnote2 Based on these results, it seems that over 95% of the public believe that the vaccines were trialled against more exacting criteria than is actually the case.
Figure 1. Answers to the survey question about what pivotal trials of COVID-19 vaccines were designed to show.
Some responsibility for public misunderstanding must rest with politicians. New Zealand’s Prime Minister infamously claimed that in matters of COVID-19 and vaccines: ‘Dismiss anything else, we will continue to be your single source of truth’ (Creighton, Citation2021). This is the same politician who drew an equivalence between COVID-19 vaccination and the approach to dealing with measles (Cooke, Citation2021). Yet the measles vaccine gives durable and near-complete protection (vaccine efficacy ≥ 97%) against infection while COVID-19 vaccines provide only incidental and short-term protection against infection.
Most people get their information from (social) media rather than from medical journals, so responsibility for public misunderstanding also rests with the local media. Over six months after the BMJ articles noted above, one of the two main print media outlets in New Zealand included the following claim: ‘It [the vaccine] will prevent most if not all cases appearing and potentially prevent them becoming infected at all’ (Witton, Citation2021). Months later the same outlet claimed: ‘The Pfizer vaccine was about 95 per cent effective at preventing infection from the original Wuhan strain’ (Macdonald, Citation2021). Neither claim describes the RCTs, that were not set up to test for protection against infection. In addition to not correctly reporting the relevant scientific literature, the media promoted opinions from supposedly independent local academics who got talking points about the vaccines from the government in a carefully orchestrated public relations campaign (MacNamara, Citation2021). Even after this was revealed, the views of these commentators continued to be promoted, with no disclaimer about the reliance on government-provided talking points and other potential conflicts of interest.
Conclusions
The ‘policy implications’ section of papers written by academic economists often includes calls for information interventions, whether needed or not. These seem unobjectionable, low-cost and give analyses written by people far-removed from policy-making a practical looking sheen. However, in this particular example, an information intervention is exactly what is needed. Somehow or other, the New Zealand public wrongly gained the impression that the COVID-19 vaccines were trialled against a more comprehensive and demanding set of criteria than was actually the case.
One can think of Myrdal’s idea of circular cumulative causation at play here. Whether in hope or in ignorance, politicians (and public health bureaucrats, who increasingly resemble politicians) overstated the testing criteria used for the vaccines. A trusting public believe what they are told, and so expect the vaccines to be effective in protecting against SARS-CoV-2 infection and in reducing mortality risk. Politicians who rely on focus groups and internal polling when considering things like vaccine mandates then find that a misinformed public supports these interventions. The imposition of the mandates is likely to have further added to the misunderstanding because, surely, a kind and caring government would only mandate something that is in people’s best interests because it is ‘safe and effective’.
Observant readers will note there is no mention in this paper of omicron, or other variants. The short-term and incidental protection against SARS-CoV-2 infection is a feature of the COVID-19 vaccines that applies both to the original Wuhan strain the trials were based on and to subsequent variants. Indeed, as early as October 2021, prior to the vaccine mandate and traffic light system being introduced in New Zealand, Pfizer’s own scientists published research (using an observational sample of 3.5 million people covered by a large U.S. health insurer) showing that vaccine efficacy against SARS-CoV-2 infection fell by almost ten percentage points per month, irrespective of the variant (Tartof et al., Citation2021). Likewise, the ambiguous evidence on mortality effects of COVID-19 vaccination discussed in Appendix A is not specific to any of the variants. If the public misunderstanding described here persists, a prolonged era of costly policies is likely.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes
1 Retrieved from: https://www.fda.gov/media/151733/download. The 95% confidence interval of the odds ratio is 0.7–2.4 (above 1.0 indicates more deaths in vaccinees). Amongst 1000 bootstrap resamples of trial replicate data (for 38 deaths amongst n= 42,267 participants), 78% of replications show more deaths in the vaccinees.
2 The pollster (Curia Market Research) weighted the n=852 responses to represent the voting-age adult population. The sample size gives a maximum 95% confidence interval (CI) of 3.1%, for an outcome with 50:50 odds. Response-specific CIs are calculated from https://sample-size.net/confidence-interval-proportion/.
3 To see this, consider an analogy with international migration; another treatment that has duration-dependent heterogeneity (Gibson, McKenzie, & Stillman, Citation2013). When a program like a visa lottery is analysed, the treatment group is all of the lottery-selected immigrants (and perhaps their extended family in the source area as an outcome of interest), even those just arrived in the destination country who may (temporarily) be in a bad financial situation as they repay moving costs (McKenzie, Gibson, & Stillman, Citation2007, Citation2010). If instead the treatment group was redefined as those who had moved and had successfully adapted to life in the destination country it would provide an overstated estimate of average impact. A similar bias is likely when studies select within vaccinees based on time since last dose, given that there appears to be a duration-dependent heterogeneity in impacts of the COVID-19 vaccines.
4 The all-cause deaths data are from the Németh, Jdanov, and Shkolnikov (Citation2021) Short Term Mortality Fluctuations database. The average age of the vaccinated cohort in the Nordström et al. (Citation2022) study is 59 (±19 years) and so excluding the 0–14 years age group from the Németh et al database should represent approximately the same population.
5 A further reason for ruling out reverse causality is that all-cause death rates and excess mortality are usually not reported until some weeks or months after the fact so there is likely to be a lack of real-time awareness of greater mortality risk that otherwise could potentially act as a spur to vaccination. The daily media counts of COVID-19 deaths reported during the pandemic are highly unusual with no similar up-to-date reporting of total deaths or deaths from most other causes (except perhaps the road toll). For example, amongst the 36 OECD countries whose excess mortality data are used in the chart below, the lag in the reporting of the excess mortality rates is such that, as of late April 2022, the most recent month that all of the countries had data available was January 2022, so there was a lag of three months. Only one-half of countries had excess mortality data available for the prior month and so it is hard to see how awareness of these data could act as a contemporaneous cause of COVID-19 vaccination rates.
6 As of the end of 2020, the 36 OECD countries studied here had administered less than one-third of a percent of the vaccine doses that they have administered to date, so it is reasonable to consider 2020 as unvaccinated time.
7 Data for New Zealand deaths (1948 onwards) are from https://minhealthnz.shinyapps.io/mortality-web-tool/
References
- Abu-Raddad, L., Chemaitelly, H., Ayoub, H., AlMukdad, S., Yassine, H., Al-Katib, H., … Al-Kanaani, Z. (2022). Effect of mRNA vaccine boosters against SARS-CoV-2 Omicron infection in Qatar. New England Journal of Medicine, (online early view). doi:10.1056/NEJMoa2200797
- Allen, D. (2022). COVID-19 lockdown cost/benefits: A critical assessment of the literature. International Journal of the Economics of Business, 29(1), 1–32.
- Auld, C., & Toxvaerd, F. (2021). The great COVID-19 vaccine rollout: Behavioral and policy responses. National Institute Economic Review, 257(1), 14–35.
- Benn, C., Schaltz-Buchholzer, F., Nielsen, S., Netea, M., & Aaby, P. (2022). Randomised clinical trials of COVID-19 vaccines: Do adenovirus-vector vaccines have beneficial non-specific effects? Preprints with the Lancet. doi:10.2139/ssrn.4072489
- Bjørnskov, C. (2021). Did lockdown work? An economist’s cross-country comparison. CESifo Economic Studies, 67(3), 318–331.
- Brewer, N. (2021). What works to increase vaccination uptake? Academic Pediatrics, 21(4), S9–S16.
- Cohn, B., Cirillo, P., Murphy, C., Krigbaum, N., & Wallace, A. (2022). SARS-CoV-2 vaccine protection and deaths among US veterans during 2021. Science, 375(6578), 331–336.
- Cooke, H. (2021, August 13). Border-opening trial this year, vaccine speed-up. The Dominion Post. https://www.pressreader.com/new-zealand/the-dominion-post/20210813/281689732877170(open in a new window).
- Creighton, A. (2021, July 20). Suppression plans straight from CCP play book. The Australian. https://www.theaustralian.com.au/commentary/suppression-plans-straight-from-ccp-play-book/news-story/71242afe71048b13dd88fe3f4c2fafd9(open in a new window).
- Doshi, P. (2020). Will COVID-19 vaccines save lives? Current trials aren’t designed to tell us. BMJ, 371, m4037.
- Doshi, P. (2021, August 23). Does the FDA think these data justify the first full approval of a Covid-19 vaccine. BMJ Opinion. https://blogs.bmj.com/bmj/2021/08/23/does-the-fda-think-these-data-justify-the-first-full-approval-of-a-covid-19-vaccine/(open in a new window)
- Doshi, P., Godlee, F., & Abbasi, K. (2022). COVID-19 vaccines and treatments: We must have raw data, now. BMJ, 376, o102.
- Doshi, P., & Kaplan, R. (2021). Inaccurate statement. JAMA, 325, 1039.
- Gibson, J. (2022a). Government mandated lockdowns do not reduce COVID-19 deaths: Implications for evaluating the stringent New Zealand response. New Zealand Economic Papers, 56(1), 17–28.
- Gibson, J. (2022b). Hard, not early: Putting the New Zealand COVID-19 response in context. New Zealand Economic Papers, 56(1), 1–8.
- Gibson, J., McKenzie, D., & Stillman, S. (2013). Accounting for selectivity and duration-dependent heterogeneity when estimating the impact of emigration on incomes and poverty in sending areas. Economic Development & Cultural Change, 61(2), 247–280.
- Godlee, F. (2020). COVID-19: Less haste, more safety. BMJ, 370, m3258.
- Heatley, D. (2022). A social cost–benefit framework for COVID-19 policy decisions. New Zealand Economic Papers, 56(1), 41–48.
- Imai, C., Toizumi, M., Hall, L., Lambert, S., Halton, K., & Merollini, K. (2018). A systematic review and meta-analysis of the direct epidemiological and economic effects of seasonal influenza vaccination on healthcare workers. PloS one, 13(6), e0198685.
- Karaivanov, A., Kim, D., Lu, S., & Shigeoka, H. (2021). COVID-19 vaccination mandates and vaccine uptake. Working Paper 29563, National Bureau of Economic Research.
- Lally, M. (2021a). A cost-benefit analysis of Covid-19 vaccine mandates. mimeo https://www.wgtn.ac.nz/__data/assets/pdf_file/0011/1983899/r-covid-vaccine-mandates.pdf(open in a new window).
- Lally, M. (2021b). The costs and benefits of COVID-19 lockdowns in New Zealand. medRxiv. doi:10.1101/2021.07.15.21260606
- Macdonald, N. (2021, September 14). Covid-19: To boost or not to boost? The Press. https://www.stuff.co.nz/national/health/coronavirus/126331885/covid19-to-boost-or-not-to-boost(open in a new window).
- MacNamara, K. (2021, November 2). The suffocating spin of NZ’s vaccine taskforce. New Zealand Herald. https://www.nzherald.co.nz/business/kate-macnamara-the-suffocating-spin-of-nzs-vaccine-taskforce/YGMWBQ53GRTUVLLK3TFDW5ZMVQ/(open in a new window).
- Mathieu, E., Ritchie, H., Ortiz-Ospina, E., Roser, M., Hasell, J., Appel, C., … Rodés-Guirao, L., (2021). A global database of COVID-19 vaccinations. Nature Human Behaviour, 5(1), 947–953.
- McKenzie, D., Gibson, J., & Stillman, S. (2007). Moving to opportunity, leaving behind what? Evaluating the initial effects of a migration policy on incomes and poverty in source areas. New Zealand Economic Papers, 41(2), 197–223.
- McKenzie, D., Gibson, J., & Stillman, S. (2010). How important is selection? Experimental vs. non-experimental measures of the income gains from migration. Journal of the European Economic Association, 8(4), 913–945.
- McKenzie, P. (2022, January 10). New Zealand not prepared for Omicron outbreak expected in ‘matter of weeks’, experts warn. The Guardian. https://www.theguardian.com/world/2022/jan/10/new-zealand-not-prepared-for-omicron-outbreak-expected-in-matter-of-weeks-experts-warn(open in a new window).
- Meslé, M., Brown, J., Mook, P., Hagan, J., Pastore, R., Bundle, N., … Dykhanovska, T. (2021). Estimated number of deaths directly averted in people 60 years and older as a result of COVID-19 vaccination in the WHO European region, December 2020 to November 2021. Eurosurveillance, 26(1), 2101021.
- Mills, M., & Rüttenauer, T. (2022). The effect of mandatory COVID-19 certificates on vaccine uptake: Synthetic-control modelling of six countries. The Lancet Public Health, 7(1), 15–22.
- Neil, M., Fenton, N., Smalley, J., Craig, C., Guetzkow, J., McLachlan, S., … Rose, J. (2022). Official mortality data for England suggest systematic mis-categorisation of vaccine status and uncertain effectiveness of COVID-19 vaccination. Working Paper QMUL. doi:10.13140/RG.2.2.28055.09124
- Németh, L., Jdanov, D., & Shkolnikov, V. (2021). An open-sourced, web-based application to analyse weekly excess mortality based on the short-term mortality fluctuations data series. PloS One, 16(2), e0246663.
- Nordström, P., Ballin, M., & Nordström, A. (2022). Risk of infection, hospitalisation, and death up to 9 months after a second dose of COVID-19 vaccine: A retrospective, total population cohort study in Sweden. The Lancet, 399(10327), 814–823.
- Sachdeva, S. (2022, April 22). Defence Force staff face dismissal over vaccine mandate. Newsroom. https://www.newsroom.co.nz/defence-force-staff-face-dismissal-over-vaccine-mandate(open in a new window).
- Tartof, S., Slezak, J., Fischer, H., Hong, V., Ackerson, B., Ranasinghe, O., … McLaughlin, J. (2021). Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: A retrospective cohort study. The Lancet, 398(10309), 1407–1416.
- Walensky, R., Walke, H., & Fauci, A. (2021). SARS-CoV-2 variants of concern in the United States – challenges and opportunities. JAMA, 325, 1037.
- Weekes, J. (2022, April 14). Covid 19 Omicron: Orange day dawns after curve flattened and return to office looms for many. New Zealand Herald. https://www.nzherald.co.nz/nz/politics/covid-19-omicron-orange-day-dawns-after-curve-flattened-and-return-to-office-looms-for-many(open in a new window).
- White, E., Yang, X., Blackman, C., Feifer, R. A., Gravenstein, S., & Mor, V. (2021). Incident SARS-CoV-2 infection among mRNA-vaccinated and unvaccinated nursing home residents. New England Journal of Medicine, 385(5), 474–476.
- Witton, B. (2021). Coronavirus: Vaccine hesitancy will fade as more information comes to light – experts. The Press, February 2. https://www.stuff.co.nz/national/health/coronavirus/124119830/coronavirus-vaccine-hesitancy-will-fade-as-more-information-comes-to-light–experts(open in a new window).
A
Ambiguous evidence on mortality effects of COVID-19 vaccination
The main text notes that there were 25% more deaths amongst vaccinees (as of March 2021) than amongst control group members in the pivotal Pfizer trial but there were too few deaths to provide much confidence about population proportions; there is a 22% risk of Type I error if the trial evidence is used to conclude that population-wide mortality would be 25% higher following universal use of the BNT162b2 vaccine. This unsatisfactory state of the evidence is due to the poor external validity of the trials, whose participants mostly were younger and middle-aged adults who have very low risk of death from COVID-19 (or from other causes).
Attempts to provide further evidence on the mortality effects of COVID-19 vaccination use at least four approaches. The first is to gain statistical power by pooling across the various trials used for the vaccines. In a simple meta-analysis, Benn, Schaltz-Buchholzer, Nielsen, Netea, and Aaby (Citation2022) pool the data from the published results for the pivotal RCTs for the adenovirus-vector vaccines (AstraZeneca, Johnson & Johnson, Sputnik) and for the mRNA vaccines (Moderna and Pfizer). They find 103 deaths (from all causes) amongst the mRNA vaccinees for every 100 deaths amongst the control group members and the 95% confidence interval ranges from 63 to 171. Hence, there is no evidence of any overall reduction in mortality from using the mRNA vaccines, with the lower risk of COVID-19 deaths offset by higher risk of cardiovascular deaths (although neither disease-specific effect is statistically significant). With adenovirus-vector vaccines, vaccinees had 37 deaths per 100 deaths for control group members, with the 95% confidence interval from 19 to 70. A lot of this is through lower non-COVID mortality, particularly in the Johnson & Johnson trial that had a shorter follow-up time than the other trials. In addition to this limitation, lack of access to the raw data means that issues of unbalanced protocol deviations between treatment and control arms in the trials and other possible threats to internal validity cannot be accounted for. For these reasons, there are ongoing calls for release of the raw data from the pivotal RCTs for the COVID-19 vaccines (Doshi, Godlee, & Abbasi, Citation2022).
The second approach to studying mortality effects of COVID-19 vaccination applies reported relative risk reduction rates from the RCTs to data on vaccine uptake, to calculate how many lives are saved by the vaccination effort. For example, Meslé et al. (Citation2021) claim vaccination averted almost 0.5 million deaths in old people in the WHO European Region, where this was calculated by combining weekly data on actual deaths with country-specific vaccine uptake rates and using an assumed 95% relative risk reduction rate (where this parameter is linked back to the pivotal RCTs). There are at least two problems with this type of approach, which effectively just assumes the answer rather than empirically estimating an actual effect. First, the relative risk reduction reported by the RCTs was for symptomatic COVID-19, and not for deaths. Second, any spillovers into deaths from other causes are ignored, yet the RCTs suggest that vaccination affects other types of deaths, such as cardiovascular deaths.
A third way of studying effects of COVID-19 vaccination uses observational data to match vaccinated and unvaccinated individuals, often from population registers or health system administrative data. This type of study is used widely (albeit often focussed on infections not mortality). For example Nordström, Ballin, and Nordström (Citation2022) tracked 1.7 million Swedes in a national registry from January to October 2021, finding peak vaccine efficacy (VE) against infection of 92% for the Pfizer vaccine at 2–4 weeks post second dose, falling to 47% by month 4-6, and zero from month 7 onwards. The VE for severe outcomes (hospitalization and death) fell less rapidly but six months after second dose one cannot rule out zero effect (partly due to confidence intervals for these outcomes being so wide, even with population registry data). Cohn, Cirillo, Murphy, Krigbaum, and Wallace (Citation2022) track 0.78 million U.S. veterans (of whom 48% were age 65+) from February to October 2021, using a time-varying Cox proportional hazards model to estimate Kaplan-Meier survival curves for the unvaccinated and for those with two doses of Pfizer or Moderna or one dose of Janssen. After four months, survival odds for the unvaccinated aged 65 + had fallen below 0.9, while for the vaccinated they were about 0.95 at that time. Unlike the RCTs, having a big share of elderly ensured enough all-cause deaths (ca. 22,000) to have precise estimates. In this same study, VE against SARS-CoV-2 infection fell below 0.5 for Janssen and Pfizer vaccines after six months. A variant of these studies compares within the vaccinees, such as Abu-Raddad et al. (Citation2022) who form a cohort of 2.2 million people in Qatar who had received at least two doses of mRNA vaccines, with matching and 35 day follow-up carried out on 0.5 million who either had or had not received a third, booster, dose.
There are at least two concerns with these studies, from the standpoint of economics. First, they assume ‘selection on observables’ where choice of getting vaccinated is due to attributes visible to the researchers in their databases. If unobservable factors – e.g. risk preferences or personal beliefs – affect this choice and also affect health outcomes, there will be bias in combining the effect of the unobservables with the effect of the treatment. Some people have borne substantial costs, such as job loss, in order to remain unvaccinated so unobservables are likely to be important drivers of their behaviour and so it seems unwise to ignore them. It is for this reason that RCTs are used, as randomization should ensure that unobservables are, on average, the same between the treatment group and control group. Yet despite the importance that economists place on bias due to unobservables, with Nobel prizes in 2021, 2019 and 2000 (Heckman’s half-share) related to research designs and statistical methods designed to mitigate this issue, it is ignored in studies that compare vaccinees with the unvaccinated.
The second concern is that the treated group in these studies is the subset of vaccinees that survive for one-week or two-weeks post vaccination, biasing estimates of the effects of vaccination per se.Footnote3 For example, Nordström et al. (Citation2022) define the treated group as people whose second dose was at least 15 days earlier while Abu-Raddad et al. (Citation2022) define it as those whose third dose was at least 7 days earlier. This matters for two reasons: impacts from the excluded group may exceed apparent impacts calculated from differences between the treatment and comparison group; and second, adverse outcomes occurring immediately after vaccination may be ignored or wrongly attributed to the untreated group, overstating apparent vaccine efficacy. For the first effect, the Nordström et al. (Citation2022) study had 277 COVID-19 hospitalizations or deaths (the two outcomes are not separated) in the treatment group during the follow-up period (which lasted 4 months, on average) while the comparison group had 825 hospitalizations or deaths. Even under an extreme assumption, that these are all deaths and not hospitalizations, the treatment effect (the difference in totals between the two groups) is 548 deaths; a small number relative to 3939 vaccinated individuals dying within 14 days of their second dose and who were therefore excluded from the study by defining treatment as getting a second dose and surviving at least 15 days.
It is a pity Nordström et al. (Citation2022) do not discuss these 3939 deaths post-vaccination, which show a very elevated mortality risk. These deaths were amongst 4.03 million double-dosed people in the total cohort (matching with the unvaccinated used a random subset), amounting to one death per 1024 people per fortnight (an annual death rate of 2.6 percentage points). Sweden’s weekly all-cause mortality during 2015–2019 (so, pre-COVID), for ages 15 and above in the April to August period that corresponds to the second dose rollout, averaged one death per 2580 people per fortnight (annual death rate of one percent).Footnote4 Even in 2020, when COVID-19 deaths elevated all-cause mortality, the average was one death per 2220 people and the week with the highest mortality corresponded to one death per 1670 people. When set against these background death rates, the 3939 deaths in the two weeks after the second dose appear to be very high and so it is a pity that the analysis ignores them.
The second issue, of either ignoring or misallocating outcomes in the period immediately after a dosing episode, and thereby affecting calculated VE, can be illustrated using a study by White et al. (Citation2021) of SARS-CoV-2 infections in nursing homes. This study has the convenient feature of reporting outcomes in the 0–14 days period post-dose, along with outcomes in later periods, whereas many other studies omit to report outcomes in the 0–14 days period. For the n = 18242 vaccinated residents, 822 had infections in the first 14 days post-dose and 250 had infections in the subsequent period, for a total of n = 1072 who were infected. For the n = 3990 residents who were unvaccinated, 173 had infections in the 0–14 days period and 69 in the subsequent period, for n = 242 across both periods. If the vaccination event is the treatment, the relative risk reduction (RRR) is: . In other words, vaccination provided almost no reduction in infection risk. Yet if the post-vaccination period is not counted, as happens if the treatment group is defined as people whose dose was at least 15 days ago, the RRR appears to be 0.21. The RRR seems even higher, at 0.95, if the people dosed less than 15 days ago are considered as unvaccinated, as their adverse outcomes (an increased risk of infection) in the 0–14 day period are attributed to the untreated group.
In the Cohn et al. (Citation2022) study of U.S. veterans, which uses time-varying vaccination status, the 0–14 day period post-vaccination is treated as unvaccinated time. This study reports that survival probability fell fastest for the ‘unvaccinated’ (as they define them, to include the first 14 days post-vaccination) in the first two weeks (a six percentage point drop for the age 65 + group, and an 0.06 drop for those <65). Hence, it would help to have another analysis, where days 0–14 contribute to the hazard rate for the vaccinated time phase rather than being treated as unvaccinated time. More generally, Neil et al. (Citation2022) show how delayed death reporting as vaccine rollout is ramping up will lead any vaccine, even a placebo, to seemingly reduce mortality. The same statistical illusion will be created if the death of a person occurring in the same week as that person is vaccinated is treated as an unvaccinated, rather than a vaccinated, death. This bias, along with others, is shown by these authors to affect the official vaccine mortality surveillance reports from the U.K. Office of National Statistics.
Given these issues with the observational studies that match vaccinees with the unvaccinated, another approach to studying the mortality effects of COVID-19 vaccination is to use aggregate data. There is neither a random assignment mechanism available nor a plausible source of exogenous variation that could be used as an instrumental variable in cross-country studies, so the empirical relationships have to be interpreted as conditional correlations. Nevertheless, the concern that correlations either reflect vaccination rates as consequences rather than as causes (i.e. reverse causality) or else reflect the impact of omitted factors, can at least partly be mitigated by using panel data with two-way fixed effects (as in Auld & Toxvaerd, Citation2021) to control for time-invariant unobserved country factors and space-invariant temporal factors that might otherwise confound the relationships.Footnote5 Also, one can guard against basing conclusions on the post hoc ergo propter hoc (after this, therefore because of this) fallacy by working with excess mortality data, which accounts for the expected number of deaths in each time period of the year in each country (as derived from seasonal patterns observed during 2015–2019, which is prior to any impact of COVID-19 on mortality). With such data, one can relate COVID-19 vaccination rates to deviations from the historical mortality pattern for each country, to counter the argument that people are always dying so a post-vaccination death need not be due to the vaccine as some people would have died anyway. Moreover, even if COVID-19, and responses to it such as quarantining arrivals from overseas, shift seasonal mortality patterns so that deaths during 2015–2019 are now less useful for calculating excess mortality, the fact that the pandemic has lasted over two years with the first year almost entirely unvaccinated and the second year having mass rollout of the vaccines lends itself to a form of difference-in-differences analysis.Footnote6 Specifically, the change in excess mortality between a given month in 2020 (with no vaccines available) and the same month in 2021 (with vaccines available) can be related to the vaccination rate. One would expect times and places with higher vaccination rates to have reduced excess mortality if the vaccines are, on net, saving lives.
Figure shows changes between 2020 and 2021 in same-month average excess mortality p-scores (the percentage by which all-cause deaths deviate from expected deaths). In other words, March 2020 is compared with March 2021, May with May and so on. The data are for the 36 OECD countries (excluding Costa Rica and Columbia who joined in 2020 and 2021). Overall, the excess mortality p-scores were 1.1 percentage points higher in 2021, with the second year of the pandemic having a greater overall death toll for these countries. In these countries, across the country-month averages for 2021, the mean vaccination rate is 85 doses per 100 people, ranging from 0 to 220, according to Our World in Data (Mathieu et al., Citation2021).
Figure A1. Aggregate effects of COVID-19 vaccination: OECD countries.
There is a positive and statistically significant relationship between COVID-19 vaccination rates and the change in excess mortality p-scores. If a country-month averages one more dose per 100 people, the excess mortality p-score in 2021 is 0.3 points higher than in 2020. Thus, the times and places that were more heavily vaccinated have a bigger rise in excess mortality from 2020 to 2021, controlling for unobservable attributes of countries and of time periods (with these fixed effects partialled out in Figure A1). If the vaccination rate is lagged one month, to further rule out the possibility of reverse (contemporaneous) causation, the slope is a little lower but still with similar levels of statistical significance, at 0.21 ± 0.09. These standard errors are clustered at country level, given the panel structure of the data.
In addition to the unexpected pattern of a higher vaccination rate being related to a bigger rise in excess mortality, the fact that excess mortality in 2021 was higher than in 2020 is notable. A priori, one might expect the opposite because 2021 had an additional tool – vaccines – not available in 2020. Adding an option should not impair overall performance. Moreover, lower mortality in the second year of a pandemic than in the first year might be expected if the most vulnerable (the elderly and those with co-morbidities) had already perished. This is consistent with what was seen in the two prior pandemics to affect New Zealand, the H2N2 influenza (the ‘Asian flu’) in 1957–1958 and the H3N2 influenza (the ‘Hong Kong flu’) in 1968–1969. In those two pandemics, deaths in the second year were 2.7% (1.2%) lower than in the first year based on 1957–1958 (1968–1969) all-causes mortality data.Footnote7 Overall then there are several puzzles about mortality during the COVID-19 pandemic, and evidence for the effects of vaccination on mortality risk is far more ambiguous than public discussion seems to suggest.