1 Citing an earlier version of this article, the Australian Bureau of Statistics implemented our recommended approach in CPI construction, starting from the December Quarter of 2017; see the Australian Bureau of Statistics (2017 Australian Bureau of Statistics (2017), “An Implementation Plan to Maximise the Use of Transactions Data in the CPI,” Information Paper 6401.0.60.004, Canberra. [Google Scholar], 2018 Australian Bureau of Statistics (2018), “6401.0—Consumer Price Index, Australia, Dec 2017,” Media Release, available at http://www.abs.gov.au/ausstats/abs@.nsf/lookup/6401.0Media/%20Release1Dec/%202017. [Google Scholar]). Belgium also implemented a variant from 2020; see Van Loon and Roels (2019 Van Loon, K., and Roels, D. (2019), “Évaluation des méthodes multilatérales de calcul de l’indice,” economie No. 06, StatBel, available at https://statbel.fgov.be/en/news/update-consumer-price-index-cpi-year-2020. [Google Scholar]). See Diewert and Fox (2020 Diewert, W. E., and Fox, K. J. (2020), “Measuring Real Consumption and CPI Bias Under Lockdown Conditions,” NBER Working Paper 27144, Cambridge, MA, available at https://www.nber.org/papers/w27144.[Crossref] , [Google Scholar]) on the benefits of scanner data in CPI construction under pandemic conditions.

2 Using Adobe Analytics data on online transactions from January 2014 to September 2017, Goolsbee and Klenow (2018 Goolsbee, A. D., and Klenow, P. J. (2018), “Internet Rising, Prices Falling: Measuring Inflation in a World of E-Commerce,” American Economic Review, Papers and Proceedings, 108, 488–492. DOI: https://doi.org/10.1257/pandp.20181038.[Crossref] , [Google Scholar], p. 490) found that “roughly half of the sales volume online is for products that did not exist in the previous year. Even without apparel, the figure is 44%. The products that disappear, meanwhile, had about 24% of total sales before they left the market (22% excluding apparel).”

3 Given that it is a variant of the GEKS approach advocated by Ivancic, Diewert and Fox (2011 Ivancic, L., Diewert, W. E., and Fox, K. J. (2011), “Scanner Data, Time Aggregation and the Construction of Price Indexes,” Journal of Econometrics, 161, 24–35.[Crossref], [Web of Science ®] , [Google Scholar]) in the scanner data context, it is sometimes called “GEKS-Törnqvist.

4 Ivancic, Diewert, and Fox (2009 Ivancic, L., Diewert, W. E., and Fox, K. J. (2009), “Scanner Data, Time Aggregation and the Construction of Price Indexes,” Discussion Paper 09-09, Department of Economics, University of British Columbia, Vancouver, Canada. [Google Scholar], 2011 Ivancic, L., Diewert, W. E., and Fox, K. J. (2011), “Scanner Data, Time Aggregation and the Construction of Price Indexes,” Journal of Econometrics, 161, 24–35.[Crossref], [Web of Science ®] , [Google Scholar]) used the GEKS and the WTPD multilateral methods. Khamis (1970 Khamis, S. H. (1970), “Properties and Conditions for the Existence of a New Type of Index Number,” Sankhya B, 32, 81–98. [Google Scholar], pp. 83–85, 1972, p. 101) noted that multilateral indexes could be applied to time series.

5 In practice, the reference basket is often determined by the last survey of household expenditures.

6 A superlative index number formula has the property that it is exactly equal to a Konüs (1924 Konüs, A. A. (1924), “The Problem of the True Index of the Cost of Living” (translated in 1939), Econometrica, 7, 10–29.[Crossref] , [Google Scholar]) true cost of living index provided that the purchasing households have preferences that can be represented by certain functional forms, where these functional forms can approximate arbitrary preferences to the accuracy of a second-order approximation.

7 This index requires that all prices in each period are positive, which is a restrictive assumption. Implicitly, we assume that if say commodity n is unavailable in period t so that $q_n^t=0$, then there is a positive reservation price $p_n^t>0$ that will induce potential purchasers to demand a zero amount of the commodity. This methodological approach to new and disappearing goods follows Hicks (1940 Hicks, J. R. (1940), “The Valuation of the Social Income,” Economica, 7, 105–140. DOI: https://doi.org/10.2307/2548691.[Crossref] , [Google Scholar], p. 114). The practical problem facing price statisticians is: how exactly are these reservation prices to be determined? See, for example, Diewert and Feenstra (2017 Diewert, W. E., and Feenstra, R. (2017), “Estimating the Benefits and Costs of New and Disappearing Products,” Vancouver School of Economics Discussion Paper 17–10, University of British Columbia. [Google Scholar]), Diewert, Fox, and Schreyer (2019 Diewert, W. E., Fox, K. J., and Schreyer, P. (2019), “Experimental Economics and the New Goods Problem,” Discussion Paper 19-03, Vancouver School of Economics, University of British Columbia. [Google Scholar]), and Brynjolfsson et al. (2019 Brynjolfsson, E., Diewert, W. E., Eggers, F., Fox, K. J., and Gannamaneni, A. (2019), “The Digital Economy, GDP and Consumer Welfare: Theory and Evidence,” NBER Working Paper 25695, Cambridge, MA, available at https://www.nber.org/papers/w25695. [Google Scholar], 2020 Brynjolfsson, E., Diewert, W. E., Eggers, F., Fox, K. J., and Gannamaneni, A. (2020), “Measuring the Impact of Free Goods on Real Household Consumption,” American Economic Association—Papers & Proceedings, 110, 25–30.[Crossref] , [Google Scholar]) for possible solutions.

8 The U.S. Bureau of Labor Statistics uses the Törnqvist price index as its target index for its chained CPI; see Bureau of Labor Statistics (2007 Bureau of Labor Statistics (2007), BLS Handbook of Methods: Ch 17. The Consumer Price Index, Washington, DC: Bureau of Labor Statistics, available at https://www.bls.gov/opub/hom/pdf/homch17.pdf. [Google Scholar]).

9 See Alterman, Diewert, and Feenstra (1999 Alterman, W. F., Diewert, W. E., and Feenstra, R. C. (1999), International Trade Price Indexes and Seasonal Commodities, Washington, DC: Bureau of Labor Statistics. [Google Scholar], pp. 61–65) for cases when the Törnqvist index P_T will satisfy the circularity test. Specifically, they show that if the log of price ratios trends linearly with time, and the expenditure shares also trend linearly with time, then the Törnqvist index will satisfy the circularity test exactly.

10 See the online appendix for a simple numerical example. Szulc (1983 Szulc, B. J. (1983), “Linking Price Index Numbers,” in Price Level Measurement, eds. W. E. Diewert and C. Montmarquette, Ottawa: Statistics Canada, pp. 537–566. [Google Scholar], p. 548) described relative prices as bouncing when relative price changes are first positive, then negative (or vice versa). He showed that with chaining this can cause an increase in the spread between Paasche and Laspeyres indexes. This is due to the chained Laspeyres exceeding the fixed base Laspeyres, while the chained Paasche is below the fixed base Paasche. Reducing this spread is seen as attractive, as the true price change should lie between the bounds of the Paasche and Laspeyres indexes. Normally chaining reduces this spread, but not if price bouncing is a dominant feature of the data.

11 This is due to Walsh (1901 Walsh, C. M. (1901), The Measurement of General Exchange Value, New York: Macmillan. [Google Scholar], p. 389, 1921, p. 540).

12 There is a possible fourth method to simply compute a sequence of 12 year over year monthly indexes, so that say January prices in the previous year would be compared with January prices in the current year and so on. Handbury, Watanabe and Weinstein (2013) used this methodological approach and it was recommended by the ILO (2004, chap. 22) as a valid year over year index that would avoid seasonality problems. However, central banks and other users require month to month CPIs.

13 A strongly seasonal commodity is one that is not present in the marketplace for all months of the year. Having an augmented year of at least 13 months ensures that there are at least two months in each window in which the strongly seasonal goods can appear and hence have their inflation measured between these two months, for example, January to January.

14 Ivancic, Diewert, and Fox (2011 Ivancic, L., Diewert, W. E., and Fox, K. J. (2011), “Scanner Data, Time Aggregation and the Construction of Price Indexes,” Journal of Econometrics, 161, 24–35.[Crossref], [Web of Science ®] , [Google Scholar], p. 33, footnote 19): “While a RWGEKS index, such as the RYGEKS, will not satisfy transitivity [i.e., circularity] in practice and hence will be potentially subject to chain drift, comparisons within each window are transitive. Using this approach, chain drift is therefore unlikely to be a significant problem in any context likely to be faced by a statistical agency. Also, alternative approaches to linking the indexes could be investigated, such as using different overlapping periods for doing the linking, taking the geometric mean of overlapping comparisons in multiple windows, and so forth. The most obvious approach is pursued in this article and works well in our empirical applications. An investigation into alternative approaches is left for future research.” We return to this issue explicitly in Section 3.5. The Australian Bureau of Statistics (2017 Australian Bureau of Statistics (2017), “An Implementation Plan to Maximise the Use of Transactions Data in the CPI,” Information Paper 6401.0.60.004, Canberra. [Google Scholar]) implemented the version of the rolling window GEKS index recommended by the current article, from the fourth quarter of 2017 for twenty-eight expenditure classes, accounting for approximately 17% of the CPI weight as of April 2017. Belgium also implemented this methodology from 2020; Van Loon and Roels (2019 Van Loon, K., and Roels, D. (2019), “Évaluation des méthodes multilatérales de calcul de l’indice,” economie No. 06, StatBel, available at https://statbel.fgov.be/en/news/update-consumer-price-index-cpi-year-2020. [Google Scholar]). Statistics Netherlands computed RYGEKS indexes for some components of its CPI on an experimental basis with good results but they did not implement the method officially; see de Haan and van der Grient (2011 de Haan, J., and van der Grient, H. A. (2011), “Eliminating Chain Drift in Price Indexes Based on Scanner Data,” Journal of Econometrics, 161, 36–46. DOI: https://doi.org/10.1016/j.jeconom.2010.09.004.[Crossref], [Web of Science ®] , [Google Scholar]) and de Haan (2015a de Haan, J. (2015a), “A Framework for Large Scale Use of Scanner Data in the Dutch CPI,” Paper Presented at the 14th Meeting of the Ottawa Group, Tokyo, available at http://www.stat.go.jp/english/info/meetings/og2015/pdf/t6s11p33/_pap.pdf. [Google Scholar], 2015b de Haan, J. (2015b), “Rolling Year Time Dummy Indexes and the Choice of Splicing Method,” Room Document at the 14th Meeting of the Ottawa Group, Tokyo, available at http://www.stat.go.jp/english/info/meetings/og2015/pdf/t1s3room. [Google Scholar]). Statistics New Zealand have implemented a version of RYGEKS, with adjustments for quality change, for consumer electronics scanner data; see Krsinich (2015 Krsinich, F. (2015), “Implementation of Consumer Electronics Scanner Data in the New Zealand CPI,” Statistics New Zealand, Paper Presented at the New Zealand Association of Economists Conference, Wellington, New Zealand. [Google Scholar]).

15 de Haan and van der Grient (2011 de Haan, J., and van der Grient, H. A. (2011), “Eliminating Chain Drift in Price Indexes Based on Scanner Data,” Journal of Econometrics, 161, 36–46. DOI: https://doi.org/10.1016/j.jeconom.2010.09.004.[Crossref], [Web of Science ®] , [Google Scholar], p. 41) called the indexes (4) GEKS-Törnqvist indexes. Fox and Syed (2016 Fox, K. J., and Syed, I. (2016), “Price Discounts and he Measurement of Inflation,” Journal of Econometrics, 191, 398–406. DOI: https://doi.org/10.1016/j.jeconom.2015.12.010.[Crossref], [Web of Science ®] , [Google Scholar], p. 401) called the indexes defined by (4) CCD indexes. Caves, Christensen, and Diewert (1982 Caves, D. W., Christensen, L. R., and Diewert, W. E. (1982), “Multilateral Comparisons of Output, Input, and Productivity Using Superlative Index Numbers,” Economic Journal, 92, 73–86. DOI: https://doi.org/10.2307/2232257.[Crossref], [Web of Science ®] , [Google Scholar]) used the GEKS methodology in the quantity context; that is, they used bilateral Törnqvist quantity indexes as their basic building block rather than bilateral Törnqvist price indexes.

16 The price levels ρ^t defined by (5) satisfy the following version of the multiperiod identity test (9): for periods r, s and t, we have $({\rho }^r/{\rho }^s)({\rho }^t/{\rho }^r)({\rho }^s/{\rho }^t)=1$. The price levels ρ^t also satisfy other properties that are thought to be attractive, such as the time reversal test (which means that reversing the order of time results in the inverse of the initial price level) and the circularity test (which means that all comparisons between periods are consistent and can be compared regardless of the order of comparisons). A less obvious attractive property is that the price level ρ^t also satisfies the following proportionality test, which means that scaling all prices by the same constant results in the index being scaled by the same constant: ${\rho }^t(\lambda p^t)=\lambda {\rho }^t(p^t)$ for arbitrary scalar $\lambda >0$ where ${\rho }^t(p^t)$ is the function ${\rho }^t(p^1,\dots ,p^T,q^1,\dots ,q^T)$ defined by (5) regarded as a function of the period t price vector, p^t. Note that the period t share vector s^t and the sample average share vector $\overline{s}\equiv [{\overline{s}}_1,\dots ,{\overline{s}}_N]$ also depends on p^t.

17 Equation (7) can be used to establish the following bilateral identity test: if $p^t=p^{\tau }$ and $q^t=q^{\tau }$ (which implies $s^t=s^{\tau }$), then ${\rho }^t={\rho }^{\tau }$.

18 A normalization on the parameters such as ${\alpha }^1=0$ is required to identify the parameters.

19 See also Diewert (2004 Diewert, W. E. (2004), “On the Stochastic Approach to Linking the Regions in the ICP,” Department of Economics, Discussion Paper 04-16, University of British Columbia, Vancouver, BC. [Google Scholar], 2005 Diewert, W. E. (2005), “Weighted Country Product Dummy Variable Regressions and Index Number Formulae,” The Review of Income and Wealth, 51, 561–571.[Crossref] , [Google Scholar]).

20 Rao (2005 Rao, D. S. P. (2005), “On the Equivalence of the Weighted Country Product Dummy (CPD) Method and the Rao System for Multilateral Price Comparisons,” Review of Income and Wealth, 51, 571–580. DOI: https://doi.org/10.1111/j.1475-4991.2005.00169.x.[Crossref] , [Google Scholar], p. 577). See the online appendix for a fairly simple definition of the optimal β_n.

21 In the context of the CES model to be discussed later, the WTPD model is consistent with CES preferences if the elasticity of substitution is either 1 or plus infinity.

22 We believe that this is a new result.

23 see Marris (1984 Marris, R. (1984), “Comparing the Incomes of Nations: A Critique of the International Comparison Project,” Journal of Economic Literature, 22, 40–57.[Web of Science ®] , [Google Scholar], p. 52) and Diewert (1999 Diewert, W. E. (1999), “Axiomatic and Economic Approaches to International Comparisons,” in International and Interarea Comparisons of Income, Output and Prices, Studies in Income and Wealth (Vol. 61), eds. A. Heston and R. E. Lipsey, Chicago: The University of Chicago Press, pp. 13–87. [Google Scholar], p. 49).

24 The usual method for obtaining a solution to Equations (14) and (15) is to iterate between these equations. Thus set $b=1_N$, a vector of ones and use Equation (15) to obtain an initial sequence for the P^t. Substitute these P^t estimates into Equation (14) and obtain a sequence of b_n estimates. Substitute these b_n estimates into Equation (15) and obtain a new sequence of P^t estimates. Continue iterating between the two systems until convergence is achieved. In the online appendix, we derive an alternative method which is more efficient.

25 Khamis (1972, p. 101) also derived equations (16) in the time series context.

26 This method for linking the two windows was suggested by Ivancic, Diewert, and Fox (2011 Ivancic, L., Diewert, W. E., and Fox, K. J. (2011), “Scanner Data, Time Aggregation and the Construction of Price Indexes,” Journal of Econometrics, 161, 24–35.[Crossref], [Web of Science ®] , [Google Scholar], p. 33). Melser (2018 Melser, D. (2018), “Scanner Data Price Indexes: Addressing Some Unresolved Issues,” Journal of Business & Economic Statistics, 36, 516–522.[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) suggested an alternative method for linking which similarly treats each period symmetrically.

27 See ABS (2016 Australian Bureau of Statistics (2016), “Making Greater Use of Transactions Data to Compile the Consumer Price Index,” Information Paper 6401.0.60.003.[Crossref] , [Google Scholar]), de Haan and van der Grient (2011 de Haan, J., and van der Grient, H. A. (2011), “Eliminating Chain Drift in Price Indexes Based on Scanner Data,” Journal of Econometrics, 161, 36–46. DOI: https://doi.org/10.1016/j.jeconom.2010.09.004.[Crossref], [Web of Science ®] , [Google Scholar]), and de Haan (2015a).

28 From January 2015, the Bureau of Labor Statistics uses this functional form to construct elementary indexes in the initial estimates of the Chained Consumer Price Index for All Urban Consumers; see Bureau of Labor Statistics (2018 Bureau of Labor Statistics (2018), Improving Initial Estimates of the Chained Consumer Price Index, Washington, DC: Bureau of Labor Statistics, available at https://www.bls.gov/opub/mlr/2018. [Google Scholar]).

29 See the empirical evidence on the magnitude of σ using Australian scanner data in Ivancic, Diewert, and Fox (2010 Ivancic, L., Diewert, W. E., and Fox, K. J. (2010), “Using a Constant Elasticity of Substitution Index to Estimate a Cost of Living Index: From Theory to Practice,” Australian School of Business Research Paper No. 2010 ECON 15, University of New South Wales, Sydney, Australia. [Google Scholar]). Alternative values of the α (distribution) parameters, which influence the relative expenditure shares, could be considered. For each of these choices, our point of interest would be in simulations which vary the elasticity of substitution, σ.

30 To simplify programming, $\sigma =1.001$ was actually used throughout for the “σ = 1” case.

31 “The quantity shifts associated with sales are dramatic. Consumers react instantaneously to discounts and purchase large quantities of the good-as a matter of fact, they hardly buy the good when it is not on sale.” de Haan and van der Grient (2011 de Haan, J., and van der Grient, H. A. (2011), “Eliminating Chain Drift in Price Indexes Based on Scanner Data,” Journal of Econometrics, 161, 36–46. DOI: https://doi.org/10.1016/j.jeconom.2010.09.004.[Crossref], [Web of Science ®] , [Google Scholar], p. 37).

32 If the Törnqvist bilateral indexes satisfied the circularity test exactly, then these differences would all be 0 and there would be no chain drift problem with the use of chained Törnqvist indexes. However, as was seen in Section 4.1, the circularity test does not hold exactly and there is a chain drift problem.

33 In the context international comparisons, this method of linking has been pursued by, for example, Hill (1999 Hill, R. J. (1999), “Comparing Price Levels Across Countries Using Minimum Spanning Trees,” The Review of Economics and Statistics, 81, 135–142. DOI: https://doi.org/10.1162/003465399767923881.[Crossref], [Web of Science ®] , [Google Scholar]) and Diewert (2013b Diewert, W. E. (2013b), “Methods of Aggregation Above the Basic Heading Level Within Regions,” in Measuring the Real Size of the World Economy: The Framework, Methodology and Results of the International Comparison Program-ICP, Washington, DC: The World Bank, pp. 121–167.[Crossref] , [Google Scholar]), and by Hill (2001 Hill, R. J. (2001), “Measuring Inflation and Growth Using Spanning Trees,” International Economic Review, 42, 167–185.[Crossref], [Web of Science ®] , [Google Scholar]) in the time series context.

34 See Diewert (2009 Diewert, W. E. (2009), “Similarity Indexes and Criteria for Spatial Linking,” in Purchasing Power Parities of Currencies: Recent Advances in Methods and Applications, ed. D. S. P. Rao, Cheltenham, UK: Edward Elgar, pp. 183–216. [Google Scholar]) for a discussion of the relative merits of the various measures.

35 This measure is a weighted generalization of the nonproportionality measure suggested by Allen and Diewert (1981 Allen, R. C., and Diewert, W. E. (1981), “Direct Versus Implicit Superlative Index Number Formulae,” Review of Economics and Statistics, 63, 430–435. DOI: https://doi.org/10.2307/1924361.[Crossref], [Web of Science ®] , [Google Scholar], p. 433).

36 See the online appendix for details of the application.

37 Available from the Kilts Center, University of Chicago Booth School of Business: https://www.chicagobooth.edu/research/kilts/datasets/dominicks.

38 The tables include additional series which are not plotted; CCDI and GEKS give almost identical results so the GEKS series are not plotted. Similarly, the chained Fisher is plotted, but not the chained Törnquist index, and neither of the fixed base index series are plotted but can be found in the tables.

39 See Diewert and Feenstra (2017 Diewert, W. E., and Feenstra, R. (2017), “Estimating the Benefits and Costs of New and Disappearing Products,” Vancouver School of Economics Discussion Paper 17–10, University of British Columbia. [Google Scholar]), Brynjolfsson et al. (2019 Brynjolfsson, E., Diewert, W. E., Eggers, F., Fox, K. J., and Gannamaneni, A. (2019), “The Digital Economy, GDP and Consumer Welfare: Theory and Evidence,” NBER Working Paper 25695, Cambridge, MA, available at https://www.nber.org/papers/w25695. [Google Scholar]), and Diewert, Schreyer, and Fox (2019) for more on the use reservation prices in these contexts.