Abstract: The conduction of a reliable statistical analysis is based on the recognition of the statistical features of the time series at stake and on the underlying probabilistic assumptions of the applied model. Our purpose is to illustrate this kind of analysis using Granger’s groundbreaking ideas. Our Monte Carlo results show that in the presence of stationary and non-stationary time series, the standard ordinary least squares inference could be misleading. We will graphically address the empirical distribution of the estimator’s bias as well as the inconvenience of using standard errors to illustrate how the true variation is underestimated. We recommend following Granger’s suggestions, which we highlight with originality using a “measurement in economics” perspective. Our quantitative exercises are replicable to the extent that we fully shared our codes in addition to using an open-access database of the seminal paper written by Nelson and Plosser (1982). Our main conclusion is simple: empirical researchers should be cautious when drawing qualitative findings based on a standard ordinary least squares inference carried out in the context of a regression analysis.
JEL Classification: C15C15,C22C22,C87C87,reliable statistical analysisreliable statistical analysis,standard OLS inferencestandard OLS inference,empirical biasempirical bias,replicabilityreplicability,measurement in economicsmeasurement in economics.
Resumen: La realización de un análisis estadístico confiable se fundamenta en el reconocimiento de las características estadísticas de las series de tiempo en juego y de los supuestos probabilísticos subyacentes del modelo aplicado. Nuestro propósito es ilustrar este tipo de análisis utilizando algunas ideas de Granger. Nuestros resultados de Monte Carlo muestran que en presencia de series de tiempo estacionarias y no estacionarias, la inferencia basada en los mínimos cuadrados ordinarios puede ser engañosa. Abordamos gráficamente la distribución empírica del sesgo del estimador y el inconveniente del uso de errores estándar que subestiman su verdadera variación. Recomendaremos seguir las sugerencias de Granger, que destacamos con originalidad utilizando una perspectiva desde la “medición en economía”. Nuestros ejercicios cuantitativos son replicables en la medida en que compartimos completamente nuestros códigos y utilizamos la base de datos de acceso abierto del documento seminal escrito por Nelson y Plosser (1982). Nuestra conclusión principal es simple: los investigadores empíricos deben ser absolutamente cautelosos al momento de extraer conclusiones cualitativas basadas en una inferencia estándar de mínimos cuadrados ordinarios realizada en el contexto de un análisis de regresión.
Clasificación JEL: C15, C22, C87, análisis estadístico confiable, inferencia estándar basada en los MCO, sesgo empírico, replicabilidad, medición en economía.
Artículos de investigación
Granger revisited: t values and the empirical OLS bias with stationary and non-stationary time series using Monte Carlo simulations
Granger revisitado: valores t y el sesgo empírico de OLS con series temporales estacionarias y no estacionarias utilizando simulaciones de Monte Carlo
Received: 13 February 2020
Accepted: 18 May 2020
The most common estimator used in applied econometrics is that of ordinary least squares simply due to its “ideal” theoretical characteristics. However, Clive Granger used nonsense correlations introduced by Yule (1926) and a framework based on balanced and unbalanced equations to illustrate some “non-ideal” outcomes; Granger explored the fact that in the presence of stationary and non-stationary time series, the standard ordinary least squares inference could be misleading.
Our purpose is to illustrate the meaning of the conduction of a reliable statistical analysis using some of Granger’s groundbreaking ideas. In section 2, we offer a background to understand some fundamental principles. In sections 3 and 4, we present our simulation results of the empirical bias of OLS. In section 5, we share some final thoughts.
Granger coined the term cointegration to describe a regression in which a pair of variables maintain a genuine long-run relationship instead of a nonsense one, which he referred to as spurious. Looked upon as a long-memory uncovering process, the invention of cointegration was preceded by Granger’s concerns about nonsense correlations introduced by Yule (1926) and a framework based on balanced and unbalanced equations. As a result of such endeavors, Granger was awarded the 2003 Nobel prize in Economic Sciences.
The path to the Nobel was not an easy one, in fact it was quite the opposite. In the following quote Granger (2010, p. 3) explains the difficulties he faced publishing his findings: “Econometrica rejected the paper for various reasons, such as wanting a deeper theory and some discussion of the testing question and an application. As I knew little about testing I was very happy to accept Rob’s offer of help with the revision. I re-did the representation theorem and he produced the test and application, giving a paper by Granger and Engle which evolved into a paper by Engle and Granger, whilst I was away for six months leave in Oxford and Canberra. This new paper was submitted to Econometrica but it was also rejected for not being sufficiently original. I was anxious to submit it to David Hendry’s new econometrics journal but Rob wanted to explore other possibilities. These were being considered when the editor of Econometrica asked us to re-submit because they were getting so many submissions on this topic that they needed our paper as some kind of base reference.”
To introduce his viewpoint, Yule (1926) proposed the correlation coefficient between mortality per 1,000 persons in England and Wales and the proportion of Church of England Marriages per 1,000 of all marriages as an example. The correlation obtained was 0.9512! Yule (1926) conducted a reliable statistical analysis to the extent: 1) he was aware that the correlation between two variables is a meaningful measure of its linear relationship only if its means are constant over time; 2) he identified the high autocorrelation of the selected variables, and 3) he concluded that the underlying probabilistic assumption of the correlation coefficient was violated and, therefore, the obtained measure was likely misleading.1 In this regard, Johnston and DiNardo (1997, p. 10) added: “no British politician proposed closing down the Church of England to confer immortality on the electorate.”
As an extension of his work on spurious regressions during the early 1980s, Granger investigated what he named “balanced models”, defined as follows (Granger, 1993, p. 12): “An equation which has Xt having the same dominant features as Yt will be called balanced. Thus, if Xt is I(0), Yt is I(1), the equation will not be balanced. It is a necessary condition that a specified model be balanced for the model to be satisfactory.” As a rule of thumb in applied economics, we point out that the level of a variable is I(1), and its first difference is I(0). The specification of the Granger’s Ordinary Least Squares (OLS) bivariate regression is the following:
Having in mind “the balanced equation law”, we would expect that Ut has the same property of Yt , and so the third assumption of the classical linear regressions model will not be obeyed, meaning we would expect some low Durbin-Watson statistics, a well-known symptom of spurious regression.
To explain a nonsense regression, Granger used to quote a book written long ago by a coauthor of Yule, M. G. Kendall. We have two AR(1) processes:
with β1 = β2 = β, that is Z1t and Z2t both obeyed the same autoregressive model, and Ut are “white noise” innovations independent of each other at all pairs of time. The sample correlation (R) between n consecutive terms of Z1t and Z2t has the following variance (Kendall, 1954, p. 113):
where n is the sample size. According to Granger (2003, p. 558), if β “is near one and n not very large, then the var(R) will be quite big, which can only be achieved if the distribution of R values has large weights near the extreme values of -1 and 1, which will correspond to” a “significant” coefficient value in a bivariate regression of Z1t on Z2t .
The idea of cointegration between a pair of variables I(1), which generates a linear combination I(0), was a natural extension of Granger’s concerns on spurious regression and balanced models. Suppose Yt and Wt are integrated of order one, and we run the following regression:
Yt and Wt are cointegrated if eliminates its common stochastic trend. In this particular case, is called the “cointegrating” parameter, and Ut is I(0) -implying a stable and unique long-run equilibrium relationship between Yt and Wt .
Lastly, a permanent concern of Professor Granger was the generation of spurious results. Posthumously, he sent us an elegant warning (White and Granger, 2011, p. 15): “When looking for causal relations between stochastic trends, it is important to recognize that Reichenbach’s principle of common cause (Reichenbach, 1956) does not necessarily apply. Briefly, Reichenbach’s principle holds that if we see correlation between two series, then one series must cause the other, or there must be an underlying common cause. With stochastic trends, however, observed correlations may be spurious, as pointed out long ago by Yule (1926).” Their message should guide our empirical work: a high correlation between a pair of variables should not lead us to assume a cause-effect relationship or the existence of a common cause, but rather to recognize it as a result of expressing the statistical properties of the variables at stake in addition to the sensitivity of the statistic used.
In their seminal paper, Granger and Newbold (1974) performed a mini Monte Carlo as a way “to find evidence of spurious regressions” (Granger, 2003, p. 558).2 They generated pairs of independent random walks without drift, each of length 50 (T = 50), and run 100 bivariate regressions. Considering the knowledge at the beginning of the seventies, in their simulation it was expected that roughly 95 percent of |t| values on the estimated parameter would be less than 1.96, but it was the case only in 23 occasions.3 Since the random walks were unrelated by design, the t values were misleading, and so, using their simulation results Granger and Newbold (1974, p. 115) suggested a new critical value of 11.2 when assessing the significance of the coefficient at the 5% level. At present we know that this suggestion is inconvenient, in the sense that Phillips (1986, p. 318) demonstrated that it has not asymptotic sense to apply the correction based on the standardized statistic, that is, .
Inspired by Granger and Newbold (1974) we designed a Monte Carlo simulation. In our case, each random walk has 100 terms -approximately the same length of the variables contained in the database of Nelson and Plosser (1982) that we will use later-, and the number of replications was the nowadays standard 10,000. Box 1 shows our (EViews) program.
’Create a workfile undated, range 1 to 10,000
!reps = 10000
for !i=1 to !reps
genr innovationrw1{!i}=@nrnd
genr innovationrw2{!i}=@nrnd
smpl 1 1
genr rw1{!i}=0
genr rw2{!i}=0
smpl 2 100
genr rw1{!i}=rw1{!i}(-1)+innovationrw1{!i}
genr rw2{!i}=rw2{!i}(-1)+innovationrw2{!i}
smpl 1 100
matrix(!reps,2) results
equation eq{!i}.ls rw1{!i}=c(1)*rw2{!i}
results(!i,1)=eq{!i}.@coefs(1)
results(!i,2)=eq{!i}.@tstats(1)
d innovationrw1{!i}
d innovationrw2{!i}
d rw1{!i}
d rw2{!i}
d eq{!i}
next
’Copy and paste “results” to Excel
’In EViews type: data NONSENSE TsNONSENSE
’Copy and paste from Excel to EViews NONSENSE TsNONSENSE
It is necessary to specify that the “!reps” command sets the number of replicas; in the following lines, we create (“genr”) a pair of stationary time series that contain pseudo-random draws from a standard normal distribution. Initially, we focused on the first observations (“smpl 1 1”) to fix the initial conditions of the two random walks. We then generated the rest of its values by stacking the innovations generated. To create a new object, specifically speaking a matrix with two columns, we invoke the command “matrix”, we estimate the regression using OLS (“.ls”), and we deposit the values of the estimated coefficients and its corresponding t values in “results”. It is necessary to erase the inputs of the exercise for obvious reasons (“d” from delete), and to use the “next” command to end the code. Finally, with the last three suggestion lines, we change the format of the estimated coefficients and t values as columns of a matrix to variables (data), using the spreadsheet as a “scale” software.
The following figure shows statistics about the values of the NONSENSE estimated coefficients (y-axis frequency and x-axis coefficient values) and the table its t values tabulation.
In figure 1, the horizontal axis constitutes the bias in estimating the slope coefficient -to the extent the true value is zero- and the vertical axis is its relative frequency. In other words, we would expect coefficients with a value around zero and non-significant t values in the 95 percent of the regressions since, by design, drift-free independent random walks were used. But the results obtained were not as expected: in 7,800 cases the values of the coefficients exceeded the value of |0.2| and we barely obtained in 1,490 occasions t values less than |2|. Hence, there is a problem if we apply the “standard OLS inference” (Granger et al. 2001, p. 900). In our days we know that the heart of the matter is the use of standard errors which underestimate the true variation of the OLS estimators; unfortunately, there is not a practical solution, that is, an alternative mechanical estimator of the standard errors useful to construct an adjusted t statistic (Sun, 2004).
According to Phillips (2003, p. 2), “a primary limitation on empirical knowledge is that the true model for any given data is unknown and, in all practical cases, unknowable. Even if a formulated model were correct, it would still depend on parameters that need to be estimated from data”.
Using a particular type of stationary time series Greene (2003, pp. 44-45) proposed the following Monte Carlo experiment: 1) Generate two stationary random variables, Wt and Xt ; 2) Generate Ut = 0.5Wt , and Yt = 0.5Xt + Ut ; 3) Run 10,000 regressions, and 4) Randomly select 500 samples of 100 observation and make a graph. As we read in other books frequently, his conclusion was simple (Greene, 2003, p. 45): “note that the distribution of slopes has a mean roughly equal to the ‘true values’ of 0.5”. It is convenient to note the following:
In Granger et al. (2001) and Granger (2003, p. 560), it was shown that spurious regressions can also occur, “although less clearly”, with stationary time series. They generated two AR(1) processes as in our equations (2) and (3) with 0 < β1 = β2 = β ≤ 1, and run regressions following equation (1) with sample sizes varying between 100 and 10,000 -to suggest that the nonsense regression issue is not a small sample property. Table 2 is a summary of their results:
According to Granger (2003, p. 557), Yule (1926) is a much-cited paper but not sufficiently understood. The content of Table 2 supports this point of view in the sense that, as in Yule (1926), the clue is the degree of the autocorrelation of involved variables in a regression. It is not necessary to have random walks to generate nonsense exercises. It is just great, with AR(1) processes with coefficients between 0.25 and 0.9, and distinct sample sizes, Granger et al. (2001) were in a position to put all the applied econometricians in check.
In their seminal (Campbell and Perron, 1991, p. 147) paper, Nelson and Plosser (1982, p. 147) analyzed 27 variables in “natural logs except for the bond yield”: real GNP, nominal GNP, real per capita GNP, industrial production, employment, unemployment rate, GDP deflator, consumer prices, wages, real wages, money stock, velocity, bond yields, and common stock prices. Theoretically speaking, there is a mixture of stationary and non-stationary time series. So as not to enter into an unnecessary discussion, the next figure shows all the variables and Table 3 its autocorrelation coefficients.
To run four simulations, we use as a point of reference the minor values of the autocorrelations contained in Table 3. In each case, for example, we run the following general program:
’Create a workfile undated, range 1 to 10,000
!reps = 10000
for !i=1 to !reps
genr innovationx1{!i}=@nrnd
genr innovationx2{!i}=@nrnd
smpl 1 1
genr x1{!i}=0
genr x2{!i}=0
smpl 2 100
genr x1{!i}=(0.754)*x1{!i}(-1)+innovationx1{!i}
genr x2{!i}=(0.754)*x2{!i}(-1)+innovationx2{!i}
smpl 1 100
matrix(!reps,2) results
equation eq{!i}.ls x1{!i}=c(1)*x2{!i}
results(!i,1)=eq{!i}.@coefs(1)
results(!i,2)=eq{!i}.@tstats(1)
d innovationx1{!i}
d innovationx2{!i}
d x1{!i}
d x2{!i}
d eq{!i}
next
’Export “results” to Excel
’In EViews: data NONSENSE tsNONSENSE
’Copy and paste from Excel to EViews NONSENSE tsNONSENSE
The user should only change the autoregressive values in the program (lines 10 and 11). The following figure and table show statistics about the values of the estimated coefficients and its t values (y-axis frequency and x-axis values). Following Patterson (2000, pp. 317-26), we improved its titles.
Before commenting on our results, we want to focus on a critical aspect of the dissemination of sciences. In 2011 Crocker and Cooper, editors of the prestige Science, wrote a note entitled “Addressing scientific fraud”, showing evidence that fraud is a common practice. In this regard, as the first line of defense, Crocker and Cooper (2011, p. 1182) ask for “greater transparency with data”.
At least in economics, Crocker and Cooper (2011) hit the nail on the head. To cite but one terrible example, Duvendack, Palmer-Jones, and Reed (2015, pp. 181-182) uncovered the following: “What can we learn from our analysis of replication studies? Most importantly, and perhaps not too surprisingly, the main takeaway is that, conditional on the replication having been published, there is a high rate of disconfirmation. Over the full set of replication studies, approximately two out of every three studies were unable to confirm the original findings. Another 12 percent disconfirmed at least one major finding of the original study, while confirming others (Mixed?). In other words, nearly 80 percent of replication studies have found major flaws in the original research.”
Hence, we used an open-access database of a seminal paper and explained our computational code step by step. We can now begin to analyze our results.
In Figure 3, the horizontal axis measures the bias in estimating the slope coefficient, and the vertical axis is its relative frequency. Once again, in our simulations, we obtained unexpected coefficient values and t values. For Granger (2003, p. 560), the implication of these results is that “applied econometricians should not worry about spurious regressions only when dealing with I(1), unit root, processes.” But his caveat has been practically ignored by a large portion of the specialized literature. Excellent econometric literature -both textbooks and journal articles- exposes OLS without looking at the statistical properties of the variables we are dealing with, visualizes non-stationary time series in some moments as dominated by a deterministic trend and in others by a stochastic one, and last but not least, limits the problem of spurious regressions to the case of integrated variables.4
Measurement in economics is not a unified field but fragmented in subfields such as econometrics, index theory, and national accounts (Boumans, 2007, p. 3). It is not an exaggeration to point out that currently, econometric analysis constitutes a central measuring instrument used in our science. It is worth mentioning that, as other measuring instruments, initially functioned as an “artifact of measurement”, but rapidly became an “analytical device” (Klein, 2001 and Morgan 2001). In this regard, the guru of measurement in economics warned us that (Boumans 2005, p. 121): “a relevant problem of instruments used to make unobservables visible is how to distinguish between the facts about the phenomenon and the artifacts created by the instrument.”
Therefore, it is always necessary to keep in mind that the output of the econometric analysis shapes our scientific study of objects. Granger’s path-breaking ideas already showed us the dangers of carrying out a regression that does not take into account the statistical characteristics of the analyzed variables and of ignoring the fact that the properties of the OLS estimator depend crucially upon these characteristics. We should be cautious when drawing qualitative conclusions based on a standard OLS inference carried out in the context of a regression analysis. Indeed, it is urgent that the specialized literature completely assimilates Granger’s contributions regarding the risk of obtaining spurious results in economics.