Sources of error in forecasts and expectations: U.K. economic models, 1984–8

This paper presents a comparative analysis of the sources of error in forecasts for the UK economy published over a recent four-year period by four independent groups. This analysis rests on the archiving at the ESRC Macroeconomic Modelling Bureau of the original forecasts together with all their accompanying assumptions and adjustments. A method of decomposing observed forecast errors so as to distinguish the contributions of forecaster and model is set out; the impact of future expectations treated in a ‘model-consistent’ or ‘rational’ manner is specifically considered. The results show that the forecaster's adjustments make a substantial contribution to forecast performance, a good part of which comes from adjustments that bring the model on track at the start of the forecast period. The published ex-ante forecasts are usually superior to pure model-based ex-post forecasts, whose performance indicates some misspecification of the underlying models.     

Prediction,diagnosis, and causal thinking in forecasting

While forecasting involves forward/predictive thinking, it depends crucially on prior diagnosis for suggesting a model of the phenomenon, for defining‘relevant’variables, and for evaluating forecast accuracy via the model. The nature of diagnostic thinking is examined with respect to these activities. We first consider the difficulties of evaluating forecast accuracy without a causal model of what generates outcomes. We then discuss the development of models by considering how attention is directed to variables via analogy and metaphor as well as by what is unusual or abnormal. The causal relevance of variables is then assessed by reference to probabilistic signs called‘cues to causality’. These are: temporal order, constant conjunction, contiguity in time and space, number of alternative explanations, similarity, predictive validity, and robustness. The probabilistic nature of the cues is emphasized by discussing the concept of spurious correlation and how causation does not necessarily imply correlation. Implications for improving forecasting are considered with respect to the above issues.     

Estimation and forecasting in first‐order vector autoregressions with near to unit roots and conditional heteroscedasticity

This paper investigates the effects of imposing invalid cointegration restrictions or ignoring valid ones on the estimation, testing and forecasting properties of the bivariate, first‐order, vector autoregressive (VAR(1)) model. We first consider nearly cointegrated VARs, that is, stable systems whose largest root, l_max, lies in the neighborhood of unity, while the other root, l_min, is safely smaller than unity. In this context, we define the ‘forecast cost of type I’ to be the deterioration in the forecasting accuracy of the VAR model due to the imposition of invalid cointegration restrictions. However, there are cases where misspecification arises for the opposite reasons, namely from ignoring cointegration when the true process is, in fact, cointegrated. Such cases can arise when l_max equals unity and l_min is less than but near to unity. The effects of this type of misspecification on forecasting will be referred to as ‘forecast cost of type II’. By means of Monte Carlo simulations, we measure both types of forecast cost in actual situations, where the researcher is led (or misled) by the usual unit root tests in choosing the unit root structure of the system. We consider VAR(1) processes driven by i.i.d. Gaussian or GARCH innovations. To distinguish between the effects of nonlinear dependence and those of leptokurtosis, we also consider processes driven by i.i.d. t(2) innovations. The simulation results reveal that the forecast cost of imposing invalid cointegration restrictions is substantial, especially for small samples. On the other hand, the forecast cost of ignoring valid cointegration restrictions is small but not negligible. In all the cases considered, both types of forecast cost increase with the intensity of GARCH effects. Copyright © 2009 John Wiley & Sons, Ltd.     

Multi‐model Forecasts of the West Texas Intermediate Crude Oil Spot Price

We measure the performance of multi‐model inference (MMI) forecasts compared to predictions made from a single model for crude oil prices. We forecast the West Texas Intermediate (WTI) crude oil spot prices using total OECD petroleum inventory levels, surplus production capacity, the Chicago Board Options Exchange Volatility Index and an implementation of a subset autoregression with exogenous variables (SARX). Coefficient and standard error estimates obtained from SARX determined by conditioning on a single ‘best model’ ignore model uncertainty and result in underestimated standard errors and overestimated coefficients. We find that the MMI forecast outperforms a single‐model forecast for both in‐ and out‐of‐sample datasets over a variety of statistical performance measures, and further find that weighting models according to the Bayesian information criterion generally yields superior results both in and out of sample when compared to the Akaike information criterion. Copyright © 2016 John Wiley & Sons, Ltd.     

Henderson‐Trending of Macroeconomic Variables and Forecasting Accuracy

Using a structural time‐series model, the forecasting accuracy of a wide range of macroeconomic variables is investigated. Specifically of importance is whether the Henderson moving‐average procedure distorts the underlying time‐series properties of the data for forecasting purposes. Given the weight of attention in the literature to the seasonal adjustment process used by various statistical agencies, this study hopes to address the dearth of literature on ‘trending’ procedures. Forecasts using both the trended and untrended series are generated. The forecasts are then made comparable by ‘detrending’ the trended forecasts, and comparing both series to the realised values. Forecasting accuracy is measured by a suite of common methods, and a test of significance of difference is applied to the respective root mean square errors. It is found that the Henderson procedure does not lead to deterioration in forecasting accuracy in Australian macroeconomic variables on most occasions, though the conclusions are very different between the one‐step‐ahead and multi‐step‐ahead forecasts. Copyright © 2011 John Wiley & Sons, Ltd.     

Reducing uncertainty in short-term projections: Linkage of monthly and quarterly models

This paper shows how monthly data and forecasts can be used in a systematic way to improve the predictive accuracy of a quarterly macroeconometric model. The problem is formulated as a model pooling procedure (equivalent to non-recursive Kalman filtering) where a baseline quarterly model forecast is modified through ‘add-factors’ or ‘constant adjustments’. The procedure ‘automatically’ constructs these adjustments in a covariance-minimizing fashion to reflect the revised expectation of the quarterly model's forecast errors, conditional on the monthly information set. Results obtained using Federal Reserve Board models indicate the potential for significant reduction in forecast error variance through application of these procedures.     

The importance of non-linearities in large forecasting models with stochastic error processes

This paper considers the consequences of the stochastic error process in large non-linear forecasting models. As such models are non-linear, the deterministic forecast is neither the mean nor the mode of the density function of the endogenous variables. Under a specific assumption as to the class of the non-linearity it is shown that the deterministic forecast is actually the vector of marginal medians of the density function. Stochastic simulation techniques are then used to test whether one large forecasting model actually lies within this class.     

Testing for unreliable estimators and insignificant forecasts in combined forecasts

The reliability and precision of the weights used in combining individual forecasts, irrespective of the method of combination, is important in evaluating a combined forecast. The objective of this study is not to suggest the ‘best’ method of combining individual forecasts, but rather to propose exploratory procedures, that make use of all available sample information contained in the covariance matrix of individual forecast errors, to (1) detect if the weights used in combining forecasts are ‘reliable’ (and ‘stable’ if it is known that the covariance matrix of forecast errors is stationary over time) and (2) test for ‘insignificant’ individual forecasts used in forming a combined forecast. We present empirical applications using two-year sales and individual forecast data provided by a major consumer durables manufacturer to illustrate the feasibility of our proposed procedures.     

Effect of regressor forecast error on the variance of regression forecasts

It is well understood that the standard formulation for the variance of a regression‐model forecast produces interval estimates that are too narrow, principally because it ignores regressor forecast error. While the theoretical problem has been addressed, there has not been an adequate explanation of the effect of regressor forecast error, and the empirical literature has supplied a disparate variety of bits and pieces of evidence. Most business‐forecasting software programs continue to supply only the standard formulation. This paper extends existing analysis to derive and evaluate large‐sample approximations for the forecast error variance in a single‐equation regression model. We show how these approximations substantially clarify the expected effects of regressor forecast error. We then present a case study, which (a) demonstrates how rolling out‐of‐sample evaluations can be applied to obtain empirical estimates of the forecast error variance, (b) shows that these estimates are consistent with our large‐sample approximations and (c) illustrates, for ‘typical’ data, how seriously the standard formulation can understate the forecast error variance. Copyright © 2000 John Wiley & Sons, Ltd.     

Efficient use of information in the formation of subjective industry forecasts

The paper considers the use of information by a panel of expert industry forecasters, focusing on their information-processing biases. The panel forecasts construction output by sector up to three years ahead. It is found that the biases observed in laboratory experiments, particularly ‘anchoring’, are observable. The expectations are formed by adjusting the previous forecast to take new information into account. By analysing forecast errors it is concluded that the panel overweight recently released information and do not understand the dynamics of the industry. However, their forecasts, both short and long term, are better than an alternative econometric model, and combining the two sources of forecasts leads to a deterioration in forecast accuracy. The expert forecasts can be ‘de-biased’, and this leads to the conclusion that it is better to optimally process information sources than to combine (optimally) alternative forecasts.     

Consistent forecast intervals when the forecast-period exogenous variables are stochastic

Derivation of prediction intervals in the k-variable regression model is problematic when future-period values of exogenous variables are not known with certainty. Even in the most favourable case when the forecasts of the exogenous variables are jointly normal, the distribution of the forecast error is non-normal, and thus traditional asymptotic normal theory does not apply. This paper presents an alternative bootstrap method. In contrast to the traditional predictor of the future value of the endogenous variable, which is known to be inconsistent, the bootstrap predictor converges weakly to the true value. Monte Carlo results show that the bootstrap prediction intervals can achieve approximately nominal coverage.     

Local to unity,long‐horizon forecasting thresholds for model selection in the AR(1)

This article introduces a novel framework for analysing long‐horizon forecasting of the near non‐stationary AR(1) model. Using the local to unity specification of the autoregressive parameter, I derive the asymptotic distributions of long‐horizon forecast errors both for the unrestricted AR(1), estimated using an ordinary least squares (OLS) regression, and for the random walk (RW). I then identify functions, relating local to unity ‘drift’ to forecast horizon, such that OLS and RW forecasts share the same expected square error. OLS forecasts are preferred on one side of these ‘forecasting thresholds’, while RW forecasts are preferred on the other. In addition to explaining the relative performance of forecasts from these two models, these thresholds prove useful in developing model selection criteria that help a forecaster reduce error. Copyright © 2004 John Wiley & Sons, Ltd.     

The multi‐chain Markov switching model

In many real phenomena the behaviour of a certain variable, subject to different regimes, depends on the state of other variables or the same variable observed in other subjects, so the knowledge of the state of the latter could be important to forecast the state of the former. In this paper a particular multivariate Markov switching model is developed to represent this case. The transition probabilities of this model are characterized by the dependence on the regime of the other variables. The estimation of the transition probabilities provides useful information for the researcher to forecast the regime of the variables analysed. Theoretical background and an application are shown. Copyright © 2005 John Wiley & Sons, Ltd.     

Choosing among competing econometric forecasts: Regression‐based forecast combination using model selection

Forecast combination based on a model selection approach is discussed and evaluated. In addition, a combination approach based on ex ante predictive ability is outlined. The model selection approach which we examine is based on the use of Schwarz (SIC) or the Akaike (AIC) Information Criteria. Monte Carlo experiments based on combination forecasts constructed using possibly (misspecified) models suggest that the SIC offers a potentially useful combination approach, and that further investigation is warranted. For example, combination forecasts from a simple averaging approach MSE‐dominate SIC combination forecasts less than 25% of the time in most cases, while other ‘standard’ combination approaches fare even worse. Alternative combination approaches are also compared by conducting forecasting experiments using nine US macroeconomic variables. In particular, artificial neural networks (ANN), linear models, and professional forecasts are used to form real‐time forecasts of the variables, and it is shown via a series of experiments that SIC, t‐statistic, and averaging combination approaches dominate various other combination approaches. An additional finding is that while ANN models may not MSE‐dominate simpler linear models, combinations of forecasts from these two models outperform either individual forecast, for a subset of the economic variables examined. Copyright © 2001 John Wiley & Sons, Ltd.     

Forecast accuracy after pretesting with an application to the stock market

In econometrics, as a rule, the same data set is used to select the model and, conditional on the selected model, to forecast. However, one typically reports the properties of the (conditional) forecast, ignoring the fact that its properties are affected by the model selection (pretesting). This is wrong, and in this paper we show that the error can be substantial. We obtain explicit expressions for this error. To illustrate the theory we consider a regression approach to stock market forecasting, and show that the standard predictions ignoring pretesting are much less robust than naive econometrics might suggest. We also propose a forecast procedure based on the ‘neutral Laplace estimator’, which leads to an improvement over standard model selection procedures. Copyright © 2004 John Wiley & Sons, Ltd.     

Can cointegration‐based forecasting outperform univariate models? An application to Asian exchange rates

Conventional wisdom holds that restrictions on low‐frequency dynamics among cointegrated variables should provide more accurate short‐ to medium‐term forecasts than univariate techniques that contain no such information; even though, on standard accuracy measures, the information may not improve long‐term forecasting. But inconclusive empirical evidence is complicated by confusion about an appropriate accuracy criterion and the role of integration and cointegration in forecasting accuracy. We evaluate the short‐ and medium‐term forecasting accuracy of univariate Box–Jenkins type ARIMA techniques that imply only integration against multivariate cointegration models that contain both integration and cointegration for a system of five cointegrated Asian exchange rate time series. We use a rolling‐window technique to make multiple out of sample forecasts from one to forty steps ahead. Relative forecasting accuracy for individual exchange rates appears to be sensitive to the behaviour of the exchange rate series and the forecast horizon length. Over short horizons, ARIMA model forecasts are more accurate for series with moving‐average terms of order >1. ECMs perform better over medium‐term time horizons for series with no moving average terms. The results suggest a need to distinguish between ‘sequential’ and ‘synchronous’ forecasting ability in such comparisons. Copyright © 2002 John Wiley & Sons, Ltd.     

Modifying quantitative forecasts of livestock production using expert judgments: An application to the australian lamb industry

The judgmental modification of quantitative forecasts has become increasingly adopted in the production of agricultural commodity outlook information. Such modifications allow current period information to be incorporated into the forecast value, and ensure that the forecast is realistic in the context of current industry trends. This paper investigates the potential value of this approach in production forecasting in the Australian lamb industry. Several individual and composite econometric models were used to forecast a lamb-slaughtering series with a selected forecast being given to a panel of lamb industry specialists for consideration and modification. The results demonstrate that this approach offers considerable accuracy advantages in the short-term forecasting of livestock market variables, such as slaughtering, whose values can be strongly influenced by current industry conditions.     

Seasonality in large-scale macroeconometric models

Macroeconomic indicators are typically appraised in seasonally adjusted form, and forecasts are often presented in a similar way (as annual changes, for example). Moreover, the quarterly macroeconomic models used in forecasting are commonly estimated from seasonally adjusted data. Nevertheless, these models can generate forecasts with seasonal patterns, and this paper assesses the cause and cure of this phenomenon. It is found that forecast seasonality is induced by seasonality in the various inputs: exogenous variables, residual adjustments, the dynamic specification of certain equations, and annual changes in policy variables. Series changing annually but observed quarterly are termed ‘intercalated series’, and are simple examples of periodic processes. Forecast seasonality can be removed by appropriate adjustment of all these inputs. Models containing explicit future expectations variables solved in a model-consistent manner are also considered: numerical sensitivity to the terminal quarter may result from terminal conditions that require adjustment when seasonality is present.     

Analysis of the US business cycle with a Vector‐Markov‐switching model

This paper identifies turning points for the US ‘business cycle’ using information from different time series. The model, a multivariate Markov‐switching model, assumes that each series is characterized by a mixture of two normal distributions (a high and low mean) with the switching from one to the other determined by a common Markov process. The procedure is applied to the series composing the composite coincident indicator in the USA to obtain business cycle turning points. The business cycle chronology is closer to the NBER reference cycle than the turning points obtained from the individual series using a univariate model. The model is also used to forecast the series with some encouraging results. Copyright © 2001 John Wiley & Sons, Ltd.     

Some practical aspects of forecasting in organizations

This paper presents the writer's experience, over a period of 25 years, in analysing organizational systems and, in particular, concentrates on the overall forecasting activity. The paper first looks at the relationship between forecasting and decision taking–with emphasis on the fact that forecasting is a means to aid decision taking and not an end in itself. It states that there are many types of forecasting problems, each requiring different methods of treatment. The paper then discusses attitudes which are emerging about the relative advantages of different forecasting techniques. It suggests a model building process which requires‘experience’and‘craftsmanship’, extensive practical application, frequent interaction between theory and practice and a methodology that eventually leads to models that contain no detectable inadequacies. Furthermore, it argues that although models which forecast a time series from its past history have a very important role to play, for effective policy making it is necessary to augment the model by introducing policy variables, again in a systematic not an ‘ad hoc’ manner. Finally, the paper discusses how forecasting systems can be introduced into the management process in the first place and how they should be monitored and updated when found wanting.