Merging monthly and quarterly forecasts: Experience with mqem

Forecasts from quarterly econometric models are typically revised on a monthly basis to reflect the information in current economic data. The revision process usually involves setting targets for the quarterly values of endogenous variables for which monthly observations are available and then altering the intercept terms in the quarterly forecasting model to achieve the target values. A formal statistical approach to the use of monthly data to update quarterly forecasts is described and the procedure is applied to the Michigan Quarterly Econometric Model of the US Economy. The procedure is evaluated in terms of both ex post and ex ante forecasting performance. The ex ante results for 1986 and 1987 indicate that the method is quite promising. With a few notable exceptions, the formal procedure produces forecasts of GNP growth that are very close to the published ex ante forecasts.     

System-based weights versus series-specific weights in the combination of forecasts

System-based combination weights for series r/step-length h incorporate relative accuracy information from other forecast step-lengths for r and from other series for step-length h. Such weights are examined utilizing the West and Fullerton (1996) data set-4275 ex ante employment forecasts from structural simultaneous equation econometric models for 19 metropolitan areas at 10 quarterly step-lengths and a parallel set of 4275 ARIMA forecasts. The system-based weights yielded combined forecasts of higher average accuracy and lower risk of large inaccuracy than seven alternative strategies: (1) averaging; (2) relative MSE weights; (3) outperformance (per cent best) weights; (4) Bates and Granger (1969) optimal weights with a convexity constraint imposed; (5) unconstrained optimal weights; (6) select a ‘best’ method (ex ante) by series and; (7) experiment in the Bischoff (1989) sense and select either method (2) or (6) based on the outcome of e experiment. Accuracy gains of the system-based combination were concentrated at step-lengths two to five. Although alternative (5) was generally outperformed, none of the six other alternatives was systematically most accurate when evaluated relative to each other. This contrasts with Bischoff's (1989) results that held promise for an empirically applicable guideline to determine whether or not to combine.     

Regional econometric income forecast accuracy

Econometric prediction accuracy for personal income forecasts is examined for a region of the United States. Previously published regional structural equation model (RSEM) forecasts exist ex ante for the state of New Mexico and its three largest metropolitan statistical areas: Albuquerque, Las Cruces and Santa Fe. Quarterly data between 1983 and 2000 are utilized at the state level. For Albuquerque, annual data from 1983 through 1999 are used. For Las Cruces and Santa Fe, annual data from 1990 through 1999 are employed. Univariate time series, vector autoregressions and random walks are used as the comparison criteria against structural equation simulations. Results indicate that ex ante RSEM forecasts achieved higher accuracy than those simulations associated with univariate ARIMA and random walk benchmarks for the state of New Mexico. The track records of the structural econometric models for Albuquerque, Las Cruces and Santa Fe are less impressive. In some cases, VAR benchmarks prove more reliable than RSEM income forecasts. In other cases, the RSEM forecasts are less accurate than random walk alternatives. Copyright © 2005 John Wiley & Sons, Ltd.     

Choosing among competing econometric forecasts: Regression‐based forecast combination using model selection

Forecast combination based on a model selection approach is discussed and evaluated. In addition, a combination approach based on ex ante predictive ability is outlined. The model selection approach which we examine is based on the use of Schwarz (SIC) or the Akaike (AIC) Information Criteria. Monte Carlo experiments based on combination forecasts constructed using possibly (misspecified) models suggest that the SIC offers a potentially useful combination approach, and that further investigation is warranted. For example, combination forecasts from a simple averaging approach MSE‐dominate SIC combination forecasts less than 25% of the time in most cases, while other ‘standard’ combination approaches fare even worse. Alternative combination approaches are also compared by conducting forecasting experiments using nine US macroeconomic variables. In particular, artificial neural networks (ANN), linear models, and professional forecasts are used to form real‐time forecasts of the variables, and it is shown via a series of experiments that SIC, t‐statistic, and averaging combination approaches dominate various other combination approaches. An additional finding is that while ANN models may not MSE‐dominate simpler linear models, combinations of forecasts from these two models outperform either individual forecast, for a subset of the economic variables examined. Copyright © 2001 John Wiley & Sons, Ltd.     

Forecasting the economic recession in the U.K. 1979-1982: A comparison of model-based ex ante Forecasts

This paper compares the forecasts of recession and recovery made by five non-government U.K. teams modelling the economy (Cambridge Econometrics, the London Business School, the National Institute of Economic and Social Research, the Cambridge Economic Policy Group and the Liverpool Research Group). The paper concentrates on annual ex ante projections as published over the period 1978-1982, i.e. forecasts made, before the event, of the onset, length, depth and character of the economic recession in the U.K. which began in 1979. The comparison is in terms of year by year changes in production, unemployment, prices and other variables. It concludes that no group was systematically better or worse than other groups (confirming U.S. experience) and that the groups tended to perform better in their chosen areas of specialization, e.g. medium-term groups did better at forecasting the medium-term outcome.     

Assessing the historical accuracy of regional economic forecasts

This study addresses for the first time systematic evaluation of a widely used class of forecasts, regional economic forecasts. Ex ante regional structural equation model forecasts are analysed for 19 metropolitan areas. One- to ten-quarter-ahead forecasts are considered and the seven-year sample spans a complete business cycle. Counter to previous speculation in the literature, (1) dependency on macroeconomic forecasting model inputs does not substantially erode accuracy relative to univariate extrapolative methodologies and (2) stochastic time series models do not on average, yield more accurate regional economic predictions than structural models. Similar to findings in other studies, clear preferences among extrapolative methodologies do not emerge. Most general conclusions, however, are subject to caveats based on step-length effects and region-specific effects.     

Co-integration,error correction and improved medium-term regional VAR forecasting

This study investigates possible improvements in medium-term VAR forecasting of state retail sales and personal income when the two series are co-integrated and represent an error-correction system. For each of North Carolina and New York, three regional vector autoregression (VAR) models are specified; an unrestricted two-equation model consisting of the two state variables, a five-equation unrestricted model with three national variables added and a Bayesian (BVAR) version of the second model. For each state, the co-integration and error-correction relationship of the two state variables is verified and an error-correction version of each model specified. Twelve successive ex ante five-year forecasts are then generated for each of the state models. The results show that including an error-correction mechanism when statistically significant improves medium-term forecasting accuracy in every case.     

An empirical investigation of combinations of economic forecasts

This paper examines the effects of combining three econometric and three times-series forecasts of growth and inflation in the U.K. If forecasts are unbiased then a combination exploiting this fact will be more efficient than an unrestricted combination. Ex post econometric forecasts may be biased but ex ante they are unbiased. The results of the study are that a restricted linear combination of the econometric forecasts is superior to an unrestricted combination and also to the unweighted mean of the forecasts. However, it is not preferred to the best of the individual forecasts.     

Comparing ex-ante forecasts from a sem and var model: An application to the italian economy

The predictive performance of a large-scale structural econometric model (SEM) of the Italian economy the Prometeia model is compared in this paper with a vector autoregressive (VAR) model estimated for a selection of six main variables of interest. The paper concentrates on the quarterly ex-ante forecasts of GDP growth rate and the annual forecasts of GDP growth and inflation rate, over the period 1980-85. It concludes that no forecaster is systematically better than the other. In particular, the VAR model outperforms the SEM in short-run forecasts, suggesting that, for the latter, more careful attention should be addressed to questions of dynamic specification. On the other hand, for longer intervals, the SEM forecasts are more accurate than the VAR forecasts, in that they can benefit from the judgemental interventions of the model users and the model can pick up the non-linearities of the economy which cannot be captured by the VAR. Given the different kinds of information that can be extracted from the two approaches, it seems more reasonable to consider them as complementary rather than alternative tools for modelling and forecasting. Therefore, rather than attempting to establish the superiority of one type of model over the other, this kind of comparisons should be seen as a useful diagnostic tool for detecting types of model misspecification.     

Method and situational factors in sales forecast accuracy

In this study the interaction of forecasting method (econometric versus exponential smoothing) and two situational factors are evaluated for their effects upon accuracy. Data from two independent sets of ex ante quarterly forecasts for 19 classes of mail were used to test hypotheses. Counter to expectations, the findings revealed that forecasting method did not interact with the forecast time horizon (short versus long term). However, as hypothesized, forecasting method interacted significantly with product/market definition (First Class versus other mail), an indicator of buyer sensitivity to marketing/environmental changes. Results are discussed in the context of future research on forecast accuracy.     

Causal forces: Structuring knowledge for time-series extrapolation

This paper examines a strategy for structuring one type of domain knowledge for use in extrapolation. It does so by representing information about causality and using this domain knowledge to select and combine forecasts. We use five categories to express causal impacts upon trends: growth, decay, supporting, opposing, and regressing. An identification of causal forces aided in the determination of weights for combining extrapolation forecasts. These weights improved average ex ante forecast accuracy when tested on 104 annual economic and demographic time series. Gains in accuracy were greatest when (1) the causal forces were clearly specified and (2) stronger causal effects were expected, as in longer-range forecasts. One rule suggested by this analysis was: ‘Do not extrapolate trends if they are contrary to causal forces.’ We tested this rule by comparing forecasts from a method that implicitly assumes supporting trends (Holt's exponential smoothing) with forecasts from the random walk. Use of the rule improved accuracy for 20 series where the trends were contrary; the MdAPE (Median Absolute Percentage Error) was 18% less for the random walk on 20 one-year ahead forecasts and 40% less for 20 six-year-ahead forecasts. We then applied the rule to four other data sets. Here, the MdAPE for the random walk forecasts was 17% less than Holt's error for 943 short-range forecasts and 43% less for 723 long-range forecasts. Our study suggests that the causal assumptions implicit in traditional extrapolation methods are inappropriate for many applications.     

The usefulness of heuristic N(E)RLS algorithms for combining forecasts

There exists theoretical and empirical evidence on the efficiency and robustness of Non-negativity Restricted Least Squares combinations of forecasts. However, the computational complexity of the method hinders its widespread use in practice. We examine various optimizing and heuristic computational algorithms for estimating NRLS combination models and provide certain CPU-time reducing implementations. We empirically compare the combination weights identified by the alternative algorithms and their computational demands based on a total of more than 66,000 models estimated to combine the forecasts of 37 firm-specific accounting earnings series. The ex ante prediction accuracies of combined forecasts from the optimizing versus heuristic algorithms are compared. The effects of fit sample size, model specification, multicollinearity, correlations of forecast errors, and series and forecast variances on the relative accuracy of the optimizing versus heuristic algorithms are analysed. The results reveal that, in general, the computationally simple heuristic algorithms perform as well as the optimizing algorithms. No generalizable conclusions could be reached, however, about which algorithm should be used based on series and forecast characteristics. © 1997 John Wiley & Sons, Ltd.     

Ex post and ex ante analysis of provisional data

In this paper we suggest a framework to assess the degree of reliability of provisional estimates as forecasts of final data, and we re‐examine the question of the most appropriate way in which available data should be used for ex ante forecasting in the presence of a data‐revision process. Various desirable properties for provisional data are suggested, as well as procedures for testing them, taking into account the possible non‐stationarity of economic variables. For illustration, the methodology is applied to assess the quality of the US M1 data production process and to derive a conditional model whose performance in forecasting is then tested against other alternatives based on simple transformations of provisional data or of past final data. Copyright © 1999 John Wiley & Sons, Ltd.     

The role of macroeconometric models in forecasting and policy analysis in the united states

This article stresses how little is known about the quality, particularly the relative quality, of macroeconometric models. Most economists make a strict distinction between the quality of a model per se and the accuracy of solutions based on that model. While this distinction is valid, it leaves unanswered how to compare the‘validity’of conditional models. The standard test, the accuracy of ex post simulations, is not definitive when models with differing degrees of exogeneity are compared. In addition, it is extremely difficult to estimate the relative quantitative importance of conceptual problems of models, such as parameter instability across‘policy regimes’ In light of the difficulty in comparisons of conditional macroeconometric models, many model-builders and users assume that the best models are those that have been used to make the most accurate forecasts are those made with the best models. Forecasting experience indicates that forecasters using macroeconometric models have produced more accurate macroeconomic forecasts than either naive or sophisticated unconditional statistical models. It also suggests that judgementally adjusted forecasts have been more accurate than model-based forecasts generated mechanically. The influence of econometrically-based forecasts is now so pervasive that it is difficult to find examples of‘purely judgemental’forecasts.     

On the Predictive Content of Autoregression Residuals: A Semiparametric,Copula‐Based Approach to Time Series Prediction

This paper proposes an adjustment of linear autoregressive conditional mean forecasts that exploits the predictive content of uncorrelated model residuals. The adjustment is motivated by non‐Gaussian characteristics of model residuals, and implemented in a semiparametric fashion by means of conditional moments of simulated bivariate distributions. A pseudo ex ante forecasting comparison is conducted for a set of 494 macroeconomic time series recently collected by Dees et al. (Journal of Applied Econometrics 2007; 22: 1–38). In total, 10,374 time series realizations are contrasted against competing short‐, medium‐ and longer‐term purely autoregressive and adjusted predictors. With regard to all forecast horizons, the adjusted predictions consistently outperform conditionally Gaussian forecasts according to cross‐sectional mean group evaluation of absolute forecast errors and directional accuracy. Copyright © 2012 John Wiley & Sons, Ltd.     

How successful are dynamic factor models at forecasting output and inflation? A meta‐analytic approach

This paper uses a meta‐analysis to survey existing factor forecast applications for output and inflation and assesses what causes large factor models to perform better or more poorly at forecasting than other models. Our results suggest that factor models tend to outperform small models, whereas factor forecasts are slightly worse than pooled forecasts. Factor models deliver better predictions for US variables than for UK variables, for US output than for euro‐area output and for euro‐area inflation than for US inflation. The size of the dataset from which factors are extracted positively affects the relative factor forecast performance, whereas pre‐selecting the variables included in the dataset did not improve factor forecasts in the past. Finally, the factor estimation technique may matter as well. Copyright © 2008 John Wiley & Sons, Ltd.     

Combining experts' opinions using a normal-wishart model

This paper examines how a Bayesian decision maker might update her distributions for continuous variables X_i, i=1, 2, …, upon hearing experts' forecasts expressed as quantiles. To utilize the relationship between the decision maker and experts, and to avoid problems associated with different scales and ranges of the variables, we assume that the decision maker transforms the experts' quantiles in terms of her own prior distribution for each X_i. A model using such a transformation is presented and its properties are examined.     

N-step combinations of forecasts

While there is general agreement that a linear combination of forecasts can outperform the individual forecasts, there is controversy about the appropriateness of the combination method to be used in a given situation. Hence, in any given application it may be more beneficial to combine different sets of combined forecasts rather than picking one of them. This paper introduces the concept of N-step combinations of forecasts which involves combining the combined forecasts obtained from different combination procedures used at the preceding step. Using quarterly GNP data, evidence supporting the increase in the accuracy of the one-period-ahead ex-ante forecasts as the combination step increases is provided. The MSE, MAE, MAPE and their corresponding standard deviations are used to evaluate the accuracy of the forecasts obtained.     

Forecasting the duration of short‐term deflation episodes

The paper proposes a simulation‐based approach to multistep probabilistic forecasting, applied for predicting the probability and duration of negative inflation. The essence of this approach is in counting runs simulated from a multivariate distribution representing the probabilistic forecasts, which enters the negative inflation regime. The marginal distributions of forecasts are estimated using the series of past forecast errors, and the joint distribution is obtained by a multivariate copula approach. This technique is applied for estimating the probability of negative inflation in China and its expected duration, with the marginal distributions computed by fitting weighted skew‐normal and two‐piece normal distributions to autoregressive moving average ex post forecast errors and using the multivariate Student t copula.