Testing for unreliable estimators and insignificant forecasts in combined forecasts

The reliability and precision of the weights used in combining individual forecasts, irrespective of the method of combination, is important in evaluating a combined forecast. The objective of this study is not to suggest the ‘best’ method of combining individual forecasts, but rather to propose exploratory procedures, that make use of all available sample information contained in the covariance matrix of individual forecast errors, to (1) detect if the weights used in combining forecasts are ‘reliable’ (and ‘stable’ if it is known that the covariance matrix of forecast errors is stationary over time) and (2) test for ‘insignificant’ individual forecasts used in forming a combined forecast. We present empirical applications using two-year sales and individual forecast data provided by a major consumer durables manufacturer to illustrate the feasibility of our proposed procedures.     

Disaggregation methods to expedite product line forecasting

This paper addresses the issue of forecasting individual items within a product line; where each line includes several independent but closely related products. The purpose of the research was to reduce the overall forecasting burden by developing and assessing schemes of disaggregating forecasts of a total product line to the related individual items. Measures were developed to determine appropriate disaggregated methodologies and to compare the forecast accuracy of individual product forecasts versus disaggregated totals. Several of the procedures used were based upon extensions of the combination of forecast research and applied to disaggregations of total forecasts of product lines. The objective was to identify situations when it was advantageous to produce disaggregated forecasts, and if advantageous, which method of disaggregation to utilize. This involved identification of the general conceptual characteristics within a set of product line data that might cause a disaggregation method to produce relatively accurate forecasts. These conceptual characteristics provided guidelines for forecasters on how to select a disaggregation method and under what conditions a particular method is applicable.     

Evaluating the predictive accuracy of volatility models

Standard statistical loss functions, such as mean‐squared error, are commonly used for evaluating financial volatility forecasts. In this paper, an alternative evaluation framework, based on probability scoring rules that can be more closely tailored to a forecast user's decision problem, is proposed. According to the decision at hand, the user specifies the economic events to be forecast, the scoring rule with which to evaluate these probability forecasts, and the subsets of the forecasts of particular interest. The volatility forecasts from a model are then transformed into probability forecasts of the relevant events and evaluated using the selected scoring rule and calibration tests. An empirical example using exchange rate data illustrates the framework and confirms that the choice of loss function directly affects the forecast evaluation results. Copyright © 2001 John Wiley & Sons, Ltd.     

Evaluating forecasts: a look at aggregate bias and accuracy measures

In this paper an investigation is made of the properties and use of two aggregate measures of forecast bias and accuracy. These are metrics used in business to calculate aggregate forecasting performance for a family (group) of products. We find that the aggregate measures are not particularly informative if some of the one‐step‐ahead forecasts are biased. This is likely to be the case in practice if frequently employed forecasting methods are used to generate a large number of individual forecasts. In the paper, examples are constructed to illustrate some potential problems in the use of the metrics. We propose a simple graphical display of forecast bias and accuracy to supplement the information yielded by the accuracy measures. This support includes relevant boxplots of measures of individual forecasting success. This tool is simple but helpful as the graphic display has the potential to indicate forecast deterioration that can be masked by one or both of the aggregate metrics. The procedures are illustrated with data representing sales of food items. Copyright © 2005 John Wiley & Sons, Ltd.     

Composite model building for foreign exchange rates

It has been shown in recent economic and statistical studies that composite forecasts may produce more accurate forecasts than individual ones. The purpose of this study is to develop composite forecasting models that may produce forecasts superior to the individual forecast implicit in forward exchange rates. In an efficient market one would expect to find little improvement with the composite models relative to the forward exchange rate.     

Judgemental revision of sales forecasts: A longitudinal eetension

When quantitative models are used for short-term multi-item sales forecasts it is possible that the managers who use such forecasts may disagree with at least some of the estimates obtained, and wish to change them so that they become more consistent with their own (subjective) evaluation of the marketplace. This study reports on an analysis of the effectiveness of judgemental revision of sales forecasts over six quarterly forecasting periods. The results give general support for the practice of forecast manipulation as a means of improving forecasting accuracy. It is also observed that the effectiveness of revision activity varies across different time periods.     

Realized Volatility Forecast of Stock Index Under Structural Breaks

We investigate the realized volatility forecast of stock indices under the structural breaks. We utilize a pure multiple mean break model to identify the possibility of structural breaks in the daily realized volatility series by employing the intraday high‐frequency data of the Shanghai Stock Exchange Composite Index and the five sectoral stock indices in Chinese stock markets for the period 4 January 2000 to 30 December 2011. We then conduct both in‐sample tests and out‐of‐sample forecasts to examine the effects of structural breaks on the performance of ARFIMAX‐FIGARCH models for the realized volatility forecast by utilizing a variety of estimation window sizes designed to accommodate potential structural breaks. The results of the in‐sample tests show that there are multiple breaks in all realized volatility series. The results of the out‐of‐sample point forecasts indicate that the combination forecasts with time‐varying weights across individual forecast models estimated with different estimation windows perform well. In particular, nonlinear combination forecasts with the weights chosen based on a non‐parametric kernel regression and linear combination forecasts with the weights chosen based on the non‐negative restricted least squares and Schwarz information criterion appear to be the most accurate methods in point forecasting for realized volatility under structural breaks. We also conduct an interval forecast of the realized volatility for the combination approaches, and find that the interval forecast for nonlinear combination approaches with the weights chosen according to a non‐parametric kernel regression performs best among the competing models. Copyright © 2014 John Wiley & Sons, Ltd.     

Combination of Forecasts across Estimation Windows: An Application to Air Travel Demand

This paper applies combining forecasts of air travel demand generated from the same model but over different estimation windows. The combination approach used resorts to Pesaran and Pick (Journal of Business Economics and Statistics 2011; 29 : 307–318), but the empirical application is extended in several ways. The forecasts are based on a seasonal Box–Jenkins model (SARIMA), which is adequate to forecast monthly air travel demand with distinct seasonal patterns at the largest German airport: Frankfurt am Main. Furthermore, forecasts with forecast horizons from 1 to 12 months ahead, which are based on different average estimation windows, expanding windows and single rolling windows, are compared with baseline forecasts based on an expanding window of the observations after a structural break. The forecast exercise shows that the average window forecasts mostly outperform the alternative single window forecasts. Copyright © 2016 John Wiley & Sons, Ltd.     

Judgmental adjustment of earnings forecasts

Earnings forecasts have received a great deal of attention, much of which has centered on the comparative accuracy of judgmental and objective forecasting methods. Recently, studies have focused on the use of combinations of subjective and objective forecasts to improve forecast accuracy. This research offers an extension on this theme by subjectively modifying an objective forecast. Specifically, ARIMA forecasts are judgmentally adjusted by analysts using a structured approach based on Saaty's (1980) analytic hierarchy process. The results show that the accuracy of the unadjusted objective forecasts can be improved when judgmentally adjusted.     

Combination forecasts of output growth in a seven‐country data set

This paper uses forecast combination methods to forecast output growth in a seven‐country quarterly economic data set covering 1959–1999, with up to 73 predictors per country. Although the forecasts based on individual predictors are unstable over time and across countries, and on average perform worse than an autoregressive benchmark, the combination forecasts often improve upon autoregressive forecasts. Despite the unstable performance of the constituent forecasts, the most successful combination forecasts, like the mean, are the least sensitive to the recent performance of the individual forecasts. While consistent with other evidence on the success of simple combination forecasts, this finding is difficult to explain using the theory of combination forecasting in a stationary environment. Copyright © 2004 John Wiley & Sons, Ltd.     

Out‐of‐sample equity premium prediction: A scenario analysis approach

We propose two methods of equity premium prediction with single and multiple predictors respectively and evaluate their out‐of‐sample performance using US stock data with 15 popular predictors for equity premium prediction. The first method defines three scenarios in terms of the expected returns under the scenarios and assumes a Markov chain governing the occurrence of the scenarios over time. It employs predictive quantile regressions of excess return on a predictor for three quantiles to estimate the occurrence of the scenarios over an in‐sample period and the transition probabilities of the Markov chain, predicts the expected returns under the scenarios, and generates an equity premium forecast by combining the predicted expected returns under three scenarios with the estimated transition probabilities. The second method generates an equity premium forecast by combining the individual forecasts from the first method across all predictors. For most of predictors, the first method outperforms the benchmark method of historical average and the traditional predictive linear regression with a single predictor both statistically and economically, and the second method consistently performs better than several competing methods used in the literature. The performance of our methods is further examined under different scenarios and economic conditions, and is robust for two different out‐of‐sample periods and specifications of the scenarios.     

Forecast Combination for Euro Area Inflation: A Cure in Times of Crisis?

The period of extraordinary volatility in euro area headline inflation starting in 2007 raised the question whether forecast combination methods can be used to hedge against bad forecast performance of single models during such periods and provide more robust forecasts. We investigate this issue for forecasts from a range of short‐term forecasting models. Our analysis shows that there is considerable variation of the relative performance of the different models over time. To take that into account we suggest employing performance‐based forecast combination methods—in particular, one with more weight on the recent forecast performance. We compare such an approach with equal forecast combination that has been found to outperform more sophisticated forecast combination methods in the past, and investigate whether it can improve forecast accuracy over the single best model. The time‐varying weights assign weights to the economic interpretations of the forecast stemming from different models. We also include a number of benchmark models in our analysis. The combination methods are evaluated for HICP headline inflation and HICP excluding food and energy. We investigate how forecast accuracy of the combination methods differs between pre‐crisis times, the period after the global financial crisis and the full evaluation period, including the global financial crisis with its extraordinary volatility in inflation. Overall, we find that forecast combination helps hedge against bad forecast performance and that performance‐based weighting outperforms simple averaging. Copyright © 2017 John Wiley & Sons, Ltd.     

Can cointegration‐based forecasting outperform univariate models? An application to Asian exchange rates

Conventional wisdom holds that restrictions on low‐frequency dynamics among cointegrated variables should provide more accurate short‐ to medium‐term forecasts than univariate techniques that contain no such information; even though, on standard accuracy measures, the information may not improve long‐term forecasting. But inconclusive empirical evidence is complicated by confusion about an appropriate accuracy criterion and the role of integration and cointegration in forecasting accuracy. We evaluate the short‐ and medium‐term forecasting accuracy of univariate Box–Jenkins type ARIMA techniques that imply only integration against multivariate cointegration models that contain both integration and cointegration for a system of five cointegrated Asian exchange rate time series. We use a rolling‐window technique to make multiple out of sample forecasts from one to forty steps ahead. Relative forecasting accuracy for individual exchange rates appears to be sensitive to the behaviour of the exchange rate series and the forecast horizon length. Over short horizons, ARIMA model forecasts are more accurate for series with moving‐average terms of order >1. ECMs perform better over medium‐term time horizons for series with no moving average terms. The results suggest a need to distinguish between ‘sequential’ and ‘synchronous’ forecasting ability in such comparisons. Copyright © 2002 John Wiley & Sons, Ltd.     

The usefulness of heuristic N(E)RLS algorithms for combining forecasts

There exists theoretical and empirical evidence on the efficiency and robustness of Non-negativity Restricted Least Squares combinations of forecasts. However, the computational complexity of the method hinders its widespread use in practice. We examine various optimizing and heuristic computational algorithms for estimating NRLS combination models and provide certain CPU-time reducing implementations. We empirically compare the combination weights identified by the alternative algorithms and their computational demands based on a total of more than 66,000 models estimated to combine the forecasts of 37 firm-specific accounting earnings series. The ex ante prediction accuracies of combined forecasts from the optimizing versus heuristic algorithms are compared. The effects of fit sample size, model specification, multicollinearity, correlations of forecast errors, and series and forecast variances on the relative accuracy of the optimizing versus heuristic algorithms are analysed. The results reveal that, in general, the computationally simple heuristic algorithms perform as well as the optimizing algorithms. No generalizable conclusions could be reached, however, about which algorithm should be used based on series and forecast characteristics. © 1997 John Wiley & Sons, Ltd.     

To combine or not to combine? issues of combining forecasts

This paper addresses issues such as: Does it always pay to combine individual forecasts of a variable? Should one combine an unbiased forecast with one that is heavily biased? Should one use optimal weights as suggested by Bates and Granger over twenty years ago? A simple model which accounts for the main features of individual forecasts is put forward. Bayesian analysis of the model using noninformative and informative prior probability densities is provided which extends and generalizes results obtained by Winkler (1981) and compared with non-Bayesian methods of combining forecasts relying explicitly on a statistical model for the individual forecasts. It is shown that in some instances it is sensible to use a simple average of individual forecasts instead of using Bates and Granger type weights. Finally, model uncertainty is considered and the issue of combining different models for individual forecasts is addressed.     

Forecast intervals of net migration: the case of the Netherlands

This study addresses problems concerning the forecasting of net migration in the preparation of population forecasts. "As the width of forecast intervals for migration in single years differs strongly from that of an interval for average migration during the forecast period, it is important that the forecaster indicates which type of interval is presented. A comparison of forecast intervals for net migration obtained from an ARIMA model to intervals in official Dutch national population forecasts shows that the uncertainty on migration has been underestimated in past official forecasts."     

Political manoeuvrings as sources of measurement errors in forecasts

We test the extent to which political manoeuvrings can be the sources of measurement errors in forecasts. Our objective is to examine the forecast error based on a simple model in which we attempt to explain deviations between the March budget forecast and the November forecast, and deviations between the outcome and the March budget forecast in the UK. The analysis is based on forecasts made by the general government. We use the forecasts of the variables as alternatives to the outcomes. We also test for political spins in the GDP forecast updates and the GDP forecast errors. We find evidence of partisan and electoral effects in forecast updates and forecast errors. Copyright © 2005 John Wiley & Sons, Ltd.     

Professional economic forecasts: Are they worth their costs?

Most economic forecast evaluations dating back 20 years show that professional forecasters add little to the forecasts generated by the simplest of models. Using various types of forecast error criteria, these evaluations usually conclude that the professional forecasts are little better than the no-change or ARIM A type forecast. It is our contention that this conclusion is mistaken because the conventional error criteria may not capture why forecasts are ma& or how they are used. Using forecast directional accuracy, the criterion which has been found to be highly correlated with profits in an interest rate setting, we find that professional GNP forecasts dominate the cheaper alternatives. Moreover, there appears to be no systematic relationship between this preferred criterion and the error measures used in previous studies.     

Economic and statistical measures of forecast accuracy

This paper argues in favour of a closer link between the decision and the forecast evaluation problems. Although the idea of using decision theory for forecast evaluation appears early in the dynamic stochastic programming literature, and has continued to be used with meteorological forecasts, it is hardly mentioned in standard academic texts on economic forecasting. Some of the main issues involved are illustrated in the context of a two‐state, two‐action decision problem as well as in a more general setting. Relationships between statistical and economic methods of forecast evaluation are discussed and links between the Kuipers score used as a measure of forecast accuracy in the meteorology literature and the market timing tests used in finance are established. An empirical application to the problem of stock market predictability is also provided, and the conditions under which such predictability could be explained in the presence of transaction costs are discussed. Copyright © 2000 John Wiley & Sons, Ltd.     

A comparison of temperature density forecasts from GARCH and atmospheric models

Density forecasts for weather variables are useful for the many industries exposed to weather risk. Weather ensemble predictions are generated from atmospheric models and consist of multiple future scenarios for a weather variable. The distribution of the scenarios can be used as a density forecast, which is needed for pricing weather derivatives. We consider one to 10‐day‐ahead density forecasts provided by temperature ensemble predictions. More specifically, we evaluate forecasts of the mean and quantiles of the density. The mean of the ensemble scenarios is the most accurate forecast for the mean of the density. We use quantile regression to debias the quantiles of the distribution of the ensemble scenarios. The resultant quantile forecasts compare favourably with those from a GARCH model. These results indicate the strong potential for the use of ensemble prediction in temperature density forecasting. Copyright © 2004 John Wiley & Sons, Ltd.