A non-linear combination of experts' forecasts: A Bayesian approach

A general Bayesian approach to combining n expert forecasts is developed. Under some moderate assumptions on the distributions of the expert errors, it leads to a consistent, monotonic, quasi-linear average formula. This generalizes Bordley's results.     

Do experts' adjustments on model‐based SKU‐level forecasts improve forecast quality?

Model‐based SKU‐level forecasts are often adjusted by experts. In this paper we propose a statistical methodology to test whether these expert forecasts improve on model forecasts. Application of the methodology to a very large database concerning experts in 35 countries who adjust SKU‐level forecasts for pharmaceutical products in seven distinct categories leads to the general conclusion that expert forecasts are equally good at best, but are more often worse than model‐based forecasts. We explore whether this is due to experts putting too much weight on their contribution, and this indeed turns out to be the case. Copyright © 2009 John Wiley & Sons, Ltd.     

An empirical investigation of combinations of economic forecasts

This paper examines the effects of combining three econometric and three times-series forecasts of growth and inflation in the U.K. If forecasts are unbiased then a combination exploiting this fact will be more efficient than an unrestricted combination. Ex post econometric forecasts may be biased but ex ante they are unbiased. The results of the study are that a restricted linear combination of the econometric forecasts is superior to an unrestricted combination and also to the unweighted mean of the forecasts. However, it is not preferred to the best of the individual forecasts.     

System-based weights versus series-specific weights in the combination of forecasts

System-based combination weights for series r/step-length h incorporate relative accuracy information from other forecast step-lengths for r and from other series for step-length h. Such weights are examined utilizing the West and Fullerton (1996) data set-4275 ex ante employment forecasts from structural simultaneous equation econometric models for 19 metropolitan areas at 10 quarterly step-lengths and a parallel set of 4275 ARIMA forecasts. The system-based weights yielded combined forecasts of higher average accuracy and lower risk of large inaccuracy than seven alternative strategies: (1) averaging; (2) relative MSE weights; (3) outperformance (per cent best) weights; (4) Bates and Granger (1969) optimal weights with a convexity constraint imposed; (5) unconstrained optimal weights; (6) select a ‘best’ method (ex ante) by series and; (7) experiment in the Bischoff (1989) sense and select either method (2) or (6) based on the outcome of e experiment. Accuracy gains of the system-based combination were concentrated at step-lengths two to five. Although alternative (5) was generally outperformed, none of the six other alternatives was systematically most accurate when evaluated relative to each other. This contrasts with Bischoff's (1989) results that held promise for an empirically applicable guideline to determine whether or not to combine.     

The combination of macroeconomic forecasts

The paper examines combined forecasts based on two components: forecasts produced by Chase Econometrics and those produced using the Box-Jenkins ARIMA technique. Six series of quarterly ex ante and simulated ex ante forecasts are used over 37 time periods and ten horizons. The forecasts are combined using seven different methods. The best combined forecasts, judged by average relative root-mean-square error, are superior to the Chase forecasts for three variables and inferior for two, though averaged over all six variables the Chase forecasts are slightly better. A two-step procedure produces forecasts for the last half of the sample which, on average, are slightly better than the Chase forecasts.     

Subjective versus objective combining of forecasts: An experiment

The paper reports results of an experiment conducted to evaluate subjective versus objective combination of forecasts. The subjects were undergraduate students at Texas A&M. The students forecasted two different types of time series. The results found show that the subjective combination of forecasts improves their accuracy as compared with individual efforts. Four ex-ante weighting methods were also used to combine the forecasts. They all improve the accuracy of the forecasts. The best results, though, were from the subjective combination of forecasts.     

Professional economic forecasts: Are they worth their costs?

Most economic forecast evaluations dating back 20 years show that professional forecasters add little to the forecasts generated by the simplest of models. Using various types of forecast error criteria, these evaluations usually conclude that the professional forecasts are little better than the no-change or ARIM A type forecast. It is our contention that this conclusion is mistaken because the conventional error criteria may not capture why forecasts are ma& or how they are used. Using forecast directional accuracy, the criterion which has been found to be highly correlated with profits in an interest rate setting, we find that professional GNP forecasts dominate the cheaper alternatives. Moreover, there appears to be no systematic relationship between this preferred criterion and the error measures used in previous studies.     

N-step combinations of forecasts

While there is general agreement that a linear combination of forecasts can outperform the individual forecasts, there is controversy about the appropriateness of the combination method to be used in a given situation. Hence, in any given application it may be more beneficial to combine different sets of combined forecasts rather than picking one of them. This paper introduces the concept of N-step combinations of forecasts which involves combining the combined forecasts obtained from different combination procedures used at the preceding step. Using quarterly GNP data, evidence supporting the increase in the accuracy of the one-period-ahead ex-ante forecasts as the combination step increases is provided. The MSE, MAE, MAPE and their corresponding standard deviations are used to evaluate the accuracy of the forecasts obtained.     

Combining expert‐adjusted forecasts

It is well known that a combination of model‐based forecasts can improve upon each of the individual constituent forecasts. Most forecasts available in practice are, however, not purely based on econometric models but entail adjustments, where experts with domain‐specific knowledge modify the original model forecasts. There is much evidence that expert‐adjusted forecasts do not necessarily improve the pure model‐based forecasts. In this paper we show, however, that combined expert‐adjusted model forecasts can improve on combined model forecasts, in the case when the individual expert‐adjusted forecasts are not better than their associated model‐based forecasts. We discuss various implications of this finding.     

Combination of Forecasts across Estimation Windows: An Application to Air Travel Demand

This paper applies combining forecasts of air travel demand generated from the same model but over different estimation windows. The combination approach used resorts to Pesaran and Pick (Journal of Business Economics and Statistics 2011; 29 : 307–318), but the empirical application is extended in several ways. The forecasts are based on a seasonal Box–Jenkins model (SARIMA), which is adequate to forecast monthly air travel demand with distinct seasonal patterns at the largest German airport: Frankfurt am Main. Furthermore, forecasts with forecast horizons from 1 to 12 months ahead, which are based on different average estimation windows, expanding windows and single rolling windows, are compared with baseline forecasts based on an expanding window of the observations after a structural break. The forecast exercise shows that the average window forecasts mostly outperform the alternative single window forecasts. Copyright © 2016 John Wiley & Sons, Ltd.     

Sources of error in forecasts and expectations: U.K. economic models, 1984–8

This paper presents a comparative analysis of the sources of error in forecasts for the UK economy published over a recent four-year period by four independent groups. This analysis rests on the archiving at the ESRC Macroeconomic Modelling Bureau of the original forecasts together with all their accompanying assumptions and adjustments. A method of decomposing observed forecast errors so as to distinguish the contributions of forecaster and model is set out; the impact of future expectations treated in a ‘model-consistent’ or ‘rational’ manner is specifically considered. The results show that the forecaster's adjustments make a substantial contribution to forecast performance, a good part of which comes from adjustments that bring the model on track at the start of the forecast period. The published ex-ante forecasts are usually superior to pure model-based ex-post forecasts, whose performance indicates some misspecification of the underlying models.     

Do US Macroeconomic Forecasters Exaggerate their Differences?

Application of the Bernhardt et al. (Journal of Financial Economics 2006; 80 (3): 657–675) test of herding to the calendar‐year annual output growth and inflation forecasts suggests forecasters tend to exaggerate their differences, except at the shortest horizon, when they tend to herd. We consider whether these types of behaviour can help to explain the puzzle that professional forecasters sometimes make point predictions and histogram forecasts which are mutually inconsistent. Copyright © 2015 John Wiley & Sons, Ltd.     

Merging monthly and quarterly forecasts: Experience with mqem

Forecasts from quarterly econometric models are typically revised on a monthly basis to reflect the information in current economic data. The revision process usually involves setting targets for the quarterly values of endogenous variables for which monthly observations are available and then altering the intercept terms in the quarterly forecasting model to achieve the target values. A formal statistical approach to the use of monthly data to update quarterly forecasts is described and the procedure is applied to the Michigan Quarterly Econometric Model of the US Economy. The procedure is evaluated in terms of both ex post and ex ante forecasting performance. The ex ante results for 1986 and 1987 indicate that the method is quite promising. With a few notable exceptions, the formal procedure produces forecasts of GNP growth that are very close to the published ex ante forecasts.     

Bayesian consensus forecasts of macroeconomic variables

Economists, like other forecasters, share knowledge, data and theories in common. Consequently, their forecast errors are likely to be highly dependent. This paper reports on an empirical study of 16 macroeconomic forecasters. Composite forecasts are computed using a sequential weighting scheme that takes dependence into account; these are compared to a simple average and median forecasts. A within-sample composite is also calculated. Both these methods perform significantly better than the average or median of the forecasts. This improvement in accuracy is apparently because the dependence between the forecasters' errors is so high that the optimal composite forecasts sometimes lie outside the range of the individual forecasts.     

Testing Interval Forecasts: A GMM‐Based Approach

This paper proposes a new evaluation framework for interval forecasts. Our model‐free test can be used to evaluate interval forecasts and high‐density regions, potentially discontinuous and/or asymmetric. Using a simple J‐statistic, based on the moments defined by the orthonormal polynomials associated with the binomial distribution, this new approach presents many advantages. First, its implementation is extremely easy. Second, it allows for a separate test for unconditional coverage, independence and conditional coverage hypotheses. Third, Monte Carlo simulations show that for realistic sample sizes our GMM test has good small‐sample properties. These results are corroborated by an empirical application on SP500 and Nikkei stock market indexes. It confirms that using this GMM test leads to major consequences for the ex post evaluation of interval forecasts produced by linear versus nonlinear models. Copyright © 2011 John Wiley & Sons, Ltd.     

Daily volatility forecasts: reassessing the performance of GARCH models

Volatility plays a key role in asset and portfolio management and derivatives pricing. As such, accurate measures and good forecasts of volatility are crucial for the implementation and evaluation of asset and derivative pricing models in addition to trading and hedging strategies. However, whilst GARCH models are able to capture the observed clustering effect in asset price volatility in‐sample, they appear to provide relatively poor out‐of‐sample forecasts. Recent research has suggested that this relative failure of GARCH models arises not from a failure of the model but a failure to specify correctly the ‘true volatility’ measure against which forecasting performance is measured. It is argued that the standard approach of using ex post daily squared returns as the measure of ‘true volatility’ includes a large noisy component. An alternative measure for ‘true volatility’ has therefore been suggested, based upon the cumulative squared returns from intra‐day data. This paper implements that technique and reports that, in a dataset of 17 daily exchange rate series, the GARCH model outperforms smoothing and moving average techniques which have been previously identified as providing superior volatility forecasts. Copyright © 2004 John Wiley & Sons, Ltd.     

Comparing ex-ante forecasts from a sem and var model: An application to the italian economy

The predictive performance of a large-scale structural econometric model (SEM) of the Italian economy the Prometeia model is compared in this paper with a vector autoregressive (VAR) model estimated for a selection of six main variables of interest. The paper concentrates on the quarterly ex-ante forecasts of GDP growth rate and the annual forecasts of GDP growth and inflation rate, over the period 1980-85. It concludes that no forecaster is systematically better than the other. In particular, the VAR model outperforms the SEM in short-run forecasts, suggesting that, for the latter, more careful attention should be addressed to questions of dynamic specification. On the other hand, for longer intervals, the SEM forecasts are more accurate than the VAR forecasts, in that they can benefit from the judgemental interventions of the model users and the model can pick up the non-linearities of the economy which cannot be captured by the VAR. Given the different kinds of information that can be extracted from the two approaches, it seems more reasonable to consider them as complementary rather than alternative tools for modelling and forecasting. Therefore, rather than attempting to establish the superiority of one type of model over the other, this kind of comparisons should be seen as a useful diagnostic tool for detecting types of model misspecification.     

Estimating the strength of expert judgement: The case of US mortality forecasts

The use of expert judgement is an important part of demographic forecasting. However, because judgement enters into the forecasting process in an informal way, it has been very difficult to assess its role relative to the analysis of past data. The use of targets in demographic forecasts permits us to embed the subjective forecasting process into a simple time-series regression model, in which expert judgement is incorporated via mixed estimation. The strength of expert judgement is denned, and estimated using the official forecasts of cause-specific mortality in the United States. We show that the weight given to judgement varies in an improbable manner by age. Overall, the weight given to judgement appears too high. An alternative approach to combining expert judgement and past data is suggested.     

Combining exchange rate forecasts: What is the optimal consensus measure?

In this paper a high-quality disaggregate database is utilized to examine whether individual forecasters produce efficient exchange rate predictions and also if the properties of the forecasts change when they are combined. The paper links a number of themes in the exchange rate literature and examines various methods of forecast combination. It is demonstrated, inter alia, that some forecasters are better than others, but that most are not as good as a naive no-change prediction. Combining forecasts adds to the accuracy of the predictions, but the gains mainly reflect the removal of systematic and unstable bias.