Short-run macroeconomic forecasting: The OECD performance

Forecasts for the seven major industrial countries, Canada, France, Germany, Italy, Japan, the United Kingdom and the United States, are published on a regular basis in the OECD's Economic Outlook. This paper analyses the accuracy of the OECD annual forecasts of output and price changes and of the current balance in the balance of payments. As a reference basis, the forecasts are compared with those generated by a naive model, a random walk process. The measures of forecasting accuracy used are the mean-absolute error, the root-mean-square error, the median-absolute error, and Theil's inequality coefficient. The OECD forecasts of real GNP changes are significantly superior to those generated by the random walk process; however, the OECD price and current balance forecasts are not significantly more accurate than those obtained from the naive model. The OECD's forecasting performance has neither improved nor deteriorated over time.     

A note on diagrams for turning-point diagnostics

A new diagram for discriminating a forecasting technique's history with respect to the two types of turning-point errors, missed turns and false signals is presented. This diagram extends Theil's original prediction-realization (P-R) presentation so that each of the four quadrants and the axes locates a distinct type of forecasting error or correct forecast, thus reducing much of the former ambiguity. This improvement is accomplished without the loss of any information.     

Adjusting Judgemental Extrapolations using Theil's Method and Discounted Weighted Regression

Theil's method can be applied to judgemental forecasts to remove systematic errors. However, under conditions of change the method can reduce the accuracy of forecasts by correcting for biases that no longer apply. In these circumstances, it may be worth applying an adaptive correction model which attaches a greater weight to more recent observations. This paper reports on the application of Theil's original method and a discounted weighted regression form of Theil's method (DWR) to the judgemental extrapolations made by 100 subjects in an experiment. Extrapolations were made for both stationary and non-stationary and low- and high-noise series. The results suggest DWR can lead to significant improvements in accuracy where the underlying time-series signal becomes more discernible over time or where the signal is subject to change. Theil's method appears to be most effective when a series has a high level of noise. However, while Theil's corrections seriously reduced the accuracy of judgemental extrapolations for some series the DWR method performed well under a wide range of conditions and never significantly degraded the original forecasts. © 1997 by John Wiley & Sons, Ltd.     

Sources of error in forecasts and expectations: U.K. economic models, 1984–8

This paper presents a comparative analysis of the sources of error in forecasts for the UK economy published over a recent four-year period by four independent groups. This analysis rests on the archiving at the ESRC Macroeconomic Modelling Bureau of the original forecasts together with all their accompanying assumptions and adjustments. A method of decomposing observed forecast errors so as to distinguish the contributions of forecaster and model is set out; the impact of future expectations treated in a ‘model-consistent’ or ‘rational’ manner is specifically considered. The results show that the forecaster's adjustments make a substantial contribution to forecast performance, a good part of which comes from adjustments that bring the model on track at the start of the forecast period. The published ex-ante forecasts are usually superior to pure model-based ex-post forecasts, whose performance indicates some misspecification of the underlying models.     

Reducing uncertainty in short-term projections: Linkage of monthly and quarterly models

This paper shows how monthly data and forecasts can be used in a systematic way to improve the predictive accuracy of a quarterly macroeconometric model. The problem is formulated as a model pooling procedure (equivalent to non-recursive Kalman filtering) where a baseline quarterly model forecast is modified through ‘add-factors’ or ‘constant adjustments’. The procedure ‘automatically’ constructs these adjustments in a covariance-minimizing fashion to reflect the revised expectation of the quarterly model's forecast errors, conditional on the monthly information set. Results obtained using Federal Reserve Board models indicate the potential for significant reduction in forecast error variance through application of these procedures.     

Mincer–Zarnowitz quantile and expectile regressions for forecast evaluations under aysmmetric loss functions

Forecasts are pervasive in all areas of applications in business and daily life. Hence evaluating the accuracy of a forecast is important for both the generators and consumers of forecasts. There are two aspects in forecast evaluation: (a) measuring the accuracy of past forecasts using some summary statistics, and (b) testing the optimality properties of the forecasts through some diagnostic tests. On measuring the accuracy of a past forecast, this paper illustrates that the summary statistics used should match the loss function that was used to generate the forecast. If there is strong evidence that an asymmetric loss function has been used in the generation of a forecast, then a summary statistic that corresponds to that asymmetric loss function should be used in assessing the accuracy of the forecast instead of the popular root mean square error or mean absolute error. On testing the optimality of the forecasts, it is demonstrated how the quantile regressions set in the prediction–realization framework of Mincer and Zarnowitz (in J. Mincer (Ed.), Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance (pp. 14–20), 1969) can be used to recover the unknown parameter that controls the potentially asymmetric loss function used in generating the past forecasts. Finally, the prediction–realization framework is applied to the Federal Reserve's economic growth forecast and forecast sharing in a PC manufacturing supply chain. It is found that the Federal Reserve values overprediction approximately 1.5 times more costly than underprediction. It is also found that the PC manufacturer weighs positive forecast errors (under forecasts) about four times as costly as negative forecast errors (over forecasts).     

Towards a taxonomy of forecast error measures a factor-comparative investigation of forecast error dimensions

This study explores the nature of information conveyed by 14 error measures drawn from the literature, using real-life forecasting data from 691 individual product items over six quarterly periods. Principal components analysis is used to derive factor solutions that are subsequently compared for two forecasting methods, a version of Holt's exponential smoothing, and the random walk model (Naive 1). The results reveal four underlying forecast error dimensions that are stable across the two factor solutions. The potentially confounding influence of sales volume on the derived error dimensions is also explored via correlation analysis.     

The maximum and minimum of primary forecasts

The purpose of this paper is to suggest that the maximum (or minimum) of a number of primary forecasts may make a valuable addition to the forecasting accuracy of a combination of forecasts. Such forecasts are readily computable. Theoretical results are presented for two unbiased forecasts with correlated normally distributed errors, showing that the maximum (minimum) of two forecasts can have a smaller error variance than either of the primary forecasts and the forecast error can have low correlation with the primary errors. Empirical results are obtained for two different sets of forecasts available in the literature, and it is observed that a combination forecast including the maximum and/or minimum has attractive forecasting properties.     

Algorithms for explaining forecast revisions

Forecasts are routinely revised, and these revisions are often the subject of informal analysis and discussion. This paper argues (1) that forecast revisions are analyzed because they help forecasters and forecast users to evaluate forecasts and forecasting procedures and (2) that these analyses can be sharpened by using the forecasting model to systematically express its forecast revision as the sum of components identified with specific subsets of new information, such as data revisions and forecast errors. An algorithm for this purpose is explained and illustrated.     

Expectations,use and judgmental adjustment of external financial and economic forecasts: an empirical investigation

A survey of 124 users of externally produced financial and economic forecasts in Turkey investigated their expectations and perceptions of forecast quality and their reasons for judgmentally adjusting forecasts. Expectations and quality perceptions mainly related to the timeliness of forecasts, the provision of a clear justifiable rationale and accuracy. Cost was less important. Forecasts were frequently adjusted when they lacked a justifiable explanation, when the user felt they could integrate their knowledge into the forecast, or where the user perceived a need to take responsibility for the forecast. Forecasts were less frequently adjusted when they came from a well‐known source and were based on sound explanations and assumptions. The presence of feedback on accuracy reduced the influence of these factors. The seniority and experience of users had little effect on their attitudes or propensity to make adjustments. Copyright © 2008 John Wiley & Sons, Ltd.     

Performance evaluation of neural network architectures: the case of predicting foreign exchange correlations

In the last decade, neural networks have emerged from an esoteric instrument in academic research to a rather common tool assisting auditors, investors, portfolio managers and investment advisors in making critical financial decisions. It is apparent that a better understanding of the network's performance and limitations would help both researchers and practitioners in analysing real‐world problems. Unlike many existing studies which focus on a single type of network architecture, this study evaluates and compares the performance of models based on two competing neural network architectures, the multi‐layered feedforward neural network (MLFN) and general regression neural network (GRNN). Our empirical evaluation measures the network models' strength on the prediction of currency exchange correlation with respect to a variety of statistical tests including RMSE, MAE, U statistic, Theil's decomposition test, Henriksson–Merton market timing test and Fair–Shiller informational content test. Results of experiments suggest that the selection of proper architectural design may contribute directly to the success in neural network forecasting. In addition, market timing tests indicate that both MLFN and GRNN models have economically significant values in predicting the exchange rate correlation. On the other hand, informational content tests discover that the neural network models based on different architectures capture useful information not found in each other and the information sets captured by the two network designs are independent of one another. An auxiliary experiment is developed and confirms the possible synergetic effect from combining forecasts made by the two different network architectures and from incorporating information from an implied correlation model into the neural network forecasts. Implied correlation and random walk models are also included in our empirical experiment for benchmark comparison. Copyright © 2005 John Wiley & Sons, Ltd.     

Optimal forecast error as an unbiased estimator of abnormal return: A proposition

In the event studies, the accuracy of the abnormal returns assessment is highly dependent on the accuracy of the preceding expected return model. If the expected return model is inadequate, there is a possibility that a part of returns is labeled as abnormal returns even though they are not. Currently, we have a variety of options to set up an expected return model. To obtain unbiased abnormal returns, one should pay attention to the performance of the expected return model. In this research, we propose that the optimal forecast lemma can be consulted beforehand so that minimizing the optimal forecast error in the expected return model will yield unbiased abnormal returns. We introduce and prove a proposition that the optimal forecast error is an unbiased estimator for abnormal return. The proposition induces assessing the performance of abnormal return estimation to preemptively evaluate the out-sample forecast accuracy of the model employed. In an illustrative dataset, we examine various models. The approach requires preliminary computational effort; however, it is useful for accurately obtaining the abnormal return predictions.     

An examination of factors contributing to delphi accuracy

Three experiments examined the accuracy in the Delphi method. The first experiment assessed the accuracy of group predictions over 1-, 2- and 3- month time spans. Results indicated that predictions derived from the group were more accurate than those of 95 per cent of the individual panelists, but did not exceed in accuracy the best panelists. Experiment 2 evaluated the gross contributions of polling and feedback to Delphi accuracy. The manipulations did not improve the group's ability to forecast the probabilities of the occurrence of events, but did decrease the error in predicting when the events would occur. Experiment 3 separated the effects of polling and feedback as determinants of accuracy. Neither manipulation improved the accuracy of the group's predictions of whether an event would occur. The effect of iterated polling was to reduce the group's error in predicting the time course for those scenarios that did occur.     

Comments on testing for forecast and specification optimality using linear composites

This paper extends the method of using linear composites of forecasts for testing the efficiency of one forecast compared to a finite collection of other forecasts. It also gives necessary and sufficient conditions for forecast optimality in the mean square error sense. Information sets are found to be unnecessary for forecast optimality. Finally, the paper shows that a consistent test of forecast optimality cannot in general be obtained using linear composites. A similar conclusion applies to tests of specification optimality based on linear composites.     

Estimation and Prediction Tests of Cash Flow Forecast Accuracy

Recently, analysts' cash flow forecasts have become widely available through financial information services. Cash flow information enables practitioners to better understand the real operating performance and financial stability of a company, particularly when earnings information is noisy and of low quality. However, research suggests that analysts' cash flow forecasts are less accurate and more dispersed than earnings forecasts. We thus investigate factors influencing cash flow forecast accuracy and build a practical model to distinguish more accurate from less accurate cash flow forecasters, using past cash flow forecast accuracy and analyst characteristics. We find significant power in our cash flow forecast accuracy prediction models. We also find that analysts develop cash flow‐specific forecasting expertise and knowhow, which are distinct from those that analysts acquire from forecasting earnings. In particular, cash flow‐specific information is more useful in identifying accurate cash flow forecasters than earnings‐specific information.Copyright © 2011 John Wiley & Sons, Ltd.     

A new insight into combining forecasts for elections: The role of social media

This study is devoted to gain insight into a timely, accurate, and relevant combining forecast by considering social media (Facebook), opinion polls, and prediction markets. We transformed each type of raw data into the possibility of victory as a forecasting model. Besides the four single forecasts, namely Facebook fans, Facebook “people talking about this” (PTAT) statistics, opinion polls, and prediction markets, we generated three combined forecasts by associating various combinations of the four components. Then, we examined the predictive performance of each forecast on vote shares and the elected/non‐elected outcome across the election period. Our findings, based on the evidence of Taiwan's 2018 county and city elections, showed that incorporating the Facebook PTAT statistic with polls and prediction markets generates the most powerful forecast. Moreover, we recognized the matter of the time horizons where the best proposed model has better accuracy gains in prediction—in the “late of election,” but not in “approaching election”. The patterns of the trend of accuracy across time for each forecasting model also differ from one another. We also highlighted the complementarity of various types of data in the paper because each forecast makes important contributions to forecasting elections.     

Analyzing Macroeconomic Forecastability

This paper estimates, using stochastic simulation and a multi‐country macroeconometric model, the fraction of the forecast error variance of output changes and the fraction of the forecast error variance of inflation that are due to unpredictable asset price changes. The results suggest that between about 25% and 37% of the forecast error variance of output growth over eight quarters is due to asset price changes and between about 33% and 60% of the forecast error variance of inflation over eight quarters is due to asset price changes. These estimates provide limits to the accuracy that can be expected from macroeconomic forecasting. Copyright © 2011 John Wiley & Sons, Ltd.     

Method and situational factors in sales forecast accuracy

In this study the interaction of forecasting method (econometric versus exponential smoothing) and two situational factors are evaluated for their effects upon accuracy. Data from two independent sets of ex ante quarterly forecasts for 19 classes of mail were used to test hypotheses. Counter to expectations, the findings revealed that forecasting method did not interact with the forecast time horizon (short versus long term). However, as hypothesized, forecasting method interacted significantly with product/market definition (First Class versus other mail), an indicator of buyer sensitivity to marketing/environmental changes. Results are discussed in the context of future research on forecast accuracy.     

Forecast Combination and Bayesian Model Averaging: A Prior Sensitivity Analysis

In this study we evaluate the forecast performance of model‐averaged forecasts based on the predictive likelihood carrying out a prior sensitivity analysis regarding Zellner's g prior. The main results are fourfold. First, the predictive likelihood does always better than the traditionally employed ‘marginal’ likelihood in settings where the true model is not part of the model space. Secondly, forecast accuracy as measured by the root mean square error (RMSE) is maximized for the median probability model. On the other hand, model averaging excels in predicting direction of changes. Lastly, g should be set according to Laud and Ibrahim (1995: Predictive model selection. Journal of the Royal Statistical Society B 57 : 247–262) with a hold‐out sample size of 25% to minimize the RMSE (median model) and 75% to optimize direction of change forecasts (model averaging). We finally apply the aforementioned recommendations to forecast the monthly industrial production output of six countries, beating for almost all countries the AR(1) benchmark model. Copyright © 2011 John Wiley & Sons, Ltd.     

When are Direct Multi‐step and Iterative Forecasts Identical?

Although both direct multi‐step‐ahead forecasting and iterated one‐step‐ahead forecasting are two popular methods for predicting future values of a time series, it is not clear that the direct method is superior in practice, even though from a theoretical perspective it has lower mean squared error (MSE). A given model can be fitted according to either a multi‐step or a one‐step forecast error criterion, and we show here that discrepancies in performance between direct and iterative forecasting arise chiefly from the method of fitting, and is dictated by the nuances of the model's misspecification. We derive new formulas for quantifying iterative forecast MSE, and present a new approach for assessing asymptotic forecast MSE. Finally, the direct and iterative methods are compared on a retail series, which illustrates the strengths and weaknesses of each approach. Copyright © 2015 John Wiley & Sons, Ltd.