Mincer–Zarnowitz quantile and expectile regressions for forecast evaluations under aysmmetric loss functions

Forecasts are pervasive in all areas of applications in business and daily life. Hence evaluating the accuracy of a forecast is important for both the generators and consumers of forecasts. There are two aspects in forecast evaluation: (a) measuring the accuracy of past forecasts using some summary statistics, and (b) testing the optimality properties of the forecasts through some diagnostic tests. On measuring the accuracy of a past forecast, this paper illustrates that the summary statistics used should match the loss function that was used to generate the forecast. If there is strong evidence that an asymmetric loss function has been used in the generation of a forecast, then a summary statistic that corresponds to that asymmetric loss function should be used in assessing the accuracy of the forecast instead of the popular root mean square error or mean absolute error. On testing the optimality of the forecasts, it is demonstrated how the quantile regressions set in the prediction–realization framework of Mincer and Zarnowitz (in J. Mincer (Ed.), Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance (pp. 14–20), 1969) can be used to recover the unknown parameter that controls the potentially asymmetric loss function used in generating the past forecasts. Finally, the prediction–realization framework is applied to the Federal Reserve's economic growth forecast and forecast sharing in a PC manufacturing supply chain. It is found that the Federal Reserve values overprediction approximately 1.5 times more costly than underprediction. It is also found that the PC manufacturer weighs positive forecast errors (under forecasts) about four times as costly as negative forecast errors (over forecasts).     

The use of encompassing tests for forecast combinations

This paper proposes an algorithm that uses forecast encompassing tests for combining forecasts when there are a large number of forecasts that might enter the combination. The algorithm excludes a forecast from the combination if it is encompassed by another forecast. To assess the usefulness of this approach, an extensive empirical analysis is undertaken using a US macroeconomic dataset. The results are encouraging; the algorithm forecasts outperform benchmark model forecasts, in a mean square error (MSE) sense, in a majority of cases. The paper also compares the empirical performance of different approaches to forecast combination, and provides a rule‐of‐thumb cut‐off point for the thick‐modeling approach. Copyright © 2009 John Wiley & Sons, Ltd.     

Evaluating forecasts: a look at aggregate bias and accuracy measures

In this paper an investigation is made of the properties and use of two aggregate measures of forecast bias and accuracy. These are metrics used in business to calculate aggregate forecasting performance for a family (group) of products. We find that the aggregate measures are not particularly informative if some of the one‐step‐ahead forecasts are biased. This is likely to be the case in practice if frequently employed forecasting methods are used to generate a large number of individual forecasts. In the paper, examples are constructed to illustrate some potential problems in the use of the metrics. We propose a simple graphical display of forecast bias and accuracy to supplement the information yielded by the accuracy measures. This support includes relevant boxplots of measures of individual forecasting success. This tool is simple but helpful as the graphic display has the potential to indicate forecast deterioration that can be masked by one or both of the aggregate metrics. The procedures are illustrated with data representing sales of food items. Copyright © 2005 John Wiley & Sons, Ltd.     

Political manoeuvrings as sources of measurement errors in forecasts

We test the extent to which political manoeuvrings can be the sources of measurement errors in forecasts. Our objective is to examine the forecast error based on a simple model in which we attempt to explain deviations between the March budget forecast and the November forecast, and deviations between the outcome and the March budget forecast in the UK. The analysis is based on forecasts made by the general government. We use the forecasts of the variables as alternatives to the outcomes. We also test for political spins in the GDP forecast updates and the GDP forecast errors. We find evidence of partisan and electoral effects in forecast updates and forecast errors. Copyright © 2005 John Wiley & Sons, Ltd.     

Local to unity,long‐horizon forecasting thresholds for model selection in the AR(1)

This article introduces a novel framework for analysing long‐horizon forecasting of the near non‐stationary AR(1) model. Using the local to unity specification of the autoregressive parameter, I derive the asymptotic distributions of long‐horizon forecast errors both for the unrestricted AR(1), estimated using an ordinary least squares (OLS) regression, and for the random walk (RW). I then identify functions, relating local to unity ‘drift’ to forecast horizon, such that OLS and RW forecasts share the same expected square error. OLS forecasts are preferred on one side of these ‘forecasting thresholds’, while RW forecasts are preferred on the other. In addition to explaining the relative performance of forecasts from these two models, these thresholds prove useful in developing model selection criteria that help a forecaster reduce error. Copyright © 2004 John Wiley & Sons, Ltd.     

Combining multivariate volatility forecasts using weighted losses

The ability to improve out-of-sample forecasting performance by combining forecasts is well established in the literature. This paper advances this literature in the area of multivariate volatility forecasts by developing two combination weighting schemes that exploit volatility persistence to emphasise certain losses within the combination estimation period. A comprehensive empirical analysis of the out-of-sample forecast performance across varying dimensions, loss functions, sub-samples and forecast horizons show that new approaches significantly outperform their counterparts in terms of statistical accuracy. Within the financial applications considered, significant benefits from combination forecasts relative to the individual candidate models are observed. Although the more sophisticated combination approaches consistently rank higher relative to the equally weighted approach, their performance is statistically indistinguishable given the relatively low power of these loss functions. Finally, within the applications, further analysis highlights how combination forecasts dramatically reduce the variability in the parameter of interest, namely the portfolio weight or beta.     

Short-term forecasting of the US unemployment rate

This paper aims to assess whether Google search data are useful when predicting the US unemployment rate among other more traditional predictor variables. A weekly Google index is derived from the keyword “unemployment” and is used in diffusion index variants along with the weekly number of initial claims and monthly estimated latent factors. The unemployment rate forecasts are generated using MIDAS regression models that take into account the actual frequencies of the predictor variables. The forecasts are made in real time, and the forecasts of the best forecasting models exceed, for the most part, the root mean squared forecast error of two benchmarks. However, as the forecasting horizon increases, the forecasting performance of the best diffusion index variants decreases over time, which suggests that the forecasting methods proposed in this paper are most useful in the short term.     

Forecast Combination for Euro Area Inflation: A Cure in Times of Crisis?

The period of extraordinary volatility in euro area headline inflation starting in 2007 raised the question whether forecast combination methods can be used to hedge against bad forecast performance of single models during such periods and provide more robust forecasts. We investigate this issue for forecasts from a range of short‐term forecasting models. Our analysis shows that there is considerable variation of the relative performance of the different models over time. To take that into account we suggest employing performance‐based forecast combination methods—in particular, one with more weight on the recent forecast performance. We compare such an approach with equal forecast combination that has been found to outperform more sophisticated forecast combination methods in the past, and investigate whether it can improve forecast accuracy over the single best model. The time‐varying weights assign weights to the economic interpretations of the forecast stemming from different models. We also include a number of benchmark models in our analysis. The combination methods are evaluated for HICP headline inflation and HICP excluding food and energy. We investigate how forecast accuracy of the combination methods differs between pre‐crisis times, the period after the global financial crisis and the full evaluation period, including the global financial crisis with its extraordinary volatility in inflation. Overall, we find that forecast combination helps hedge against bad forecast performance and that performance‐based weighting outperforms simple averaging. Copyright © 2017 John Wiley & Sons, Ltd.     

Are Analysts' Loss Functions Asymmetric?

Despite displaying a statistically significant optimism bias, analysts' earnings forecasts are an important input to investors’ valuation models. Understanding the possible reasons for any bias is important if information is to be extracted from earnings forecasts and used optimally by investors. Extant research into the shape of analysts' loss functions explains optimism bias as resulting from analysts minimizing the mean absolute forecast error under symmetric, linear loss functions. When the distribution of earnings outcomes is skewed, optimalforecasts can appear biased. In contrast, research into analysts' economic incentives suggests that positive and negative earnings forecast errors made by analysts are not penalized or rewarded symmetrically, suggesting that asymmetric loss functions are an appropriate characterization. To reconcile these findings, we exploit results from economic theory relating to the Linex loss function to discriminate between the symmetric linear loss and the asymmetric loss explanations of analyst forecast bias. Under asymmetric loss functions optimal forecasts will appear biased even if earnings outcomes are symmetric. Our empirical results support the asymmetric loss function explanation. Further analysis also reveals that forecast bias varies systematically across firm characteristics that capture systematic variation in the earnings forecast error distribution. Copyright © 2011 John Wiley & Sons, Ltd.     

The Role of Financial Variables in predicting economic activity

Previous research found that the US business cycle leads the European one by a few quarters, and can therefore be useful in predicting euro area gross domestic product (GDP). In this paper we investigate whether additional predictive power can be gained by adding selected financial variables belonging to either the USA or the euro area. We use vector autoregressions (VARs) that include the US and euro area GDPs as well as growth in the Rest of the World and selected combinations of financial variables. Out‐of‐sample root mean square forecast errors (RMSEs) evidence that adding financial variables produces a slightly smaller error in forecasting US economic activity. This weak macro‐financial linkage is even weaker in the euro area, where financial indicators do not improve short‐ and medium‐term GDP forecasts even when their timely availability relative to GDP is exploited. It can be conjectured that neither US nor European financial variables help predict euro area GDP as the US GDP has already embodied this information. However, we show that the finding that financial variables have no predictive power for future activity in the euro area relates to the unconditional nature of the RMSE metric. When forecasting ability is assessed as if in real time (i.e. conditionally on the information available at the time when forecasts are made), we find that models using financial variables would have been preferred in several episodes and in particular between 1999 and 2002. Copyright 2011 John Wiley & Sons, Ltd.     

A comparative evaluation of objective and subjective weather forecasts in the united states

This paper is concerned primarily with the evaluation and comparison of objective and subjective weather forecasts. Operational forecasts of three weather elements are considered: (1) probability forecasts of precipitation occurrence, (2) categorical (i.e. non-probabilistic) forecasts of maximum and minimum temperatures and (3) categorical forecasts of cloud amount. The objective forecasts are prepared by numerical-statistical procedures, whereas the subjective forecasts are based on the judgements of individual forecasters. In formulating the latter, the forecasters consult information from a variety of sources, including the objective forecasts themselves. The precipitation probability forecasts are found to be both reliable and skilful, and evaluation of the temperature/cloud amount forecasts reveals that they are quite accurate/skilful. Comparison of the objective and subjective forecasts of precipitation occurrence indicates that the latter are generally more skilful than the former for shorter lead times (e.g. 12–24 hours), whereas the two types of forecasts are of approximately equal skill for longer lead times (e.g. 36–48 hours). Similar results are obtained for the maximum and minimum temperature forecasts. Objective cloud amount forecasts are more skilful than subjective cloud amount forecasts for all lead times. Examination of trends in performance over the last decade reveals that both types of forecasts for all three elements increased in skill (or accuracy) over the period, with improvements in objective forecasts equalling or exceeding improvements in subjective forecasts. The role and impact of the objective forecasts in the subjective weather forecasting process are discussed in some detail. The need to conduct controlled experiments and other studies of this process, with particular reference to the assimilation of information from different sources, is emphasized. Important characteristics of the forecasting system in meteorology are identified, and they are used to describe similarities and differences between weather forecasting and forecasting in other fields. Acquisition of some of these characteristics may be beneficial to other forecasting systems.     

Why do regime‐switching models forecast so badly?

Most non‐linear techniques give good in‐sample fits to exchange rate data but are usually outperformed by random walks or random walks with drift when used for out‐of‐sample forecasting. In the case of regime‐switching models it is possible to understand why forecasts based on the true model can have higher mean squared error than those of a random walk or random walk with drift. In this paper we provide some analytical results for the case of a simple switching model, the segmented trend model. It requires only a small misclassification, when forecasting which regime the world will be in, to lose any advantage from knowing the correct model specification. To illustrate this we discuss some results for the DM/dollar exchange rate. We conjecture that the forecasting result is more general and describes limitations to the use of switching models for forecasting. This result has two implications. First, it questions the leading role of the random walk hypothesis for the spot exchange rate. Second, it suggests that the mean square error is not an appropriate way to evaluate forecast performance for non‐linear models. Copyright © 1999 John Wiley & Sons, Ltd.     

Combination forecasts of output growth in a seven‐country data set

This paper uses forecast combination methods to forecast output growth in a seven‐country quarterly economic data set covering 1959–1999, with up to 73 predictors per country. Although the forecasts based on individual predictors are unstable over time and across countries, and on average perform worse than an autoregressive benchmark, the combination forecasts often improve upon autoregressive forecasts. Despite the unstable performance of the constituent forecasts, the most successful combination forecasts, like the mean, are the least sensitive to the recent performance of the individual forecasts. While consistent with other evidence on the success of simple combination forecasts, this finding is difficult to explain using the theory of combination forecasting in a stationary environment. Copyright © 2004 John Wiley & Sons, Ltd.     

Forecasting contemporaneous aggregates and the combination of forecasts: The case of the U.K. monetary aggregates

A number of papers in recent years have investigated the problems of forecasting contemporaneously aggregated time series and of combining alternative forecasts of a time series. This paper considers the integration of both approaches within the example of assessing the forecasting performance of models for two of the U.K. monetary aggregates, £M3 and MO. It is found that forecasts from a time series model for aggregate £M3 are superior to aggregated forecasts from individual models fitted to either the components or counterparts of £M3 and that an even better forecast is obtained by forming a linear combination of the three alternatives. For MO, however, aggregated forecasts from its components prove superior to either the forecast from the aggregate itself or from a linear combination of the two.     

Directed graphs,information structure and forecast combinations: an empirical examination of US unemployment rates

Previous studies show that it is not always optimal to combine forecasts of alternative models. In this paper, we propose to use the recent advances in modeling directed acyclic graphs to study the issue of forecast combinations. In forecasting US unemployment rates, we demonstrate that the proposed procedure can be a useful tool for comparing information in rival forecasts and guiding the combination of individual forecasts. Copyright © 2009 John Wiley & Sons, Ltd     

Revisions in analysts' earnings forecasts: evidence of non‐linear adaptive expectations

Prior studies use a linear adaptive expectations model to describe how analysts revise their forecasts of future earnings in response to current forecast errors. However, research shows that extreme forecast errors are less likely than small forecast errors to persist in future years. If analysts recognize this property, their marginal forecast revisions should decrease with the forecast error's magnitude. Therefore, a linear model is likely to be unsatisfactory at describing analysts' forecast revisions. We find that a non‐linear model better describes the relation between analysts' forecast revisions and their forecast errors, and provides a richer theoretical framework for explaining analysts' forecasting behaviour. Our results are consistent with analysts' recognizing the permanent and temporary nature of forecast errors of differing magnitudes. Copyright © 2000 John Wiley & Sons, Ltd.     

Forecasting the duration of short‐term deflation episodes

The paper proposes a simulation‐based approach to multistep probabilistic forecasting, applied for predicting the probability and duration of negative inflation. The essence of this approach is in counting runs simulated from a multivariate distribution representing the probabilistic forecasts, which enters the negative inflation regime. The marginal distributions of forecasts are estimated using the series of past forecast errors, and the joint distribution is obtained by a multivariate copula approach. This technique is applied for estimating the probability of negative inflation in China and its expected duration, with the marginal distributions computed by fitting weighted skew‐normal and two‐piece normal distributions to autoregressive moving average ex post forecast errors and using the multivariate Student t copula.     

Can consumer sentiment and its components forecast Australian GDP and consumption?

This paper examines whether the disaggregation of consumer sentiment data into its sub‐components improves the real‐time capacity to forecast GDP and consumption. A Bayesian error correction approach augmented with the consumer sentiment index and permutations of the consumer sentiment sub‐indices is used to evaluate forecasting power. The forecasts are benchmarked against both composite forecasts and forecasts from standard error correction models. Using Australian data, we find that consumer sentiment data increase the accuracy of GDP and consumption forecasts, with certain components of consumer sentiment consistently providing better forecasts than aggregate consumer sentiment data. Copyright © 2009 John Wiley & Sons, Ltd.     

Real‐time inflation forecast combination for time‐varying coefficient models

We use real‐time macroeconomic variables and combination forecasts with both time‐varying weights and equal weights to forecast inflation in the USA. The combination forecasts compare three sets of commonly used time‐varying coefficient autoregressive models: Gaussian distributed errors, errors with stochastic volatility, and errors with moving average stochastic volatility. Both point forecasts and density forecasts suggest that models combined by equal weights do not produce worse forecasts than those with time‐varying weights. We also find that variable selection, the allowance of time‐varying lag length choice, and the stochastic volatility specification significantly improve forecast performance over standard benchmarks. Finally, when compared with the Survey of Professional Forecasters, the results of the best combination model are found to be highly competitive during the 2007/08 financial crisis.     

Comparing the DSGE model with the factor model: an out‐of‐sample forecasting experiment

In this paper, we put dynamic stochastic general equilibrium DSGE forecasts in competition with factor forecasts. We focus on these two models since they represent nicely the two opposing forecasting philosophies. The DSGE model on the one hand has a strong theoretical economic background; the factor model on the other hand is mainly data‐driven. We show that incorporating a large information set using factor analysis can indeed improve the short‐horizon predictive ability, as claimed by many researchers. The micro‐founded DSGE model can provide reasonable forecasts for US inflation, especially with growing forecast horizons. To a certain extent, our results are consistent with the prevailing view that simple time series models should be used in short‐horizon forecasting and structural models should be used in long‐horizon forecasting. Our paper compares both state‐of‐the‐art data‐driven and theory‐based modelling in a rigorous manner. Copyright © 2008 John Wiley & Sons, Ltd.