Can out‐of‐sample forecast comparisons help prevent overfitting?

This paper shows that out‐of‐sample forecast comparisons can help prevent data mining‐induced overfitting. The basic results are drawn from simulations of a simple Monte Carlo design and a real data‐based design similar to those used in some previous studies. In each simulation, a general‐to‐specific procedure is used to arrive at a model. If the selected specification includes any of the candidate explanatory variables, forecasts from the model are compared to forecasts from a benchmark model that is nested within the selected model. In particular, the competing forecasts are tested for equal MSE and encompassing. The simulations indicate most of the post‐sample tests are roughly correctly sized. Moreover, the tests have relatively good power, although some are consistently more powerful than others. The paper concludes with an application, modelling quarterly US inflation. Copyright © 2004 John Wiley & Sons, Ltd.     

Testing Interval Forecasts: A GMM‐Based Approach

This paper proposes a new evaluation framework for interval forecasts. Our model‐free test can be used to evaluate interval forecasts and high‐density regions, potentially discontinuous and/or asymmetric. Using a simple J‐statistic, based on the moments defined by the orthonormal polynomials associated with the binomial distribution, this new approach presents many advantages. First, its implementation is extremely easy. Second, it allows for a separate test for unconditional coverage, independence and conditional coverage hypotheses. Third, Monte Carlo simulations show that for realistic sample sizes our GMM test has good small‐sample properties. These results are corroborated by an empirical application on SP500 and Nikkei stock market indexes. It confirms that using this GMM test leads to major consequences for the ex post evaluation of interval forecasts produced by linear versus nonlinear models. Copyright © 2011 John Wiley & Sons, Ltd.     

Long‐memory forecasting of US monetary indices

Several studies have tested for long‐range dependence in macroeconomic and financial time series but very few have assessed the usefulness of long‐memory models as forecast‐generating mechanisms. This study tests for fractional differencing in the US monetary indices (simple sum and divisia) and compares the out‐of‐sample fractional forecasts to benchmark forecasts. The long‐memory parameter is estimated using Robinson's Gaussian semi‐parametric and multivariate log‐periodogram methods. The evidence amply suggests that the monetary series possess a fractional order between one and two. Fractional out‐of‐sample forecasts are consistently more accurate (with the exception of the M3 series) than benchmark autoregressive forecasts but the forecasting gains are not generally statistically significant. In terms of forecast encompassing, the fractional model encompasses the autoregressive model for the divisia series but neither model encompasses the other for the simple sum series. Copyright © 2006 John Wiley & Sons, Ltd.     

Forecasting nonlinear time series with feed‐forward neural networks: a case study of Canadian lynx data

The forecasting capabilities of feed‐forward neural network (FFNN) models are compared to those of other competing time series models by carrying out forecasting experiments. As demonstrated by the detailed forecasting results for the Canadian lynx data set, FFNN models perform very well, especially when the series contains nonlinear and non‐Gaussian characteristics. To compare the forecasting accuracy of a FFNN model with an alternative model, Pitman's test is employed to ascertain if one model forecasts significantly better than another when generating one‐step‐ahead forecasts. Moreover, the residual‐fit spread plot is utilized in a novel fashion in this paper to compare visually out‐of‐sample forecasts of two alternative forecasting models. Finally, forecasting findings on the lynx data are used to explain under what conditions one would expect FFNN models to furnish reliable and accurate forecasts. Copyright © 2005 John Wiley & Sons, Ltd.     

Bias in the estimation of non‐linear transformations of the integrated variance of returns

Volatility models such as GARCH, although misspecified with respect to the data‐generating process, may well generate volatility forecasts that are unconditionally unbiased. In other words, they generate variance forecasts that, on average, are equal to the integrated variance. However, many applications in finance require a measure of return volatility that is a non‐linear function of the variance of returns, rather than of the variance itself. Even if a volatility model generates forecasts of the integrated variance that are unbiased, non‐linear transformations of these forecasts will be biased estimators of the same non‐linear transformations of the integrated variance because of Jensen's inequality. In this paper, we derive an analytical approximation for the unconditional bias of estimators of non‐linear transformations of the integrated variance. This bias is a function of the volatility of the forecast variance and the volatility of the integrated variance, and depends on the concavity of the non‐linear transformation. In order to estimate the volatility of the unobserved integrated variance, we employ recent results from the realized volatility literature. As an illustration, we estimate the unconditional bias for both in‐sample and out‐of‐sample forecasts of three non‐linear transformations of the integrated standard deviation of returns for three exchange rate return series, where a GARCH(1, 1) model is used to forecast the integrated variance. Our estimation results suggest that, in practice, the bias can be substantial. Copyright © 2006 John Wiley & Sons, Ltd.     

Why do regime‐switching models forecast so badly?

Most non‐linear techniques give good in‐sample fits to exchange rate data but are usually outperformed by random walks or random walks with drift when used for out‐of‐sample forecasting. In the case of regime‐switching models it is possible to understand why forecasts based on the true model can have higher mean squared error than those of a random walk or random walk with drift. In this paper we provide some analytical results for the case of a simple switching model, the segmented trend model. It requires only a small misclassification, when forecasting which regime the world will be in, to lose any advantage from knowing the correct model specification. To illustrate this we discuss some results for the DM/dollar exchange rate. We conjecture that the forecasting result is more general and describes limitations to the use of switching models for forecasting. This result has two implications. First, it questions the leading role of the random walk hypothesis for the spot exchange rate. Second, it suggests that the mean square error is not an appropriate way to evaluate forecast performance for non‐linear models. Copyright © 1999 John Wiley & Sons, Ltd.     

Forecasting with panel data

This paper gives a brief survey of forecasting with panel data. It begins with a simple error component regression model and surveys the best linear unbiased prediction under various assumptions of the disturbance term. This includes various ARMA models as well as spatial autoregressive models. The paper also surveys how these forecasts have been used in panel data applications, running horse races between heterogeneous and homogeneous panel data models using out‐of‐sample forecasts. Copyright © 2008 John Wiley & Sons, Ltd.     

Do experts' adjustments on model‐based SKU‐level forecasts improve forecast quality?

Model‐based SKU‐level forecasts are often adjusted by experts. In this paper we propose a statistical methodology to test whether these expert forecasts improve on model forecasts. Application of the methodology to a very large database concerning experts in 35 countries who adjust SKU‐level forecasts for pharmaceutical products in seven distinct categories leads to the general conclusion that expert forecasts are equally good at best, but are more often worse than model‐based forecasts. We explore whether this is due to experts putting too much weight on their contribution, and this indeed turns out to be the case. Copyright © 2009 John Wiley & Sons, Ltd.     

Predictable Return Distributions

Using quantile regression this paper explores the predictability of the stock and bond return distributions as a function of economic state variables. The use of quantile regression allows us to examine specific parts of the return distribution such as the tails and the center, and for a sufficiently fine grid of quantiles we can trace out the entire distribution. A univariate quantile regression model is used to examine the marginal stock and bond return distributions, while a multivariate model is used to capture their joint distribution. An empirical analysis on US data shows that economic state variables predict the stock and bond return distributions in quite different ways in terms of, for example, location shifts, volatility and skewness. Comparing the different economic state variables in terms of their out‐of‐sample forecasting performance, the empirical analysis also shows that the relative accuracy of the state variables varies across the return distribution. Density forecasts based on an assumed normal distribution with forecasted mean and variance is compared to forecasts based on quantile estimates and, in general, the latter yields the best performance. Copyright © 2015 John Wiley & Sons, Ltd.     

Forecasting US inflation by Bayesian model averaging

Recent empirical work has considered the prediction of inflation by combining the information in a large number of time series. One such method that has been found to give consistently good results consists of simple equal‐weighted averaging of the forecasts from a large number of different models, each of which is a linear regression relating inflation to a single predictor and a lagged dependent variable. In this paper, I consider using Bayesian model averaging for pseudo out‐of‐sample prediction of US inflation, and find that it generally gives more accurate forecasts than simple equal‐weighted averaging. This superior performance is consistent across subsamples and a number of inflation measures. Copyright © 2008 John Wiley & Sons, Ltd.     

Point forecasting of intraday volume using Bayesian autoregressive conditional volume models

In this paper, we apply Bayesian inference to model and forecast intraday trading volume, using autoregressive conditional volume (ACV) models, and we evaluate the quality of volume point forecasts. In the empirical application, we focus on the analysis of both in‐ and out‐of‐sample performance of Bayesian ACV models estimated for 2‐minute trading volume data for stocks quoted on the Warsaw Stock Exchange in Poland. We calculate two types of point forecasts, using either expected values or medians of predictive distributions. We conclude that, in general, all considered models generate significantly biased forecasts. We also observe that the considered models significantly outperform such benchmarks as the naïve or rolling means forecasts. Moreover, in terms of root mean squared forecast errors, point predictions obtained within the ACV model with exponential distribution emerge superior compared to those calculated in structures with more general innovation distributions, although in many cases this characteristic turns out to be statistically insignificant. On the other hand, when comparing mean absolute forecast errors, the median forecasts obtained within the ACV models with Burr and generalized gamma distribution are found to be statistically better than other forecasts.     

Modeling Realized Volatility Dynamics with a Genetic Algorithm

The heterogeneous autoregressive model of realized volatility (HAR‐RV) is inspired by the heterogeneous market hypothesis and characterizes realized volatility dynamics through a linear function of lagged daily, weekly and monthly realized volatilities with a (1, 5, 22) lag structure. Considering that different markets can have different heterogeneous structures and a market's heterogeneous structure can vary over time, we build an adaptive heterogeneous autoregressive model of realized volatility (AHAR‐RV), whose lag structure is optimized with a genetic algorithm. Using nine common loss functions and the superior predictive ability test, we find that our AHAR‐RV model and its extensions provide significantly better out‐of‐sample volatility forecasts for the CSI 300 index than the corresponding HAR models. Furthermore, the AHAR‐RV model significantly outperforms all the other models under most loss functions. Besides, we confirm that Chinese stock markets' heterogeneous structure varies over time and the (1, 5, 22) lag structure is not the optimal choice. Copyright © 2016 John Wiley & Sons, Ltd.     

Time‐Varying Parameter Realized Volatility Models

In this paper, we introduce the functional coefficient to heterogeneous autoregressive realized volatility (HAR‐RV) models to make the parameters change over time. A nonparametric statistic is developed to perform a specification test. The simulation results show that our test displays reliable size and good power. Using the proposed test, we find a significant time variation property of coefficients to the HAR‐RV models. Time‐varying parameter (TVP) models can significantly outperform their constant‐coefficient counterparts for longer forecasting horizons. The predictive ability of TVP models can be improved by accounting for VIX information. Copyright © 2016 John Wiley & Sons, Ltd.     

Nelson–Siegel,Affine and Quadratic Yield Curve Specifications: Which One is Better at Forecasting?

In this paper we compare the in‐sample fit and out‐of‐sample forecasting performance of no‐arbitrage quadratic, essentially affine and dynamic Nelson–Siegel term structure models. In total, 11 model variants are evaluated, comprising five quadratic, four affine and two Nelson–Siegel models. Recursive re‐estimation and out‐of‐sample 1‐, 6‐ and 12‐month‐ahead forecasts are generated and evaluated using monthly US data for yields observed at maturities of 1, 6, 12, 24, 60 and 120 months. Our results indicate that quadratic models provide the best in‐sample fit, while the best out‐of‐sample performance is generated by three‐factor affine models and the dynamic Nelson–Siegel model variants. Statistical tests fail to identify one single best forecasting model class. Copyright © 2011 John Wiley & Sons, Ltd.     

Comparing the DSGE model with the factor model: an out‐of‐sample forecasting experiment

In this paper, we put dynamic stochastic general equilibrium DSGE forecasts in competition with factor forecasts. We focus on these two models since they represent nicely the two opposing forecasting philosophies. The DSGE model on the one hand has a strong theoretical economic background; the factor model on the other hand is mainly data‐driven. We show that incorporating a large information set using factor analysis can indeed improve the short‐horizon predictive ability, as claimed by many researchers. The micro‐founded DSGE model can provide reasonable forecasts for US inflation, especially with growing forecast horizons. To a certain extent, our results are consistent with the prevailing view that simple time series models should be used in short‐horizon forecasting and structural models should be used in long‐horizon forecasting. Our paper compares both state‐of‐the‐art data‐driven and theory‐based modelling in a rigorous manner. Copyright © 2008 John Wiley & Sons, Ltd.     

Forecasting an aggregate of cointegrated disaggregates

This study examines the problem of forecasting an aggregate of cointegrated disaggregates. It first establishes conditions under which forecasts of an aggregate variable obtained from a disaggregate VECM will be equal to those from an aggregate, univariate time series model, and develops a simple procedure for testing those conditions. The paper then uses Monte Carlo simulations to show, for a finite sample, that the proposed test has good size and power properties and that whether a model satisfies the aggregation conditions is closely related to out‐of‐sample forecast performance. The paper then shows that ignoring cointegration and specifying the disaggregate model as a VAR in differences can significantly affect analyses of aggregation, with the VAR‐based test for aggregation possibly leading to faulty inference and the differenced VAR forecasts potentially understating the benefits of disaggregate information. Finally, analysis of an empirical problem confirms the basic results. Copyright © 2000 John Wiley & Sons, Ltd.     

Forecasting Monetary Policy Decisions in Australia: A Forecast Combinations Approach

This paper applies a triple‐choice ordered probit model, corrected for nonstationarity to forecast monetary decisions of the Reserve Bank of Australia. The forecast models incorporate a mix of monthly and quarterly macroeconomic time series. Forecast combination is used as an alternative to one multivariate model to improve accuracy of out‐of‐sample forecasts. This accuracy is evaluated with scoring functions, which are also used to construct adaptive weights for combining probability forecasts. This paper finds that combined forecasts outperform multivariable models. These results are robust to different sample sizes and estimation windows. Copyright © 2011 John Wiley & Sons, Ltd.     

A high–low model of daily stock price ranges

We observe that daily highs and lows of stock prices do not diverge over time and, hence, adopt the cointegration concept and the related vector error correction model (VECM) to model the daily high, the daily low, and the associated daily range data. The in‐sample results attest to the importance of incorporating high–low interactions in modeling the range variable. In evaluating the out‐of‐sample forecast performance using both mean‐squared forecast error and direction of change criteria, it is found that the VECM‐based low and high forecasts offer some advantages over alternative forecasts. The VECM‐based range forecasts, on the other hand, do not always dominate—the forecast rankings depend on the choice of evaluation criterion and the variables being forecast. Copyright © 2008 John Wiley & Sons, Ltd.     

Is forecasting with large models informative? Assessing the role of judgement in macroeconomic forecasts

We evaluate residual projection strategies in the context of a large‐scale macro model of the euro area and smaller benchmark time‐series models. The exercises attempt to measure the accuracy of model‐based forecasts simulated both out‐of‐sample and in‐sample. Both exercises incorporate alternative residual‐projection methods, to assess the importance of unaccounted‐for breaks in forecast accuracy and off‐model judgement. Conclusions reached are that simple mechanical residual adjustments have a significant impact on forecasting accuracy irrespective of the model in use, likely due to the presence of breaks in trends in the data. The testing procedure and conclusions are applicable to a wide class of models and of general interest. Copyright © 2010 John Wiley & Sons, Ltd.     

A New‐Keynesian DSGE model for forecasting the South African economy

This paper develops a New‐Keynesian Dynamic Stochastic General Equilibrium (NKDSGE) model for forecasting the growth rate of output, inflation, and the nominal short‐term interest rate (91 days Treasury Bill rate) for the South African economy. The model is estimated via maximum likelihood technique for quarterly data over the period of 1970:1–2000:4. Based on a recursive estimation using the Kalman filter algorithm, out‐of‐sample forecasts from the NKDSGE model are compared with forecasts generated from the classical and Bayesian variants of vector autoregression (VAR) models for the period 2001:1–2006:4. The results indicate that in terms of out‐of‐sample forecasting, the NKDSGE model outperforms both the classical and Bayesian VARs for inflation, but not for output growth and nominal short‐term interest rate. However, differences in RMSEs are not significant across the models. Copyright © 2008 John Wiley & Sons, Ltd.