Evaluating the role judgment plays in forecast accuracy

The majority of model-based forecasting efforts today rely on relatively simple techniques of estimation and the subjective adjustment of the model's results to produce forecasts. Published forecasts reflect to a great extent the judgment of the forecaster rather than what the model by itself has to say about the future. This paper examines the role judgment plays in the process of producing a macroeconometric forecast. The debate over the use of adjustment constants to alter the statistical results of a model is outlined and an empirical analysis of forecasts generated by the Michigan Quarterly Econometric Model of the US economy is presented using a unique data set which isolates the role of judgment in the forecasting process.     

The importance of the forecast

Careful forecasts, as accurate as possible, are central to the successful implementation of policy. There are fundamental reasons why policy makers cannot ‘play by ear’, adjusting policy quickly to each unexpected deviation in economic outcomes. Specific incidents are described where economic policy went awry because of faulty forecasts. The policy process is described in detail to show precisely where the forecast enters. Forecasting as a validation tool for establishing credibility in policy formation is analysed and discussed. Some estimated measure of forecast accuracy is presented, together with commentary on the necessary degrees of precision for successful implementation of policy.     

Algorithms for explaining forecast revisions

Forecasts are routinely revised, and these revisions are often the subject of informal analysis and discussion. This paper argues (1) that forecast revisions are analyzed because they help forecasters and forecast users to evaluate forecasts and forecasting procedures and (2) that these analyses can be sharpened by using the forecasting model to systematically express its forecast revision as the sum of components identified with specific subsets of new information, such as data revisions and forecast errors. An algorithm for this purpose is explained and illustrated.     

Judgemental revision of sales forecasts: Effectiveness of forecast selection

When managers make revisions to sales forecasts initially generated by a rational quantitative model it is important that the particular forecasts selected for adjustment are those which would benefit most from the adjustment process (i.e. realize high errors). This study reports an empirical investigation on this issue, spanning six quarterly forecasting periods and incorporating forecasting data on over 850 products. The results show that the errors of the forecasts chosen for revision are, in general, higher than those which were not chosen. In addition, it is shown that managesrs tend to revise forecasts which are initially low, hence possibily introducing some degree of bias into the overall forecasts.     

The use of non-time series information in sales forecasting: A case study

The contribution of product and industry knowledge to the accuracy of sales forecasting was investigated by examining the company forecasts of a leading manufacturer and marketer of consumable products. The company forecasts of 18 products produced by a meeting of marketing, sales, and production personnel were compared with those generated by the same company personnel when denied specific product knowledge and with the forecasts of selected judgemental and statistical time series methods. Results indicated that product knowledge contributed significantly to forecast accuracy and that the forecast accuracy of company personnel who possessed industry forecasting knowledge (but not product knowledge) was not significantly different from the time series based methods. Furthermore, the company forecasts were more accurate than averages of the judgemental and statistical time series forecasts. These results point to the importance of specific product information to forecast accuracy and accordingly call into question the continuing strong emphasis on improving extrapolation techniques without consideration of the inclusion of non-time series knowledge.     

Economic and statistical measures of forecast accuracy

This paper argues in favour of a closer link between the decision and the forecast evaluation problems. Although the idea of using decision theory for forecast evaluation appears early in the dynamic stochastic programming literature, and has continued to be used with meteorological forecasts, it is hardly mentioned in standard academic texts on economic forecasting. Some of the main issues involved are illustrated in the context of a two‐state, two‐action decision problem as well as in a more general setting. Relationships between statistical and economic methods of forecast evaluation are discussed and links between the Kuipers score used as a measure of forecast accuracy in the meteorology literature and the market timing tests used in finance are established. An empirical application to the problem of stock market predictability is also provided, and the conditions under which such predictability could be explained in the presence of transaction costs are discussed. Copyright © 2000 John Wiley & Sons, Ltd.     

Reducing uncertainty in short-term projections: Linkage of monthly and quarterly models

This paper shows how monthly data and forecasts can be used in a systematic way to improve the predictive accuracy of a quarterly macroeconometric model. The problem is formulated as a model pooling procedure (equivalent to non-recursive Kalman filtering) where a baseline quarterly model forecast is modified through ‘add-factors’ or ‘constant adjustments’. The procedure ‘automatically’ constructs these adjustments in a covariance-minimizing fashion to reflect the revised expectation of the quarterly model's forecast errors, conditional on the monthly information set. Results obtained using Federal Reserve Board models indicate the potential for significant reduction in forecast error variance through application of these procedures.     

Integrating time-series and end-use methods to forecast electricity sales

Two types of forecasting methods have been receiving increasing attention by electric utility forecasters. The first type, called end-use forecasting, is recognized as an approach which is well suited for forecasting during periods characterized by technological change. The method is straightforward. The stock levels of energy-consuming equipment are forecast, as well as the energy consumption characteristics of the equipment. The final forecast is the product of the stock and usage characteristics. This approach is well suited to forecasting long time periods when technological change, equipment depletion and replacement, and other structural changes are evident. For time periods of shorter duration, these factors are static and variations are more likely to result from shocks to the environment. The shocks influence the usage of the equipment. A second forecasting approach using time-series analysis has been demonstrated to be superior for these applications. This paper discusses the integration of the two methods into a unified system. The result is a time-series model whose parameter effects become dynamic in character. An example of the models being used at the Georgia Power Company is presented. It is demonstrated that a time-series model which incorporates end-use stock and usage information is superior—even in short-term forecasting situations—to a similar time-series model which excludes the information.     

Comments on testing for forecast and specification optimality using linear composites

This paper extends the method of using linear composites of forecasts for testing the efficiency of one forecast compared to a finite collection of other forecasts. It also gives necessary and sufficient conditions for forecast optimality in the mean square error sense. Information sets are found to be unnecessary for forecast optimality. Finally, the paper shows that a consistent test of forecast optimality cannot in general be obtained using linear composites. A similar conclusion applies to tests of specification optimality based on linear composites.     

Cointegration and forecast evaluation: Some lessons from National Institute Forecasts

The recent experience of macroeconomic forecasting in the United Kingdom has prompted renewed interest in the evaluation of economic forecasts. This paper uses cointegration tests to investigate what can be learnt from the forecasts produced by the National Institute of Economic and Social Research (NIESR) over the last two decades. Whilst the forecasts and outturns are found to be cointegrated, there remains evidence of systematic relationships between a number of forecast errors. Our results also fail to reject non-cointegration between different vintages of data, suggesting that considerable care should be exercised in both the choice of realisation data used and in the means by which efficiency is tested.     

Forecast evaluation and improvement using theil's decomposition

Forecasters are concerned with the accuracy of a forecast and whether the forecast can be modified to yield an improved performance. Theil has proposed statistics to measure forecast performance and to identify components of forecast error. However, the most commonly used of Theil's statistics have been shown to have serious shortcomings. This paper discusses Theil's decomposition of forecast error into bias, regression and disturbance proportions. Examples using price expectations and new housing starts data are given to show how decomposition suggests a linear correction procedure that may improve forecast accuracy.     

Awareness and use of forecasting techniques in british industry

The paper summarizes results of a mail survey of the use of formal forecasting techniques in British manufacturing companies. It appraises the state of awareness of particular techniques and the extent to which they are used in various functional applications. The extent to which the forecasts generated by the techniques influence company action is assessed; and the reasons for the non-use of particular techniques examined. The paper concludes that although an increasing number of companies appreciate the importance of forecasting, the methods used are predominantly naïve and few companies are taking steps to improve the situation through using alternative techniques or through computerizing established techniques.     

Familiarity,application, and performance of sales forecasting techniques

This article presents the results of a survey to determine the degree of familiarity and usage, accuracy obtained, and evaluation of different forecasting techniques. It was found that regression analysis, subjective techniques, exponential smoothing, and moving average were well known and used for specific situations. Accuracy was relatively high for aggregate short range forecasts, but decreased for longer range and product level forecasts.     

The uses and abuses of ‘consensus’ forecasts

Clemen's (1989) review of the forecast-combining literature amply illustrates both the interest in and the importance of this subject. This article stresses the tautological properties of various consensus measures that assure their success relative to most individual forecasts. It confirms the finding of earlier studies that for each specific macroeconomic variable roughly one-third of individual forecasters are more accurate than a consensus. However, each individual does relatively poorly for some variable while the consensus, in contrast, necessarily never fails relative to most individuals. These results, like most previous studies, describe consensus measures that are synthetic constructs derived from a pre-existing set of individual forecasts. Strictly speaking, this contemporaneous consensus is not available to individual forecasters when their forecasts are made. A prior consensus measure, which is in their information sets, was relatively much less accurate than the contemporaneous measure. Nevertheless, a small subset of individual forecasters were generally inferior to the known, prior consensus forecast.     

The impact on the performance of growth curve models of changes in parameter re-estimation period and other factors

A simulation model of a real electricity supply undertaking was used to provide a financial performance measure for growth curve forecasting models. The impact on financial performance was determined when changes were made in (1) the method of estimating the model parameters, (2) the period between re-estimations, (3) the growth curve fitted and (4) the amount of smoothing of the demand time-series. The response to variation of the parameter review period was found to behave surprisingly, in that it exhibited different signs for two different estimation methods. Changes in re-estimation period explained somewhat more of the variation in performance than did a change in growth curve. Correcting the demand series for conditions which were known to be abnormal improved performance.     

Forecasting us home sales using bvar models and survey data on households' buying attitudes for homes

This study uses Bayesian vector autoregressive models to examine the usefulness of survey data on households' buying attitudes for homes in predicting sales of homes. We find a negligible deterioration in the accuracy of forecasts of home sales when buying attitudes are dropped from a model that includes the price of homes, the mortgage rate, real personal disposable income, and die unemployment rate. This suggests that buying attitudes do not add much to the information contained in these variables. We also find that forecasts from the model that includes both buying attitudes and the aforementioned variables are similar to those generated from a model that excludes the survey data but contains the other variables. Additionally, the variance decompositions suggest that the gain from including the survey data in the model that already contains other economic variables is small.     

Some practical aspects of forecasting in organizations

This paper presents the writer's experience, over a period of 25 years, in analysing organizational systems and, in particular, concentrates on the overall forecasting activity. The paper first looks at the relationship between forecasting and decision taking–with emphasis on the fact that forecasting is a means to aid decision taking and not an end in itself. It states that there are many types of forecasting problems, each requiring different methods of treatment. The paper then discusses attitudes which are emerging about the relative advantages of different forecasting techniques. It suggests a model building process which requires‘experience’and‘craftsmanship’, extensive practical application, frequent interaction between theory and practice and a methodology that eventually leads to models that contain no detectable inadequacies. Furthermore, it argues that although models which forecast a time series from its past history have a very important role to play, for effective policy making it is necessary to augment the model by introducing policy variables, again in a systematic not an ‘ad hoc’ manner. Finally, the paper discusses how forecasting systems can be introduced into the management process in the first place and how they should be monitored and updated when found wanting.     

The impact of forecasting methodology on the accuracy of national population forecasts: evidence from the Netherlands and Czechoslovakia

"This study considers the accuracy of national population forecasts of the Netherlands and the Czechoslovak Socialist Republic.... We look at the demographic components employed in each forecast, the procedure to extrapolate fertility and the level at which assumptions for each component are formulated. Errors in total population size, fertility, mortality and foreign migration, and age structure are considered. We discuss trends in errors and methodology since 1950 and compare the situations in the two countries. The findings suggest that methodology has only a very limited impact on the accuracy of national population forecasts."     

The accuracy of individual and group forecasts from business outlook surveys

This paper reports on the accuracy of quarterly multiperiod predictions of inflation, real growth, unemployment and percentage changes in nominal GNP and two of its more volatile components. The survey data are highly differentiated; they cover 79 professional forecasters (mostly economists, analysts and corporate executives). Combining corresponding predictions from different sources can result in significant gains; thus the group mean forecasts are on the average over time more accurate than most of the corresponding sets of individual forecasts. But there is also a moderate degree of consistency in the relative performance of a sufficient number of the survey members, as evidenced in positive rank correlations among ratios of the individual to group root mean square errors.