A comparative evaluation of objective and subjective weather forecasts in the united states

This paper is concerned primarily with the evaluation and comparison of objective and subjective weather forecasts. Operational forecasts of three weather elements are considered: (1) probability forecasts of precipitation occurrence, (2) categorical (i.e. non-probabilistic) forecasts of maximum and minimum temperatures and (3) categorical forecasts of cloud amount. The objective forecasts are prepared by numerical-statistical procedures, whereas the subjective forecasts are based on the judgements of individual forecasters. In formulating the latter, the forecasters consult information from a variety of sources, including the objective forecasts themselves. The precipitation probability forecasts are found to be both reliable and skilful, and evaluation of the temperature/cloud amount forecasts reveals that they are quite accurate/skilful. Comparison of the objective and subjective forecasts of precipitation occurrence indicates that the latter are generally more skilful than the former for shorter lead times (e.g. 12–24 hours), whereas the two types of forecasts are of approximately equal skill for longer lead times (e.g. 36–48 hours). Similar results are obtained for the maximum and minimum temperature forecasts. Objective cloud amount forecasts are more skilful than subjective cloud amount forecasts for all lead times. Examination of trends in performance over the last decade reveals that both types of forecasts for all three elements increased in skill (or accuracy) over the period, with improvements in objective forecasts equalling or exceeding improvements in subjective forecasts. The role and impact of the objective forecasts in the subjective weather forecasting process are discussed in some detail. The need to conduct controlled experiments and other studies of this process, with particular reference to the assimilation of information from different sources, is emphasized. Important characteristics of the forecasting system in meteorology are identified, and they are used to describe similarities and differences between weather forecasting and forecasting in other fields. Acquisition of some of these characteristics may be beneficial to other forecasting systems.     

Linear constraints and the efficiency of combined forecasts

This note extends some recent results, achieved by Clemen, on constraining the weights of a combined forecast. There is a great potential for improving the ordinary least squares forecast by imposing linear restrictions, and it will be shown how this potential can be exhausted by using an F-test. The corresponding decision procedure leads to a pre-test forecast with good statistical properties.     

Linear constraints and the efficiency of combined forecasts

Studies of combined forecasts have typically constrained the combining weights to sum to one and have not included a constant term in the combination. In a recent paper, Granger and Ramanathan (1984) have argued in favour of an unrestricted linear combination, including a constant term. This paper shows that for the purpose of prediction it may make sense to impose restrictions on the combining model because of potential increases in forecasting efficiency. Empirical results show that small gains in forecasting efficiency can be obtained by restricting the linear combination of GNP forecasts from four econometric models.     

Awareness and use of forecasting techniques in british industry

The paper summarizes results of a mail survey of the use of formal forecasting techniques in British manufacturing companies. It appraises the state of awareness of particular techniques and the extent to which they are used in various functional applications. The extent to which the forecasts generated by the techniques influence company action is assessed; and the reasons for the non-use of particular techniques examined. The paper concludes that although an increasing number of companies appreciate the importance of forecasting, the methods used are predominantly naïve and few companies are taking steps to improve the situation through using alternative techniques or through computerizing established techniques.     

System-based weights versus series-specific weights in the combination of forecasts

System-based combination weights for series r/step-length h incorporate relative accuracy information from other forecast step-lengths for r and from other series for step-length h. Such weights are examined utilizing the West and Fullerton (1996) data set-4275 ex ante employment forecasts from structural simultaneous equation econometric models for 19 metropolitan areas at 10 quarterly step-lengths and a parallel set of 4275 ARIMA forecasts. The system-based weights yielded combined forecasts of higher average accuracy and lower risk of large inaccuracy than seven alternative strategies: (1) averaging; (2) relative MSE weights; (3) outperformance (per cent best) weights; (4) Bates and Granger (1969) optimal weights with a convexity constraint imposed; (5) unconstrained optimal weights; (6) select a ‘best’ method (ex ante) by series and; (7) experiment in the Bischoff (1989) sense and select either method (2) or (6) based on the outcome of e experiment. Accuracy gains of the system-based combination were concentrated at step-lengths two to five. Although alternative (5) was generally outperformed, none of the six other alternatives was systematically most accurate when evaluated relative to each other. This contrasts with Bischoff's (1989) results that held promise for an empirically applicable guideline to determine whether or not to combine.     

Collinearity and the use of latent root regression for combining GNP forecasts

In combining economic forecasts a problem often faced is that the individual forecasts display some degree of dependence. We discuss latent root regression for combining collinear GNP forecasts. Our results indicate that latent root regression produces more efficient combining weight estimates (regression parameter estimates) than ordinary least squares estimation (OLS), although out-of-sample forecasting performance is comparable to OLS.     

Experience in combining subjective and quantitative forecasts of open market rates

This paper presents a method of combining subjective information from open-market operators with results from a time-series forecasting model. Empirical results of forecasts for interest rates of bank reserves are presented.     

Comparing ex-ante forecasts from a sem and var model: An application to the italian economy

The predictive performance of a large-scale structural econometric model (SEM) of the Italian economy the Prometeia model is compared in this paper with a vector autoregressive (VAR) model estimated for a selection of six main variables of interest. The paper concentrates on the quarterly ex-ante forecasts of GDP growth rate and the annual forecasts of GDP growth and inflation rate, over the period 1980-85. It concludes that no forecaster is systematically better than the other. In particular, the VAR model outperforms the SEM in short-run forecasts, suggesting that, for the latter, more careful attention should be addressed to questions of dynamic specification. On the other hand, for longer intervals, the SEM forecasts are more accurate than the VAR forecasts, in that they can benefit from the judgemental interventions of the model users and the model can pick up the non-linearities of the economy which cannot be captured by the VAR. Given the different kinds of information that can be extracted from the two approaches, it seems more reasonable to consider them as complementary rather than alternative tools for modelling and forecasting. Therefore, rather than attempting to establish the superiority of one type of model over the other, this kind of comparisons should be seen as a useful diagnostic tool for detecting types of model misspecification.     

Quality/value relationships for imperfect weather forecasts in a prototype multistage decision-making model

Some theoretical results concerning the nature of the relationship between the scientific quality and economic value of imperfect weather forecasts are obtained. A prototype multistage decision-making model is considered, involving only two possible actions and two possible states of weather. This particular form of model is motivated by a real-world application known as the fruit-frost problem. For an infinite-horizon, discounted version of this model it is shown that economic value remains zero below a forecast quality threshold and then rises monotonically but nonlinearly above this threshold. In particular, the relative sensitivity of economic value to changes in the quality of forecasts increases as perfect information is approached. This curve is compared with quality/value relationships that have been obtained for other versions of the model; namely, a single-stage model and a multistage, finite-horizon model.     

Monitoring and adjusting forecasts in the presence of additive outliers

The effect of an additive outlier upon the accuracy of forecasts derived from extrapolative methods is investigated. It is demonstrated that an outlier affects not only the accuracy of the forecasts at the time of occurrence but also subsequent forecasts. Methods to adjust for additive outliers are discussed. The results of the paper are illustrated with two examples.     

Linear combination of restrictions and forecasts in time series analysis

An important tool in time series analysis is that of combining information in an optimal way. Here we establish a basic combining rule of linear predictors and show that such problems as forecast updating, missing value estimation, restricted forecasting with binding constraints, analysis of outliers and temporal disaggregation can be viewed as problems of optimal linear combination of restrictions and forecasts. A compatibility test statistic is also provided as a companion tool to check that the linear restrictions are compatible with the forecasts generated from the historical data. Copyright © 2000 John Wiley & Sons, Ltd.     

The accuracy of individual and group forecasts from business outlook surveys

This paper reports on the accuracy of quarterly multiperiod predictions of inflation, real growth, unemployment and percentage changes in nominal GNP and two of its more volatile components. The survey data are highly differentiated; they cover 79 professional forecasters (mostly economists, analysts and corporate executives). Combining corresponding predictions from different sources can result in significant gains; thus the group mean forecasts are on the average over time more accurate than most of the corresponding sets of individual forecasts. But there is also a moderate degree of consistency in the relative performance of a sufficient number of the survey members, as evidenced in positive rank correlations among ratios of the individual to group root mean square errors.     

Price forecasting and evaluation: An application in agriculture

Because of the high volatility of prices of agricultural commodities over the past decade, the importance of accurate price forecasting for decision makers has become even more acute. This paper reviews literature on forecasting and evaluation. An application with forecasting U.S. hog prices is presented which includes both economic and statistical evaluation measures. Seven forecasting approaches are described and their performances are examined over 24 quarters from 1976 to 1981. These methods include exponential smoothing, an autoregressive integrated moving average process, an econometric model, expert judgement, and a composite forecasting approach. The application gives results which support previous findings in the forecasting literature and suggests that forecasting methods can provide valuable information to the decision maker.     

Seven components of judgmental forecasting skill: Implications for research and the improvement of forecasts

A decomposition of the Brier skill score shows that the performance of judgmental forecasts depends on seven components: environmental predictability, fidelity of the information system, match between environment and forecaster, reliability of information acquisition, reliability of information processing, conditional bias, and unconditional bias. These components provide a framework for research on the forecasting process. Selected literature addressing each component is reviewed, and implications for improving judgmental forecasting are discussed.     

Forecasting contemporaneous aggregates and the combination of forecasts: The case of the U.K. monetary aggregates

A number of papers in recent years have investigated the problems of forecasting contemporaneously aggregated time series and of combining alternative forecasts of a time series. This paper considers the integration of both approaches within the example of assessing the forecasting performance of models for two of the U.K. monetary aggregates, £M3 and MO. It is found that forecasts from a time series model for aggregate £M3 are superior to aggregated forecasts from individual models fitted to either the components or counterparts of £M3 and that an even better forecast is obtained by forming a linear combination of the three alternatives. For MO, however, aggregated forecasts from its components prove superior to either the forecast from the aggregate itself or from a linear combination of the two.     

Diagnostic tracking and model specification in combined forecasts of U.K. inflation

Combinations of several forecasts are now quite commonly used as inputs into business planning models. For example, capital budgeting generally involves a synthesis of several sources of economic forecasts. In such cases, where uncertainty and risk are also being explicitly considered, the statistical specification of the combined forecasts becomes particularly important. An investigation of the monthly forecasts of annual inflation from nine leading U.K. economic models was undertaken to examine the circumstances under which well-specified and efficient combined forecasts could be produced. This has helped to refine the general guidelines for the practical use of combinations in planning models.     

Sources of error in forecasts and expectations: U.K. economic models, 1984–8

This paper presents a comparative analysis of the sources of error in forecasts for the UK economy published over a recent four-year period by four independent groups. This analysis rests on the archiving at the ESRC Macroeconomic Modelling Bureau of the original forecasts together with all their accompanying assumptions and adjustments. A method of decomposing observed forecast errors so as to distinguish the contributions of forecaster and model is set out; the impact of future expectations treated in a ‘model-consistent’ or ‘rational’ manner is specifically considered. The results show that the forecaster's adjustments make a substantial contribution to forecast performance, a good part of which comes from adjustments that bring the model on track at the start of the forecast period. The published ex-ante forecasts are usually superior to pure model-based ex-post forecasts, whose performance indicates some misspecification of the underlying models.     

Perspectives on forecasting research in accounting and finance

This paper offers some perspectives on forecasting research in accounting and finance. It is maintained that many common areas of forecasting research exist. Yet, most research has focused upon a particular (Box-Jenkins) technique and a particular (reported earnings) variable, virtually neglecting numerous other relevant forecasting research topics. This symposium issue includes papers which address several of these neglected research topics. The eight papers constituting the issue are classified into three categories: (1) univariate time-series modelling; (2) multivariate time-series modelling; and (3) comparison of experts' forecasts with those of statistical models. Following a summary of the papers, some suggestions for future research are offered.     

A model of calibration for subjective probability distributions and its application to aggregation of judgements

A simple model is proposed of the statistical structure underlying the calibration of auditors' subjective probability distributions for account balances, and potentially for other unknown quantities. The model relates calibration curve shape to two parameters which represent over- or underconfidence and over- or underestimation. It is fitted to data from expert auditors. Different types of account appear to have different calibration characteristics. The model helps predict approximately the effects on calibration of aggregating individual subject distributions. Aggregation improves accuracy, but produces a strong tendency towards underconfidence. One aggregation method, predicting the best judgement in the group and using it as the group judgement, is found to be quite effective, much better than averaging the fractiles of individual distributions.     

Econometric modelling for short‐term inflation forecasting in the euro area

This paper examines the problem of forecasting macro‐variables which are observed monthly (or quarterly) and result from geographical and sectorial aggregation. The aim is to formulate a methodology whereby all relevant information gathered in this context could provide more accurate forecasts, be frequently updated, and include a disaggregated explanation as useful information for decision‐making. The appropriate treatment of the resulting disaggregated data set requires vector modelling, which captures the long‐run restrictions between the different time series and the short‐term correlations existing between their stationary transformations. Frequently, due to a lack of degrees of freedom, the vector model must be restricted to a block‐diagonal vector model. This methodology is applied in this paper to inflation in the euro area, and shows that disaggregated models with cointegration restrictions improve accuracy in forecasting aggregate macro‐variables. Copyright © 2007 John Wiley & Sons, Ltd.