Subjective confidence in forecasts: A response to fischhoff and MacGregor

Here we evaluate the generalizability of calibration studies which have used general knowledge questions, and argue that on conceptual, methodological and empirical grounds the results have limited applicability to judgemental forecasting. We also review evidence which suggests that judgemental forecast probabilities are influenced by variables such as the desirability, imminence, time period and perceived controllability of the event to be forecast. As these variables do not apply to judgement in the domain of general knowledge, a need for research recognizing and exploring the psychological processes underlying uncertainty about the future is apparent.     

The consistency,coherence and calibration of holistic,decomposed and recomposed judgemental probability forecasts

In this paper we make an empirical investigation of the relationship between the consistency, coherence and validity of probability judgements in a real-world forecasting context. Our results indicate that these measures of the adequacy of an individual's probability assessments are not closely related as we anticipated. Twenty-nine of our thirty-six subjects were better calibrated in point probabilities than in odds and our subjects were, in general more coherent using point probabilities than odds forecasts. Contrary to our expectations we found very little difference in forecasting response and performance between simple and compound holistic forecasts. This result is evidence against the ‘divide-and-conquer’ rationale underlying most applications of normative decision theory. In addition, our recompositions of marginal and conditional assessments into compound forecasts were no better calibrated or resolved than their holistic counterparts. These findings convey two implications for forecasting. First, untrained judgemental forecasters should use point probabilities in preference to odds. Second, judgemental forecasts of complex compound probabilities may be as well assessed holistically as they are using methods of decomposition and recomposition. In addition, our study provides a paradigm for further studies of the relationship between consistency, coherence and validity in judgemental probability forecasting.     

Subjective versus objective combining of forecasts: An experiment

The paper reports results of an experiment conducted to evaluate subjective versus objective combination of forecasts. The subjects were undergraduate students at Texas A&M. The students forecasted two different types of time series. The results found show that the subjective combination of forecasts improves their accuracy as compared with individual efforts. Four ex-ante weighting methods were also used to combine the forecasts. They all improve the accuracy of the forecasts. The best results, though, were from the subjective combination of forecasts.     

An exploration of some practical issues in the use of quantitative forecasting models

Extrapolative forecasting models have been available for many years and as most organizations have the need to regularly develop forecasts one might anticipate the widespread use of these models. The evidence in Australia indicates that computer based forecasting systems are not being widely used and in fact a number of established systems have been discarded, with the issue of forecast accuracy often being mentioned as a problem area. Two experiments are carried out to examine this issue by comparing judgemental and quantitative forecasts. Other problem areas mentioned as contributing to the abandonment of forecasting systems include the difficulty of manually reviewing the computer forecasts and the effort required to carefully massage the forecast database to remove extraordinary events.     

Judgemental revision of sales forecasts: The relative performance of judgementally revised versus non-revised forecasts

The judgemental revision of sales forecasts is an issue which is receiving increasing attention in the forecasting literature. This paper compares the performance of forecasts after revision by managers with that of the forecasts which were accepted by them without revision. The data set consists of sales forecasting data from an industrial company, spanning six quarterly periods and relating to some 900 individual products. The findings show that, in general, the improvements made by managers bring the forecast errors of revised forecasts more into line with non-revised forecasts, but the change is often marginal, and the best result is equivalence between revised and non-revised forecasts.     

The effectiveness of imprecise probability forecasts

In this paper we investigate the feasibility of algorithmically deriving precise probability forecasts from imprecise forecasts. We provide an empirical evaluation of precise probabilities that have been derived from two types of imprecise probability forecasts: probability intervals and probability intervals with second-order probability distributions. The minimum cross-entropy (MCE) principle is applied to the former to derive precise (i.e. additive) probabilities; expectation (EX) is used to derive precise probabilities in the latter case. Probability intervals that were constructed without second-order probabilities tended to be narrower than and contained in those that were amplified by second-order probabilities. Evidence that this narrowness is due to motivational bias is presented. Analysis of forecasters' mean Probability Scores for the derived precise probabilities indicates that it is possible to derive precise forecasts whose external correspondence is as good as directly assessed precise probability forecasts. The forecasts of the EX method, however, are more like the directly assessed precise forecasts than those of the MCE method.     

How probable is probable? A numerical translation of verbal probability expressions

The reported experiment took place in a professional forecasting organization accustomed to giving verbal probability assessments (‘likely’, ‘probable’, etc.). It attempts to highlight the communication problems caused by verbal probability expressions and to offer possible solutions that are compatible with the forecasters overall perspective on their jobs Experts in the organization were first asked to give a numerical translation to 30 different verbal probability expressions most of which were taken from the organization's own published political forecasts. In a second part of the experiment the experts were given 15 paragraphs selected from the organization's political publications each of which contained at least one verbal expression of probability. Subjects were again asked to give a numerical translation to each verbal probability expression The results indicate that (a) there is a high variability in the interpretation of verbal probability expressions and (b) the variability is even higher in context. Possible reasons for the context effect are discussed and practical implications are suggested.     

The use of non-time series information in sales forecasting: A case study

The contribution of product and industry knowledge to the accuracy of sales forecasting was investigated by examining the company forecasts of a leading manufacturer and marketer of consumable products. The company forecasts of 18 products produced by a meeting of marketing, sales, and production personnel were compared with those generated by the same company personnel when denied specific product knowledge and with the forecasts of selected judgemental and statistical time series methods. Results indicated that product knowledge contributed significantly to forecast accuracy and that the forecast accuracy of company personnel who possessed industry forecasting knowledge (but not product knowledge) was not significantly different from the time series based methods. Furthermore, the company forecasts were more accurate than averages of the judgemental and statistical time series forecasts. These results point to the importance of specific product information to forecast accuracy and accordingly call into question the continuing strong emphasis on improving extrapolation techniques without consideration of the inclusion of non-time series knowledge.     

Effects of difficulty on judgemental probability forecasting of control response efficacy

A judgemental control task was framed as a problem of medical decision making. The control parameter of a recursive system (i.e. a patient) was initially set so that output (i.e. a diagnostic index) fell outside a designated criterion range (corresponding to health). Subjects were told to bring the system's output into the designated range by resetting this control parameter (by specifying the dose of a drug). After each of these control responses, they made a probabilistic forecast that it would have the desired effect. It was found that these forecasts were more overconfident when the control task was more difficult but that the reason for this varied. When difficulty was manipulated across subjects, there was little evidence that lower control performance was associated with any lowering of the probabilistic forecasts. When difficulty was manipulated within subjects, they did lower their forecasts for more difficult task variants but did so insufficiently. In fact, relations between probabilistic forecasts of control response efficacy and proportion of those responses that were actually effective was linear with a slope of 0.44.     

Judgemental revision of sales forecasts: Effectiveness of forecast selection

When managers make revisions to sales forecasts initially generated by a rational quantitative model it is important that the particular forecasts selected for adjustment are those which would benefit most from the adjustment process (i.e. realize high errors). This study reports an empirical investigation on this issue, spanning six quarterly forecasting periods and incorporating forecasting data on over 850 products. The results show that the errors of the forecasts chosen for revision are, in general, higher than those which were not chosen. In addition, it is shown that managesrs tend to revise forecasts which are initially low, hence possibily introducing some degree of bias into the overall forecasts.     

Judgemental revision of sales forecasts: A longitudinal eetension

When quantitative models are used for short-term multi-item sales forecasts it is possible that the managers who use such forecasts may disagree with at least some of the estimates obtained, and wish to change them so that they become more consistent with their own (subjective) evaluation of the marketplace. This study reports on an analysis of the effectiveness of judgemental revision of sales forecasts over six quarterly forecasting periods. The results give general support for the practice of forecast manipulation as a means of improving forecasting accuracy. It is also observed that the effectiveness of revision activity varies across different time periods.     

Testing for unreliable estimators and insignificant forecasts in combined forecasts

The reliability and precision of the weights used in combining individual forecasts, irrespective of the method of combination, is important in evaluating a combined forecast. The objective of this study is not to suggest the ‘best’ method of combining individual forecasts, but rather to propose exploratory procedures, that make use of all available sample information contained in the covariance matrix of individual forecast errors, to (1) detect if the weights used in combining forecasts are ‘reliable’ (and ‘stable’ if it is known that the covariance matrix of forecast errors is stationary over time) and (2) test for ‘insignificant’ individual forecasts used in forming a combined forecast. We present empirical applications using two-year sales and individual forecast data provided by a major consumer durables manufacturer to illustrate the feasibility of our proposed procedures.     

Efficient use of information in the formation of subjective industry forecasts

The paper considers the use of information by a panel of expert industry forecasters, focusing on their information-processing biases. The panel forecasts construction output by sector up to three years ahead. It is found that the biases observed in laboratory experiments, particularly ‘anchoring’, are observable. The expectations are formed by adjusting the previous forecast to take new information into account. By analysing forecast errors it is concluded that the panel overweight recently released information and do not understand the dynamics of the industry. However, their forecasts, both short and long term, are better than an alternative econometric model, and combining the two sources of forecasts leads to a deterioration in forecast accuracy. The expert forecasts can be ‘de-biased’, and this leads to the conclusion that it is better to optimally process information sources than to combine (optimally) alternative forecasts.     

Conditional distribution analyses of probabilistic forecasts

Probabilistic forecasts have good ‘external correspondence’ if events that are assigned probabilities close to 1 tend to occur frequently, whereas those assigned probabilities near 0 tend to occur rarely. This paper describes simple procedures for analysing external correspondence into meaningful components that might guide efforts to understand and improve forecasting performance. The procedures focus on differences between the judgements made by the forecaster when the target event occurs, as compared to when it does not. The illustrations involve a professional oddsmaker's predictions of baseball game outcomes, meteorologists' precipitation forecasts and physicians' diagnoses of pneumonia. The illustrations demonstrate the ability of the procedures to highlight important forecasting tendencies that are sometimes more difficult to discern by other means.     

An exploratory analysis of portfolio managers' probabilistic forecasts of stock prices

This study reports the results of an experiment that examines (1) the effects of forecast horizon on the performance of probability forecasters, and (2) the alleged existence of an inverse expertise effect, i.e., an inverse relationship between expertise and probabilistic forecasting performance. Portfolio managers are used as forecasters with substantive expertise. Performance of this ‘expert’ group is compared to the performance of a ‘semi-expert’ group composed of other banking professionals trained in portfolio management. It is found that while both groups attain their best discrimination performances in the four-week forecast horizon, they show their worst calibration and skill performances in the 12-week forecast horizon. Also, while experts perform better in all performance measures for the one-week horizon, semi-experts achieve better calibration for the four-week horizon. It is concluded that these results may signal the existence of an inverse expertise effect that is contingent on the selected forecast horizon.     

Experience in combining subjective and quantitative forecasts of open market rates

This paper presents a method of combining subjective information from open-market operators with results from a time-series forecasting model. Empirical results of forecasts for interest rates of bank reserves are presented.     

Linear constraints and the efficiency of combined forecasts

Studies of combined forecasts have typically constrained the combining weights to sum to one and have not included a constant term in the combination. In a recent paper, Granger and Ramanathan (1984) have argued in favour of an unrestricted linear combination, including a constant term. This paper shows that for the purpose of prediction it may make sense to impose restrictions on the combining model because of potential increases in forecasting efficiency. Empirical results show that small gains in forecasting efficiency can be obtained by restricting the linear combination of GNP forecasts from four econometric models.     

Regional,individual and political determinants of FOMC members' key macroeconomic forecasts

We study Federal Open Market Committee members' individual forecasts of inflation and unemployment in the period 1992–2004. Our results imply that Governors and Bank presidents forecast differently, with Governors submitting lower inflation and higher unemployment rate forecasts than bank presidents. For Bank presidents we find a regional bias, with higher district unemployment rates being associated with lower inflation and higher unemployment rate forecasts. Bank presidents' regional bias is more pronounced during the year prior to their elections or for nonvoting bank presidents. Career backgrounds or political affiliations also affect individual forecast behavior.     

A model of calibration for subjective probability distributions and its application to aggregation of judgements

A simple model is proposed of the statistical structure underlying the calibration of auditors' subjective probability distributions for account balances, and potentially for other unknown quantities. The model relates calibration curve shape to two parameters which represent over- or underconfidence and over- or underestimation. It is fitted to data from expert auditors. Different types of account appear to have different calibration characteristics. The model helps predict approximately the effects on calibration of aggregating individual subject distributions. Aggregation improves accuracy, but produces a strong tendency towards underconfidence. One aggregation method, predicting the best judgement in the group and using it as the group judgement, is found to be quite effective, much better than averaging the fractiles of individual distributions.