Subjective confidence in forecasts: A response to fischhoff and MacGregor

Here we evaluate the generalizability of calibration studies which have used general knowledge questions, and argue that on conceptual, methodological and empirical grounds the results have limited applicability to judgemental forecasting. We also review evidence which suggests that judgemental forecast probabilities are influenced by variables such as the desirability, imminence, time period and perceived controllability of the event to be forecast. As these variables do not apply to judgement in the domain of general knowledge, a need for research recognizing and exploring the psychological processes underlying uncertainty about the future is apparent.     

How probable is probable? A numerical translation of verbal probability expressions

The reported experiment took place in a professional forecasting organization accustomed to giving verbal probability assessments (‘likely’, ‘probable’, etc.). It attempts to highlight the communication problems caused by verbal probability expressions and to offer possible solutions that are compatible with the forecasters overall perspective on their jobs Experts in the organization were first asked to give a numerical translation to 30 different verbal probability expressions most of which were taken from the organization's own published political forecasts. In a second part of the experiment the experts were given 15 paragraphs selected from the organization's political publications each of which contained at least one verbal expression of probability. Subjects were again asked to give a numerical translation to each verbal probability expression The results indicate that (a) there is a high variability in the interpretation of verbal probability expressions and (b) the variability is even higher in context. Possible reasons for the context effect are discussed and practical implications are suggested.     

Subjective confidence in forecasts

Forecasts have little value to decision makers unless it is known how much confidence to place in them. Those expressions of confidence have, in turn, little value unless forecasters are able to assess the limits of their own knowledge accurately Previous research has shown very robust patterns in the judgements of individuals who have not received special training in confidence assessment: Knowledge generally increases as confidence increases. However, it increases too swiftly, with a doubling of confidence being associated with perhaps a 50 per cent increase in knowledge. With all but the easiest of tasks, people tend to be overconfident regarding how much they know These results have typically been derived from studies of judgements of general knowledge. The present study found that they also pertained to confidence in forecasts. Indeed, the confidence-knowledge curves observed here were strikingly similar to those observed previously. The only deviation was the discovery that a substantial minority of judges never expressed complete confidence in any of their forecasts. These individuals also proved to be better assessors of the extent of their own knowledge Apparently confidence in forecasts is determined by processes similar to those that determine confidence in general knowledge. Decision makers can use forecasters assessments in a relative sense, in order to predict when they are more and less likely to be correct. However, they should be hesitant to take confidence assessments literally. Someone is more likely to be right when he or she is ‘certain’than when he or she is ‘fairly confident’; but there is no guarantee that the certain forecast will come true.     

Conditional distribution analyses of probabilistic forecasts

Probabilistic forecasts have good ‘external correspondence’ if events that are assigned probabilities close to 1 tend to occur frequently, whereas those assigned probabilities near 0 tend to occur rarely. This paper describes simple procedures for analysing external correspondence into meaningful components that might guide efforts to understand and improve forecasting performance. The procedures focus on differences between the judgements made by the forecaster when the target event occurs, as compared to when it does not. The illustrations involve a professional oddsmaker's predictions of baseball game outcomes, meteorologists' precipitation forecasts and physicians' diagnoses of pneumonia. The illustrations demonstrate the ability of the procedures to highlight important forecasting tendencies that are sometimes more difficult to discern by other means.     

The consistency,coherence and calibration of holistic,decomposed and recomposed judgemental probability forecasts

In this paper we make an empirical investigation of the relationship between the consistency, coherence and validity of probability judgements in a real-world forecasting context. Our results indicate that these measures of the adequacy of an individual's probability assessments are not closely related as we anticipated. Twenty-nine of our thirty-six subjects were better calibrated in point probabilities than in odds and our subjects were, in general more coherent using point probabilities than odds forecasts. Contrary to our expectations we found very little difference in forecasting response and performance between simple and compound holistic forecasts. This result is evidence against the ‘divide-and-conquer’ rationale underlying most applications of normative decision theory. In addition, our recompositions of marginal and conditional assessments into compound forecasts were no better calibrated or resolved than their holistic counterparts. These findings convey two implications for forecasting. First, untrained judgemental forecasters should use point probabilities in preference to odds. Second, judgemental forecasts of complex compound probabilities may be as well assessed holistically as they are using methods of decomposition and recomposition. In addition, our study provides a paradigm for further studies of the relationship between consistency, coherence and validity in judgemental probability forecasting.     

Effects of difficulty on judgemental probability forecasting of control response efficacy

A judgemental control task was framed as a problem of medical decision making. The control parameter of a recursive system (i.e. a patient) was initially set so that output (i.e. a diagnostic index) fell outside a designated criterion range (corresponding to health). Subjects were told to bring the system's output into the designated range by resetting this control parameter (by specifying the dose of a drug). After each of these control responses, they made a probabilistic forecast that it would have the desired effect. It was found that these forecasts were more overconfident when the control task was more difficult but that the reason for this varied. When difficulty was manipulated across subjects, there was little evidence that lower control performance was associated with any lowering of the probabilistic forecasts. When difficulty was manipulated within subjects, they did lower their forecasts for more difficult task variants but did so insufficiently. In fact, relations between probabilistic forecasts of control response efficacy and proportion of those responses that were actually effective was linear with a slope of 0.44.     

Evaluating probability forecasts in terms of refinement and strictly proper scoring rules

This note gives an easily verified necessary and sufficient condition for one probability forecaster to empirically outperform another one in terms of all strictly proper scoring rules. Copyright © 2006 John Wiley & Sons, Ltd.     

A loose and separate certainty: Caves, Fuchs and Schack on quantum probability one

Carlton Caves, Fuchs, and Schack (2002) have recently appealed to an argument of mine (Stairs, 1983) to address a problem for their subjective Bayesian account of quantum probability. The difficulty is that on the face of it, quantum mechanical probabilities of one appear to be objective, but in that case, the Born Rule would yield a continuum of probabilities between zero and one. If so, we end up with objective probabilities strictly between zero and one. The authors claim that objective probabilities of one leads to a dilemma: give up locality or fall into contradiction. I argue that this conclusion depends on an overly strong interpretation of objectivism about quantum probabilities.     

A model of calibration for subjective probability distributions and its application to aggregation of judgements

A simple model is proposed of the statistical structure underlying the calibration of auditors' subjective probability distributions for account balances, and potentially for other unknown quantities. The model relates calibration curve shape to two parameters which represent over- or underconfidence and over- or underestimation. It is fitted to data from expert auditors. Different types of account appear to have different calibration characteristics. The model helps predict approximately the effects on calibration of aggregating individual subject distributions. Aggregation improves accuracy, but produces a strong tendency towards underconfidence. One aggregation method, predicting the best judgement in the group and using it as the group judgement, is found to be quite effective, much better than averaging the fractiles of individual distributions.     

Judgemental revision of sales forecasts: The relative performance of judgementally revised versus non-revised forecasts

The judgemental revision of sales forecasts is an issue which is receiving increasing attention in the forecasting literature. This paper compares the performance of forecasts after revision by managers with that of the forecasts which were accepted by them without revision. The data set consists of sales forecasting data from an industrial company, spanning six quarterly periods and relating to some 900 individual products. The findings show that, in general, the improvements made by managers bring the forecast errors of revised forecasts more into line with non-revised forecasts, but the change is often marginal, and the best result is equivalence between revised and non-revised forecasts.     

A comparative evaluation of objective and subjective weather forecasts in the united states

This paper is concerned primarily with the evaluation and comparison of objective and subjective weather forecasts. Operational forecasts of three weather elements are considered: (1) probability forecasts of precipitation occurrence, (2) categorical (i.e. non-probabilistic) forecasts of maximum and minimum temperatures and (3) categorical forecasts of cloud amount. The objective forecasts are prepared by numerical-statistical procedures, whereas the subjective forecasts are based on the judgements of individual forecasters. In formulating the latter, the forecasters consult information from a variety of sources, including the objective forecasts themselves. The precipitation probability forecasts are found to be both reliable and skilful, and evaluation of the temperature/cloud amount forecasts reveals that they are quite accurate/skilful. Comparison of the objective and subjective forecasts of precipitation occurrence indicates that the latter are generally more skilful than the former for shorter lead times (e.g. 12–24 hours), whereas the two types of forecasts are of approximately equal skill for longer lead times (e.g. 36–48 hours). Similar results are obtained for the maximum and minimum temperature forecasts. Objective cloud amount forecasts are more skilful than subjective cloud amount forecasts for all lead times. Examination of trends in performance over the last decade reveals that both types of forecasts for all three elements increased in skill (or accuracy) over the period, with improvements in objective forecasts equalling or exceeding improvements in subjective forecasts. The role and impact of the objective forecasts in the subjective weather forecasting process are discussed in some detail. The need to conduct controlled experiments and other studies of this process, with particular reference to the assimilation of information from different sources, is emphasized. Important characteristics of the forecasting system in meteorology are identified, and they are used to describe similarities and differences between weather forecasting and forecasting in other fields. Acquisition of some of these characteristics may be beneficial to other forecasting systems.     

Judging the probability that the next point in an observed time-series will be below,or above,a given value

We presented people with trended and untrended time series and asked them to estimate the probability that the next point would be below each of seven different reference values. The true probabilities that the point would be below these values were 0.01, 0.10, 0.25, 0.50, 0.75, 0.90 and 0.99. People overestimated probabilities of less than 0.50 and underestimated those of more than 0.50. Consequently, their subjective probability distributions were flatter than they should have been: people appeared to be under confident in their estimates of where the next point would lie. This bias was greater for the trended series. It was also greater in a second experiment in which people estimated the probability that the next item would be above the reference values. We discuss reasons for these effects and consider their implications for decision making.     

Empirical analysis of systematic errors in Chilean GDP forecasts

This paper presents a statistical comparison between the actual and predicted evolution of the Chilean GDP for the period 1986 – 1998 made by several forecasters. We show that the forecasters systematically underestimate the true growth rate of the economy. The magnitude of this bias tends to be correlated with the phase of the business cycle. Copyright © 2001 John Wiley & Sons, Ltd.     

Bayesian consensus forecasts of macroeconomic variables

Economists, like other forecasters, share knowledge, data and theories in common. Consequently, their forecast errors are likely to be highly dependent. This paper reports on an empirical study of 16 macroeconomic forecasters. Composite forecasts are computed using a sequential weighting scheme that takes dependence into account; these are compared to a simple average and median forecasts. A within-sample composite is also calculated. Both these methods perform significantly better than the average or median of the forecasts. This improvement in accuracy is apparently because the dependence between the forecasters' errors is so high that the optimal composite forecasts sometimes lie outside the range of the individual forecasts.     

Volatility forecasts using stochastic volatility models with nonlinear leverage effects

The leverage effect—the correlation between an asset's return and its volatility—has played a key role in forecasting and understanding volatility and risk. While it is a long standing consensus that leverage effects exist and improve forecasts, empirical evidence puzzlingly does not show that this effect exists for many individual stocks, mischaracterizing risk, and therefore leading to poor predictive performance. We examine this puzzle, with the goal to improve density forecasts, by relaxing the assumption of linearity of the leverage effect. Nonlinear generalizations of the leverage effect are proposed within the Bayesian stochastic volatility framework in order to capture flexible leverage structures. Efficient Bayesian sequential computation is developed and implemented to estimate this effect in a practical, on-line manner. Examining 615 stocks that comprise the S&P500 and Nikkei 225, we find that our proposed nonlinear leverage effect model improves predictive performances for 89% of all stocks compared to the conventional stochastic volatility model.     

The role of probability arguments in the history of science

The paper examines Wesley Salmon’s claim that the primary role of plausibility arguments in the history of science is to impose constraints on the prior probability of hypotheses (in the language of Bayesian confirmation theory). A detailed look at Copernicanism and Darwinism and, more briefly, Rutherford’s discovery of the atomic nucleus reveals a further and arguably more important role of plausibility arguments. It resides in the consideration of likelihoods, which state how likely a given hypothesis makes a given piece of evidence. In each case the likelihoods raise the probability of one of the competing hypotheses and diminish the credibility of its rival, and this may happen either on the basis of ‘old’ or ‘new’ evidence.     

The problem of confirmation in the Everett interpretation

I argue that the Oxford school Everett interpretation is internally incoherent, because we cannot claim that in an Everettian universe the kinds of reasoning we have used to arrive at our beliefs about quantum mechanics would lead us to form true beliefs. I show that in an Everettian context, the experimental evidence that we have available could not provide empirical confirmation for quantum mechanics, and moreover that we would not even be able to establish reference to the theoretical entities of quantum mechanics. I then consider a range of existing Everettian approaches to the probability problem and show that they do not succeed in overcoming this incoherence.     

Short‐term forecasts of euro area real GDP growth: an assessment of real‐time performance based on vintage data

Forecasters commonly predict real gross domestic product growth from monthly indicators such as industrial production, retail sales and surveys, and therefore require an assessment of the reliability of such tools. While forecast errors related to model specification and unavailability of data in real time have been assessed, the impact of data revisions on forecast accuracy has seldom been evaluated, especially for the euro area. This paper proposes to evaluate the contributions of these three sources of forecast error using a set of data vintages for the euro area. The results show that gains in accuracy of forecasts achieved by using monthly data on actual activity rather than surveys or financial indicators are offset by the fact that the former set of monthly data is harder to forecast and less timely than the latter set. These results provide a benchmark which future research may build on as more vintage datasets become available. Copyright © 2008 John Wiley & Sons, Ltd.     

Combination forecasts of output growth in a seven‐country data set

This paper uses forecast combination methods to forecast output growth in a seven‐country quarterly economic data set covering 1959–1999, with up to 73 predictors per country. Although the forecasts based on individual predictors are unstable over time and across countries, and on average perform worse than an autoregressive benchmark, the combination forecasts often improve upon autoregressive forecasts. Despite the unstable performance of the constituent forecasts, the most successful combination forecasts, like the mean, are the least sensitive to the recent performance of the individual forecasts. While consistent with other evidence on the success of simple combination forecasts, this finding is difficult to explain using the theory of combination forecasting in a stationary environment. Copyright © 2004 John Wiley & Sons, Ltd.     

Comparing the accuracy of density forecasts from competing models

A rapidly growing literature emphasizes the importance of evaluating the forecast accuracy of empirical models on the basis of density (as opposed to point) forecasting performance. We propose a test statistic for the null hypothesis that two competing models have equal density forecast accuracy. Monte Carlo simulations suggest that the test, which has a known limiting distribution, displays satisfactory size and power properties. The use of the test is illustrated with an application to exchange rate forecasting. Copyright © 2004 John Wiley & Sons, Ltd.