Conservatism and consensus-seeking among economic forecasters

This paper uses the track records of a panel of US economic forecasters participating in a consensus forecasting service to test for conservatism and consensus-seeking behaviour. The tests are based on a particular method-of-moments estimator, designed to allow for the heteroscedasticity and serial correlation which is inevitably present in errors from repeated forecasts for fixed target dates. Most forecasters prove to be conservative. When revising forecasts they give too much weight to their own past forecasts. Surprisingly, forecasters are not consensus-seeking but ‘variety-seeking’. When revising forecasts, they give too little weight to the known forecasts of other forecasters.     

The consistency,coherence and calibration of holistic,decomposed and recomposed judgemental probability forecasts

In this paper we make an empirical investigation of the relationship between the consistency, coherence and validity of probability judgements in a real-world forecasting context. Our results indicate that these measures of the adequacy of an individual's probability assessments are not closely related as we anticipated. Twenty-nine of our thirty-six subjects were better calibrated in point probabilities than in odds and our subjects were, in general more coherent using point probabilities than odds forecasts. Contrary to our expectations we found very little difference in forecasting response and performance between simple and compound holistic forecasts. This result is evidence against the ‘divide-and-conquer’ rationale underlying most applications of normative decision theory. In addition, our recompositions of marginal and conditional assessments into compound forecasts were no better calibrated or resolved than their holistic counterparts. These findings convey two implications for forecasting. First, untrained judgemental forecasters should use point probabilities in preference to odds. Second, judgemental forecasts of complex compound probabilities may be as well assessed holistically as they are using methods of decomposition and recomposition. In addition, our study provides a paradigm for further studies of the relationship between consistency, coherence and validity in judgemental probability forecasting.     

Strategic bias and professional affiliations of macroeconomic forecasters

This paper investigates strategic motives of macroeconomic forecasters and the effect of their professional affiliations. The ‘wishful expectations hypothesis’ suggests that a forecaster predicts what his employer wishes. The ‘publicity hypothesis’ argues that forecasters are evaluated by both accuracy and ability to generate publicity, and that forecasters in industries that emphasize publicity most will make most extreme and least accurate predictions. The ‘signaling hypothesis’ asserts that an extreme forecast signals confidence in own ability, because incompetent forecasters would mimic others to avoid public notice. Empirical evidence from a 26‐year panel of annual GDP forecasts is con‐sistent with the publicity hypothesis. This indicates that conventional tests of rationality are biased toward rejecting the rational expectations hypothesis. Copyright ? 2008 John Wiley & Sons, Ltd.     

A comparative evaluation of objective and subjective weather forecasts in the united states

This paper is concerned primarily with the evaluation and comparison of objective and subjective weather forecasts. Operational forecasts of three weather elements are considered: (1) probability forecasts of precipitation occurrence, (2) categorical (i.e. non-probabilistic) forecasts of maximum and minimum temperatures and (3) categorical forecasts of cloud amount. The objective forecasts are prepared by numerical-statistical procedures, whereas the subjective forecasts are based on the judgements of individual forecasters. In formulating the latter, the forecasters consult information from a variety of sources, including the objective forecasts themselves. The precipitation probability forecasts are found to be both reliable and skilful, and evaluation of the temperature/cloud amount forecasts reveals that they are quite accurate/skilful. Comparison of the objective and subjective forecasts of precipitation occurrence indicates that the latter are generally more skilful than the former for shorter lead times (e.g. 12–24 hours), whereas the two types of forecasts are of approximately equal skill for longer lead times (e.g. 36–48 hours). Similar results are obtained for the maximum and minimum temperature forecasts. Objective cloud amount forecasts are more skilful than subjective cloud amount forecasts for all lead times. Examination of trends in performance over the last decade reveals that both types of forecasts for all three elements increased in skill (or accuracy) over the period, with improvements in objective forecasts equalling or exceeding improvements in subjective forecasts. The role and impact of the objective forecasts in the subjective weather forecasting process are discussed in some detail. The need to conduct controlled experiments and other studies of this process, with particular reference to the assimilation of information from different sources, is emphasized. Important characteristics of the forecasting system in meteorology are identified, and they are used to describe similarities and differences between weather forecasting and forecasting in other fields. Acquisition of some of these characteristics may be beneficial to other forecasting systems.     

How probable is probable? A numerical translation of verbal probability expressions

The reported experiment took place in a professional forecasting organization accustomed to giving verbal probability assessments (‘likely’, ‘probable’, etc.). It attempts to highlight the communication problems caused by verbal probability expressions and to offer possible solutions that are compatible with the forecasters overall perspective on their jobs Experts in the organization were first asked to give a numerical translation to 30 different verbal probability expressions most of which were taken from the organization's own published political forecasts. In a second part of the experiment the experts were given 15 paragraphs selected from the organization's political publications each of which contained at least one verbal expression of probability. Subjects were again asked to give a numerical translation to each verbal probability expression The results indicate that (a) there is a high variability in the interpretation of verbal probability expressions and (b) the variability is even higher in context. Possible reasons for the context effect are discussed and practical implications are suggested.     

Overreaction in Survey Exchange Rate Forecasts

We use survey data on five bilateral exchange rates to provide empirical evidence of the fact that professional forecasters of foreign exchange rates behave irrationally, in the specific sense that they respond inaccurately to available information in the market when forming their predictions. In particular, we find systematic biases in the forecasts resulting in the overreaction of analysts to past information contained in the exchange rate dynamics: forecasters change their prediction more than it would be rational on the basis of past realized changes. In addition, forecasters are heterogeneous in their irrationality: low performers in previous periods show a more pronounced overreaction effect. This can be read as an indication of perpetration of past errors and continued inability to learn from the past. In the second part of the paper, we exploit the novel structure of our dataset, which consists of survey data extracted from the Bloomberg platform and readily available to anyone. This feature allows us to consider their own and others' past forecasts as part of the information set that analysts use in making their predictions. By using past forecasts as proxies for relevant macroeconomic variables, we find evidence that analysts fail to correctly process not only the information contained in the spot rate past dynamics but also the information in this broader set. We see this as confirmation of the existence of inefficiency and heterogeneity between low and high performers also when full information is available. Copyright © 2014 John Wiley & Sons, Ltd.     

The uses and abuses of ‘consensus’ forecasts

Clemen's (1989) review of the forecast-combining literature amply illustrates both the interest in and the importance of this subject. This article stresses the tautological properties of various consensus measures that assure their success relative to most individual forecasts. It confirms the finding of earlier studies that for each specific macroeconomic variable roughly one-third of individual forecasters are more accurate than a consensus. However, each individual does relatively poorly for some variable while the consensus, in contrast, necessarily never fails relative to most individuals. These results, like most previous studies, describe consensus measures that are synthetic constructs derived from a pre-existing set of individual forecasts. Strictly speaking, this contemporaneous consensus is not available to individual forecasters when their forecasts are made. A prior consensus measure, which is in their information sets, was relatively much less accurate than the contemporaneous measure. Nevertheless, a small subset of individual forecasters were generally inferior to the known, prior consensus forecast.     

Bayesian consensus forecasts of macroeconomic variables

Economists, like other forecasters, share knowledge, data and theories in common. Consequently, their forecast errors are likely to be highly dependent. This paper reports on an empirical study of 16 macroeconomic forecasters. Composite forecasts are computed using a sequential weighting scheme that takes dependence into account; these are compared to a simple average and median forecasts. A within-sample composite is also calculated. Both these methods perform significantly better than the average or median of the forecasts. This improvement in accuracy is apparently because the dependence between the forecasters' errors is so high that the optimal composite forecasts sometimes lie outside the range of the individual forecasts.     

Strategic bias,herding behaviour and economic forecasts

Professional forecasters can have other objectives as well as minimizing expected squared forecast errors. This paper studies whether the people or companies which make forecasts behave strategically with the aim of maximizing aspects such as publicity, salary or their prestige, or more generally to minimize some loss function; or whether, on the contrary, they make forecasts which resemble consensus forecasts (herding behaviour). This study also analyses whether, as forecasters gain more reputation and experience, they make more radical forecasts, that is, they deviate further from the consensus. For this the Livingston Survey is used, a panel of experts who make forecasts on the future evolution of the United States economy. Copyright © 2003 John Wiley & Sons, Ltd.     

US Inflation Expectations and Heterogeneous Loss Functions, 1968–2010

Recent literature has suggested that macroeconomic forecasters may have asymmetric loss functions, and that there may be heterogeneity across forecasters in the degree to which they weigh under‐ and over‐predictions. Using an individual‐level analysis that exploits the Survey of Professional Forecasters respondents’ histogram forecasts, we find little evidence of asymmetric loss for the inflation forecasters. Copyright © 2013 John Wiley & Sons, Ltd.     

The role of macroeconometric models in forecasting and policy analysis in the united states

This article stresses how little is known about the quality, particularly the relative quality, of macroeconometric models. Most economists make a strict distinction between the quality of a model per se and the accuracy of solutions based on that model. While this distinction is valid, it leaves unanswered how to compare the‘validity’of conditional models. The standard test, the accuracy of ex post simulations, is not definitive when models with differing degrees of exogeneity are compared. In addition, it is extremely difficult to estimate the relative quantitative importance of conceptual problems of models, such as parameter instability across‘policy regimes’ In light of the difficulty in comparisons of conditional macroeconometric models, many model-builders and users assume that the best models are those that have been used to make the most accurate forecasts are those made with the best models. Forecasting experience indicates that forecasters using macroeconometric models have produced more accurate macroeconomic forecasts than either naive or sophisticated unconditional statistical models. It also suggests that judgementally adjusted forecasts have been more accurate than model-based forecasts generated mechanically. The influence of econometrically-based forecasts is now so pervasive that it is difficult to find examples of‘purely judgemental’forecasts.     

Combining exchange rate forecasts: What is the optimal consensus measure?

In this paper a high-quality disaggregate database is utilized to examine whether individual forecasters produce efficient exchange rate predictions and also if the properties of the forecasts change when they are combined. The paper links a number of themes in the exchange rate literature and examines various methods of forecast combination. It is demonstrated, inter alia, that some forecasters are better than others, but that most are not as good as a naive no-change prediction. Combining forecasts adds to the accuracy of the predictions, but the gains mainly reflect the removal of systematic and unstable bias.     

Social networks and private spaces in economic forecasting

The outputs of economic forecasting—predictions for national economic indicators such as GDP, unemployment rates and inflation—are all highly visible. The production of these forecasts is a much more private affair, however, typically being thought of as the work of individual forecasters or forecast teams using their economic model to produce a forecast that is then made public. This conception over-emphasises the individual and the technical whilst silencing the broader social context through which economic forecasters develop the expertise that is essential for the credibility of their predictions. In particular, economic forecasts are given meaning and fine-tuned through the social and institutional networks that give forecasters access to the expertise of a heterogeneous mix of academics, policy-makers and business people. Within these broader groups, individual forecasters often create private forecast ‘clubs’, where subscribers have privileged access to the expertise of the economist, but where the forecasters also have privileged access to their clients’ own expert knowledge. In examining these aspects of the forecasters’ work I show that the visible and audible activities of modelling and forecasting are made possible and plausible by virtue of the modeller’s invisible interaction with a wider network.     

Robust evaluation of fixed‐event forecast rationality

In this paper we introduce a new testing procedure for evaluating the rationality of fixed‐event forecasts based on a pseudo‐maximum likelihood estimator. The procedure is designed to be robust to departures in the normality assumption. A model is introduced to show that such departures are likely when forecasters experience a credibility loss when they make large changes to their forecasts. The test is illustrated using monthly fixed‐event forecasts produced by four UK institutions. Use of the robust test leads to the conclusion that certain forecasts are rational while use of the Gaussian‐based test implies that certain forecasts are irrational. The difference in the results is due to the nature of the underlying data. Copyright © 2001 John Wiley & Sons, Ltd.     

Efficient use of information in the formation of subjective industry forecasts

The paper considers the use of information by a panel of expert industry forecasters, focusing on their information-processing biases. The panel forecasts construction output by sector up to three years ahead. It is found that the biases observed in laboratory experiments, particularly ‘anchoring’, are observable. The expectations are formed by adjusting the previous forecast to take new information into account. By analysing forecast errors it is concluded that the panel overweight recently released information and do not understand the dynamics of the industry. However, their forecasts, both short and long term, are better than an alternative econometric model, and combining the two sources of forecasts leads to a deterioration in forecast accuracy. The expert forecasts can be ‘de-biased’, and this leads to the conclusion that it is better to optimally process information sources than to combine (optimally) alternative forecasts.     

Do US Macroeconomic Forecasters Exaggerate their Differences?

Application of the Bernhardt et al. (Journal of Financial Economics 2006; 80 (3): 657–675) test of herding to the calendar‐year annual output growth and inflation forecasts suggests forecasters tend to exaggerate their differences, except at the shortest horizon, when they tend to herd. We consider whether these types of behaviour can help to explain the puzzle that professional forecasters sometimes make point predictions and histogram forecasts which are mutually inconsistent. Copyright © 2015 John Wiley & Sons, Ltd.     

Adjusting for information content when comparing forecast performance

Cross‐institutional forecast evaluations may be severely distorted by the fact that forecasts are made at different points in time and therefore with different amounts of information. This paper proposes a method to account for these differences when analyzing an unbalanced panel of forecasts. The method computes the timing effect and the forecaster's ability simultaneously. Monte Carlo simulation demonstrates that evaluations that do not adjust for the differences in information content may be misleading. In addition, the method is applied to a real‐world dataset of 10 Swedish forecasters for the period 1999–2015. The results show that the ranking of the forecasters is affected by the proposed adjustment.     

Heterogeneous Forecast Adjustment

There is ample empirical evidence that expert‐adjusted model forecasts can be improved. One way to potential improvement concerns providing various forms of feedback to the sales forecasters. It is also often recognized that the experts (forecasters) might not constitute a homogeneous group. This paper provides a data‐based methodology to discern latent clusters of forecasters, and applies it to a fully new large database with data on expert‐adjusted forecasts, model forecasts and realizations. For the data at hand, two clusters can clearly be identified. Next, the consequences of having clusters are discussed. Copyright © 2016 John Wiley & Sons, Ltd.     

Extraneous expert information

When a Bayesian decision maker has to choose among information sources, he should consider the anticipated impact that the information will have on his posterior distribution. In some cases he may determine at the outset that an information source will have no effect on his posterior beliefs, no matter what that source says. Such an information source is called extraneous. In this paper we discuss Bayesian conditions for extraneous information sources, and show a hypothetical example involving experts with overlapping information. Analysis of U.S. weather forecasts demonstrates how this concept can be operationalized to test hypotheses concerning the use of information by forecasters.     

Conditional distribution analyses of probabilistic forecasts

Probabilistic forecasts have good ‘external correspondence’ if events that are assigned probabilities close to 1 tend to occur frequently, whereas those assigned probabilities near 0 tend to occur rarely. This paper describes simple procedures for analysing external correspondence into meaningful components that might guide efforts to understand and improve forecasting performance. The procedures focus on differences between the judgements made by the forecaster when the target event occurs, as compared to when it does not. The illustrations involve a professional oddsmaker's predictions of baseball game outcomes, meteorologists' precipitation forecasts and physicians' diagnoses of pneumonia. The illustrations demonstrate the ability of the procedures to highlight important forecasting tendencies that are sometimes more difficult to discern by other means.