Subjective confidence in forecasts

Forecasts have little value to decision makers unless it is known how much confidence to place in them. Those expressions of confidence have, in turn, little value unless forecasters are able to assess the limits of their own knowledge accurately Previous research has shown very robust patterns in the judgements of individuals who have not received special training in confidence assessment: Knowledge generally increases as confidence increases. However, it increases too swiftly, with a doubling of confidence being associated with perhaps a 50 per cent increase in knowledge. With all but the easiest of tasks, people tend to be overconfident regarding how much they know These results have typically been derived from studies of judgements of general knowledge. The present study found that they also pertained to confidence in forecasts. Indeed, the confidence-knowledge curves observed here were strikingly similar to those observed previously. The only deviation was the discovery that a substantial minority of judges never expressed complete confidence in any of their forecasts. These individuals also proved to be better assessors of the extent of their own knowledge Apparently confidence in forecasts is determined by processes similar to those that determine confidence in general knowledge. Decision makers can use forecasters assessments in a relative sense, in order to predict when they are more and less likely to be correct. However, they should be hesitant to take confidence assessments literally. Someone is more likely to be right when he or she is ‘certain’than when he or she is ‘fairly confident’; but there is no guarantee that the certain forecast will come true.     

Combining exchange rate forecasts: What is the optimal consensus measure?

In this paper a high-quality disaggregate database is utilized to examine whether individual forecasters produce efficient exchange rate predictions and also if the properties of the forecasts change when they are combined. The paper links a number of themes in the exchange rate literature and examines various methods of forecast combination. It is demonstrated, inter alia, that some forecasters are better than others, but that most are not as good as a naive no-change prediction. Combining forecasts adds to the accuracy of the predictions, but the gains mainly reflect the removal of systematic and unstable bias.     

Strategic bias,herding behaviour and economic forecasts

Professional forecasters can have other objectives as well as minimizing expected squared forecast errors. This paper studies whether the people or companies which make forecasts behave strategically with the aim of maximizing aspects such as publicity, salary or their prestige, or more generally to minimize some loss function; or whether, on the contrary, they make forecasts which resemble consensus forecasts (herding behaviour). This study also analyses whether, as forecasters gain more reputation and experience, they make more radical forecasts, that is, they deviate further from the consensus. For this the Livingston Survey is used, a panel of experts who make forecasts on the future evolution of the United States economy. Copyright © 2003 John Wiley & Sons, Ltd.     

US Inflation Expectations and Heterogeneous Loss Functions, 1968–2010

Recent literature has suggested that macroeconomic forecasters may have asymmetric loss functions, and that there may be heterogeneity across forecasters in the degree to which they weigh under‐ and over‐predictions. Using an individual‐level analysis that exploits the Survey of Professional Forecasters respondents’ histogram forecasts, we find little evidence of asymmetric loss for the inflation forecasters. Copyright © 2013 John Wiley & Sons, Ltd.     

A comparative evaluation of objective and subjective weather forecasts in the united states

This paper is concerned primarily with the evaluation and comparison of objective and subjective weather forecasts. Operational forecasts of three weather elements are considered: (1) probability forecasts of precipitation occurrence, (2) categorical (i.e. non-probabilistic) forecasts of maximum and minimum temperatures and (3) categorical forecasts of cloud amount. The objective forecasts are prepared by numerical-statistical procedures, whereas the subjective forecasts are based on the judgements of individual forecasters. In formulating the latter, the forecasters consult information from a variety of sources, including the objective forecasts themselves. The precipitation probability forecasts are found to be both reliable and skilful, and evaluation of the temperature/cloud amount forecasts reveals that they are quite accurate/skilful. Comparison of the objective and subjective forecasts of precipitation occurrence indicates that the latter are generally more skilful than the former for shorter lead times (e.g. 12–24 hours), whereas the two types of forecasts are of approximately equal skill for longer lead times (e.g. 36–48 hours). Similar results are obtained for the maximum and minimum temperature forecasts. Objective cloud amount forecasts are more skilful than subjective cloud amount forecasts for all lead times. Examination of trends in performance over the last decade reveals that both types of forecasts for all three elements increased in skill (or accuracy) over the period, with improvements in objective forecasts equalling or exceeding improvements in subjective forecasts. The role and impact of the objective forecasts in the subjective weather forecasting process are discussed in some detail. The need to conduct controlled experiments and other studies of this process, with particular reference to the assimilation of information from different sources, is emphasized. Important characteristics of the forecasting system in meteorology are identified, and they are used to describe similarities and differences between weather forecasting and forecasting in other fields. Acquisition of some of these characteristics may be beneficial to other forecasting systems.     

Do US Macroeconomic Forecasters Exaggerate their Differences?

Application of the Bernhardt et al. (Journal of Financial Economics 2006; 80 (3): 657–675) test of herding to the calendar‐year annual output growth and inflation forecasts suggests forecasters tend to exaggerate their differences, except at the shortest horizon, when they tend to herd. We consider whether these types of behaviour can help to explain the puzzle that professional forecasters sometimes make point predictions and histogram forecasts which are mutually inconsistent. Copyright © 2015 John Wiley & Sons, Ltd.     

Bayesian consensus forecasts of macroeconomic variables

Economists, like other forecasters, share knowledge, data and theories in common. Consequently, their forecast errors are likely to be highly dependent. This paper reports on an empirical study of 16 macroeconomic forecasters. Composite forecasts are computed using a sequential weighting scheme that takes dependence into account; these are compared to a simple average and median forecasts. A within-sample composite is also calculated. Both these methods perform significantly better than the average or median of the forecasts. This improvement in accuracy is apparently because the dependence between the forecasters' errors is so high that the optimal composite forecasts sometimes lie outside the range of the individual forecasts.     

The role of judgement in macroeconomic forecasting

This paper examines the rationale for, and influence of, judgemental adjustments in macroeconomic forecasting, using two particular forecasts for the UK economy recently published by the National Institute of Economic and Social Research and the London Business School. It is found that in some cases such adjustments have a major effect on the forecasts and can also explain some of the differences in the two rival forecasts. However the number of adjustments for which this is true is not great. An implication of these findings is that, if these forecasts can be regarded as typical, then macroeconomic forecasters should be urged to give a reasonable account of the role of judgemental adjustments in their forecasts, particularly since the amount of information which would be required is not likely to be excessive.     

Heterogeneous Forecast Adjustment

There is ample empirical evidence that expert‐adjusted model forecasts can be improved. One way to potential improvement concerns providing various forms of feedback to the sales forecasters. It is also often recognized that the experts (forecasters) might not constitute a homogeneous group. This paper provides a data‐based methodology to discern latent clusters of forecasters, and applies it to a fully new large database with data on expert‐adjusted forecasts, model forecasts and realizations. For the data at hand, two clusters can clearly be identified. Next, the consequences of having clusters are discussed. Copyright © 2016 John Wiley & Sons, Ltd.     

The uses and abuses of ‘consensus’ forecasts

Clemen's (1989) review of the forecast-combining literature amply illustrates both the interest in and the importance of this subject. This article stresses the tautological properties of various consensus measures that assure their success relative to most individual forecasts. It confirms the finding of earlier studies that for each specific macroeconomic variable roughly one-third of individual forecasters are more accurate than a consensus. However, each individual does relatively poorly for some variable while the consensus, in contrast, necessarily never fails relative to most individuals. These results, like most previous studies, describe consensus measures that are synthetic constructs derived from a pre-existing set of individual forecasts. Strictly speaking, this contemporaneous consensus is not available to individual forecasters when their forecasts are made. A prior consensus measure, which is in their information sets, was relatively much less accurate than the contemporaneous measure. Nevertheless, a small subset of individual forecasters were generally inferior to the known, prior consensus forecast.     

Robust evaluation of fixed‐event forecast rationality

In this paper we introduce a new testing procedure for evaluating the rationality of fixed‐event forecasts based on a pseudo‐maximum likelihood estimator. The procedure is designed to be robust to departures in the normality assumption. A model is introduced to show that such departures are likely when forecasters experience a credibility loss when they make large changes to their forecasts. The test is illustrated using monthly fixed‐event forecasts produced by four UK institutions. Use of the robust test leads to the conclusion that certain forecasts are rational while use of the Gaussian‐based test implies that certain forecasts are irrational. The difference in the results is due to the nature of the underlying data. Copyright © 2001 John Wiley & Sons, Ltd.     

Social networks and private spaces in economic forecasting

The outputs of economic forecasting—predictions for national economic indicators such as GDP, unemployment rates and inflation—are all highly visible. The production of these forecasts is a much more private affair, however, typically being thought of as the work of individual forecasters or forecast teams using their economic model to produce a forecast that is then made public. This conception over-emphasises the individual and the technical whilst silencing the broader social context through which economic forecasters develop the expertise that is essential for the credibility of their predictions. In particular, economic forecasts are given meaning and fine-tuned through the social and institutional networks that give forecasters access to the expertise of a heterogeneous mix of academics, policy-makers and business people. Within these broader groups, individual forecasters often create private forecast ‘clubs’, where subscribers have privileged access to the expertise of the economist, but where the forecasters also have privileged access to their clients’ own expert knowledge. In examining these aspects of the forecasters’ work I show that the visible and audible activities of modelling and forecasting are made possible and plausible by virtue of the modeller’s invisible interaction with a wider network.     

Correcting the January optimism effect

Each month, various professional forecasters give forecasts for next year's real gross domestic product (GDP) growth and unemployment. January is a special month, when the forecast horizon moves to the following calendar year. Instead of deleting the January data when analyzing forecast updates, I propose a periodic version of a test regression for weak-form efficiency. An application of this periodic model for many forecasts across a range of countries shows that in January GDP forecast updates are positive, whereas the forecast updates for unemployment are negative. I document that this January optimism about the new calendar year is detrimental to forecast accuracy. To empirically analyze Okun's law, I also propose a periodic test regression, and its application provides more support for this law.     

Combining ordinal forecasts with an application in a financial market

The literature on combining forecasts has almost exclusively focused on combining point forecasts. The issues and methods of combining ordinal forecasts have not yet been fully explored, even though ordinal forecasting has many practical applications in business and social research. In this paper, we consider the case of forecasting the movement of the stock market which has three possible states (bullish, bearish and sluggish). Given the sample of states predicted by different forecasters, several statistical and operation research methods can be applied to determine the optimal weight assigned to each forecaster in combining the ordinal forecasts. The performance of these methods is examined using Hong Kong stock market forecasting data, and their accuracies are found to be better than the consensus method and individual forecasts.     

Are forecasters reluctant to revise their predictions? Some German evidence

People are reluctant to admit mistakes. This could also be true of economic forecasters. If revisions of past forecasts are costly, then it will become optimal for forecasters to only partially adjust a past forecast in the light of new information. The unwillingness to admit to the mistake in the old forecast generates a bias of the new forecast in the direction of the old forecast. We test this hypothesis for the joint predictions of the Association of German Economic Research Institutes over the last 35 years. We find some evidence for such a bias and compute the implied unwillingness to revise forecasts. Copyright © 2006 John Wiley & Sons, Ltd.     

Overreaction in Survey Exchange Rate Forecasts

We use survey data on five bilateral exchange rates to provide empirical evidence of the fact that professional forecasters of foreign exchange rates behave irrationally, in the specific sense that they respond inaccurately to available information in the market when forming their predictions. In particular, we find systematic biases in the forecasts resulting in the overreaction of analysts to past information contained in the exchange rate dynamics: forecasters change their prediction more than it would be rational on the basis of past realized changes. In addition, forecasters are heterogeneous in their irrationality: low performers in previous periods show a more pronounced overreaction effect. This can be read as an indication of perpetration of past errors and continued inability to learn from the past. In the second part of the paper, we exploit the novel structure of our dataset, which consists of survey data extracted from the Bloomberg platform and readily available to anyone. This feature allows us to consider their own and others' past forecasts as part of the information set that analysts use in making their predictions. By using past forecasts as proxies for relevant macroeconomic variables, we find evidence that analysts fail to correctly process not only the information contained in the spot rate past dynamics but also the information in this broader set. We see this as confirmation of the existence of inefficiency and heterogeneity between low and high performers also when full information is available. Copyright © 2014 John Wiley & Sons, Ltd.     

Adjusting for information content when comparing forecast performance

Cross‐institutional forecast evaluations may be severely distorted by the fact that forecasts are made at different points in time and therefore with different amounts of information. This paper proposes a method to account for these differences when analyzing an unbalanced panel of forecasts. The method computes the timing effect and the forecaster's ability simultaneously. Monte Carlo simulation demonstrates that evaluations that do not adjust for the differences in information content may be misleading. In addition, the method is applied to a real‐world dataset of 10 Swedish forecasters for the period 1999–2015. The results show that the ranking of the forecasters is affected by the proposed adjustment.     

Professional economic forecasts: Are they worth their costs?

Most economic forecast evaluations dating back 20 years show that professional forecasters add little to the forecasts generated by the simplest of models. Using various types of forecast error criteria, these evaluations usually conclude that the professional forecasts are little better than the no-change or ARIM A type forecast. It is our contention that this conclusion is mistaken because the conventional error criteria may not capture why forecasts are ma& or how they are used. Using forecast directional accuracy, the criterion which has been found to be highly correlated with profits in an interest rate setting, we find that professional GNP forecasts dominate the cheaper alternatives. Moreover, there appears to be no systematic relationship between this preferred criterion and the error measures used in previous studies.     

Professional forecasters' expectations,consistency, and international spillovers

This paper focuses on the expectation formation process of professional forecasters by relying on survey data on forecasts regarding gross domestic product growth, consumer price index inflation and 3-month interest rates for a broad set of countries. We examine the interrelation between macroeconomic forecasts and also the impact of uncertainty on forecasts by allowing for cross-country interdependencies and time variation in the coefficients. We find that professional forecasts are often in line with the Taylor rule and identify significant expectation spillovers from monetary policy in the USA.     

The accuracy of individual and group forecasts from business outlook surveys

This paper reports on the accuracy of quarterly multiperiod predictions of inflation, real growth, unemployment and percentage changes in nominal GNP and two of its more volatile components. The survey data are highly differentiated; they cover 79 professional forecasters (mostly economists, analysts and corporate executives). Combining corresponding predictions from different sources can result in significant gains; thus the group mean forecasts are on the average over time more accurate than most of the corresponding sets of individual forecasts. But there is also a moderate degree of consistency in the relative performance of a sufficient number of the survey members, as evidenced in positive rank correlations among ratios of the individual to group root mean square errors.