What is crowd forecasting?
Crowd forecasting combines predictions from a large group of people with different perspectives to estimate the probability of future events. Averaging these predictions can yield a smaller error than a typical individual forecast.
Forecasting tournaments provide a way to test these methods against events that later resolve. IARPA’s ACE program collected millions of forecasts through the Good Judgment Project and released the data for research.
What makes a crowd's forecast useful?
A large group alone doesn't guarantee a good forecast. The way people form and combine their views matters:
- People need different perspectives and information sources. A group whose members all think alike is more vulnerable to groupthink.
- Participants should make their first estimates before seeing anyone else's. This helps prevent them from simply following the group.
- Forecasters need a reason to aim for accuracy, whether that's a reputation to build, a financial reward, or points in a game.
- The method for combining forecasts matters, too. Options include a simple mean, an average weighted by past performance, or a market price.
What superforecasters do differently
Some forecasters have performed consistently well in forecasting tournaments. The Good Judgment Project found that combining the estimates of those with the strongest historical accuracy improved forecasts. Results from a particular tournament do not guarantee the same performance on different questions.
A market lets participants trade on their judgments, but liquidity, fees and contract wording also affect their results.
Where crowd forecasts are used
Forecasting questions can cover more than elections. The following are examples of questions that organizations could investigate:
Supply chains
Forecasts help businesses anticipate changes in consumer demand and logistics bottlenecks.
Public health
Data from frontline workers and health researchers helps track disease outbreaks.
Cybersecurity
Forecasters estimate the likelihood of zero-day exploits and major attacks on infrastructure.
Climate policy
Forecasts estimate how changes in legislation could affect carbon prices and renewable energy adoption.
Combining AI and human forecasts
IARPA’s Hybrid Forecasting Competition studied systems that combine human judgments with machine forecasts. This is an area of research: adding AI does not by itself establish that a forecast is accurate, and results need to be tested against resolved events.