Accurate forecasting is crucial for decision-making in fields like policy, finance, and science, but mid- and long-term predictions remain challenging to aggregate effectively. Traditional methods, which often rely on simple averages or weighted historical performance, may not fully capture a forecaster’s skill—especially for unresolved questions where data is sparse. One way to address this gap could be by using Full-Accuracy Scoring (FAS), a method that evaluates forecasters based on both their past accuracy and how their predictions for unresolved questions align with the aggregated forecast.

How FAS Works

FAS combines two key metrics to assess forecasting skill:

• Past Accuracy: Measures how well a forecaster performed on questions with known outcomes.
• "Future" Accuracy: Compares a forecaster’s predictions for unresolved questions to the aggregated forecast (serving as a proxy for the likely outcome).

By balancing these dimensions, FAS could identify skilled forecasters more quickly than traditional methods, particularly for long-term predictions where historical data is limited. For example, on platforms like Metaculus, FAS might help improve aggregated forecasts by weighting contributors more dynamically.

Potential Benefits and Stakeholders

This approach could benefit:

• Forecasting platforms: Improved accuracy could enhance credibility and attract more users.
• Decision-makers: Policymakers and investors might rely on more reliable long-term forecasts.
• Forecasters: Skilled participants could gain recognition faster, incentivizing high-quality contributions.

Platforms might adopt FAS if it proves superior to existing methods, while forecasters could be motivated by faster rewards—though some might resist if their performance is exposed as weaker.

Implementation and Challenges

A minimal test could involve partnering with a forecasting platform to apply FAS to a subset of questions, comparing its performance against traditional aggregation. Key challenges might include:

• Ensuring forecasters don’t game the system by merely copying aggregated predictions.
• Balancing the weighting between past and "future" accuracy when historical data is scarce.

If successful, FAS could be expanded across platforms, offering a more nuanced way to evaluate and aggregate forecasts—especially for long-term, uncertain events.