Aggregating predictions from multiple experts is a common challenge in fields like forecasting, decision-making, and machine learning. However, current methods often rely on simplistic heuristics or assume ideal conditions, ignoring complexities like partial information access, varying noise levels, and timing differences in expert predictions. This gap leaves room for improvement in real-world applications, from prediction markets to climate modeling, where more nuanced aggregation could yield better results.

The Idea: A Study on Optimizing Expert Forecast Aggregation

One approach to addressing this problem could involve a theoretical and computational study simulating expert predictions in controlled scenarios. For instance, experts might predict whether the sum of 10 dice rolls will exceed a certain threshold, with each expert seeing only a random subset of rolls or receiving noisy updates. The study would explore different weighting methods, such as:

• Performance-based weighting: Experts with better past accuracy receive higher weights.
• Time-based weighting: Later predictions, which may incorporate more information, are weighted more heavily.
• Hybrid methods: Combining factors like timing and performance for optimal results.

The goal would be maximizing the accuracy of the aggregated forecast, measured using proper scoring rules like the log score. Theoretical analysis could derive optimal weights under ideal conditions, while computational experiments would test robustness in more realistic, noisy environments.

Potential Applications and Execution

Insights from such a study could benefit forecasting platforms (e.g., improving aggregation algorithms), machine learning practitioners (e.g., refining ensemble methods), and decision-makers relying on expert input. An execution plan might involve:

1. Deriving theoretical solutions for simple cases.
2. Building simulations to model more complex scenarios.
3. Validating findings with real-world data from existing forecasting projects.

A minimal viable product could focus on a single toy scenario, like dice-roll predictions, to compare basic weighting approaches before scaling to more complex cases.

By systematically studying how to weight expert forecasts under realistic conditions, this approach could offer practical improvements over current heuristic methods while advancing theoretical understanding of aggregation mechanics.