Forecasting platforms like Metaculus and INFER rely on aggregation methods to combine individual predictions into more accurate forecasts. However, there's no systematic way to determine which aggregation techniques perform best as new data and methods emerge. This gap makes it hard for platforms and users to adopt the most effective approaches confidently.

Comparing Aggregation Methods

One way to address this could be to conduct a comprehensive comparison of forecast aggregation techniques using resolved questions from existing platforms. The analysis would:

• Replicate and extend previous studies with newer data
• Test additional or updated aggregation methods
• Evaluate performance using metrics like Brier score and logarithmic loss
• Share results through accessible formats like interactive dashboards

The workflow might involve collecting resolved forecasts, implementing different aggregation methods, comparing their performance, and testing robustness across various parameters and question types.

Potential Impact and Implementation

Such a comparison could benefit forecasting platforms by helping them optimize their algorithms, while researchers and individual forecasters could use the findings to improve their prediction strategies. A phased approach might start with validating the method on a small dataset before expanding to larger analyses and newer techniques.

The main challenges would include ensuring data accessibility and managing computational complexity, but these could potentially be addressed through platform collaborations and code optimization. The results could provide empirical evidence to help forecasting communities move beyond theoretical debates about aggregation methods.