Evaluating AI models on predictions the future will grade

Benchmarks leak. Once a test set is public, it ends up in the next model’s training data and the score stops meaning what it meant. ModelMirror builds evaluations that cannot leak by construction: models predict real-world events before they happen, the predictions are committed to a public, timestamped ledger, and the world supplies the answer key.

How we work

• Pre-registration by timestamp. Predictions are committed to a public git repository before outcomes exist. What a model knew is fixed by when it wrote, not by restricting what it could read.
• A public ledger. Every prediction, evaluation, and outcome is a file in a public repository. Verification needs no trust in us.
• Cross-only evaluation. Several frontier models compete; each grades the others and never itself.
• Underclaiming. Small samples are reported as small samples. Nothing is published as a result until it has resolved under the frozen process.

Projects

• FedCourtsAI
  Three frontier models predict U.S. Supreme Court events — cert grants, votes, and reasoning — on a public ledger that begins with the October 2026 term.
  Live. First forward predictions resolve in October 2026.

Contact

Issues on the relevant GitHub repository are the best channel. Updates are posted at @ModelMirrorAI.