Sup AI & Humanity's Last Exam: The Ensemble Advantage

Achieving 52.15% Accuracy on the World's Hardest AI Reasoning Benchmark

Key Findings

52.15%
New SOTA accuracy on HLE
(714/1,369 questions)
+7.41
Points over nearest competitor
(Gemini 3 Pro Preview at 44.74%)
3
Impossible Solves
Correct answers from all-wrong ensembles

1. Executive Summary

Sup AI has established a new state-of-the-art (SOTA) accuracy of 52.15% on Humanity's Last Exam (HLE), a benchmark designed to test the limits of modern AI reasoning. This white paper analyzes the underlying data to understand how an ensemble approach outperforms individual frontier models.

Our analysis reveals that Sup AI's success is driven, in part, by the strategic selection of diverse models. In particular, Google's Gemini 3 Pro Preview and xAI's Grok 4 possess unique, uncorrelated capabilities. Most remarkably, Sup AI demonstrated constructive synthesis capability, producing 3 correct answers from questions where every constituent model failed—extracting valid reasoning fragments from incorrect attempts and fusing them through confidence-weighted verification. This represents a qualitative leap beyond traditional ensemble methods.

Key Findings

  • New SOTA: 52.15% accuracy on HLE (714/1,369 questions)
  • +7.41 points over nearest competitor (Gemini 3 Pro Preview at 44.74%)
  • Specialist Effect: Grok 4 (29.05% accuracy) uniquely solved 16 questions vs. GPT-5 Pro's 9 (despite 39.53% accuracy)
  • 3 Impossible Solves: Sup AI derived correct answers from questions where every component model was wrong
  • Diversity Beats Correlation: Gemini 3 Pro + Grok 4 (0.54 correlation) outperforms Gemini 3 Pro + GPT-5 Pro (0.75 correlation)
  • Frontier of Impossibility: 575 questions (42%) remain unsolved by ALL tested models

2. The Ensemble Advantage

Sup AI orchestrates frontier models through four integrated stages:

  1. Orchestrator: Routes queries to optimal model subsets based on problem type and model specialization patterns—prioritizing diversity of failure modes over raw accuracy alone.
  2. Parallel Execution: Selected models process simultaneously with tool augmentation (web search, document parsing, code execution). For multimodal HLE questions, Sup AI pre-processes images/PDFs so text-only models can participate.
  3. Verification & Confidence: Component-level scoring using token probability distributions identifies high-quality reasoning fragments. When confidence falls below threshold or models disagree materially, the system triggers targeted retries.
  4. Synthesis Layer: Fuses validated reasoning fragments across models—enabling correct answers even when all individual models fail.

3. The 'Specialist' Effect

A common misconception in AI evaluation is that a model with higher overall accuracy is strictly better than one with lower accuracy. Our analysis of Unique Genius proves this wrong. We isolated questions where exactly one model provided the correct answer while every other model in the ensemble failed.

The results highlight the critical role of specialist models:

  • Google Gemini 3 Pro Preview was the unique solver for 23 questions, the highest of any individual model.
  • xAI Grok 4, despite having a lower overall accuracy (29.05%) than GPT-5 Pro (39.53%), delivered 16 unique correct answers-almost double GPT-5 Pro and more than every other model except Gemini 3 Pro Preview.

4. The 3 'Impossible' Solves: Constructive Synthesis Beyond Model Selection

The Phenomenon

The most significant finding of this analysis is not that Sup AI correctly aggregates correct answers—it is that Sup AI can synthesize correct answers from ensembles where no correct answer exists. In 3 instances, every model in the ensemble produced an incorrect response, yet Sup AI derived the correct solution.

This was achieved not through model selection or majority voting, but through constructive synthesis: extracting valid reasoning fragments from incorrect attempts and fusing them via confidence-weighted verification.

Quantitative Context

These 3 synthesized solutions represent a 0.52% penetration into the Impossible Range—the 575 questions (42% of HLE) where zero models provided correct answers.

5. Theoretical Ceiling

While Sup AI was able to answer 714 questions correct, there were a total of 797 questions where at least one individual LLM was able to answer them correctly. This indicates that, without further improvements, Sup AI can potentially score 58.22% on HLE.

6. Robustness & The 'IQ Curve'

💡 Insight

Sup AI remains consistently elevated above the pack after the initial 0-20% zone, indicating it successfully identifies the correct signals even when the majority of its constituent models are failing.

7. Orchestration Dynamics

The correlation heatmap reveals distinct families of reasoning, demonstrating that if two models are highly correlated, they provide near-zero marginal information gain. Low correlation, like between Gemini 3 Pro and Grok 4, is the primary driver of the ensemble's resilience.

8. The Frontier of Impossibility

This categorizes the evaluated questions by how many models answered them correctly, validating HLE as a future-proof benchmark.

9. Conclusion

The data from Humanity's Last Exam demonstrates that the next leap in AI performance will come from closing the gap between actual and theoretical ensemble performance and breaking the impossible barrier with new model capabilities.

10. About Sup AI

Sup AI is an intelligent orchestration platform that dynamically routes queries across frontier language models, synthesizing responses through confidence-weighted analysis. By leveraging the complementary strengths of diverse AI systems, Sup AI consistently outperforms any individual model on complex reasoning tasks.

11. Disclaimers & Data Availability

The individual model scores presented in this white paper are higher than their officially published benchmarks due to our evaluation methodology utilizing prompt engineering and web search. The full dataset of questions, model responses, and the per-question correctness table used for the analysis in this paper is available for verification.