Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence
Quick Answer
This paper introduces a novel benchmark for evaluating AI systems, leveraging Levin's Kt to create high-difficulty C-tests for large language models (LLMs).
Quick Take
The tests can generate items beyond current AI capabilities, assessing supra-intelligence levels. Twelve were administered these tests, revealing a decline in performance as difficulty increased, indicating a new frontier in AI evaluation.
Key Points
- Levin's Kt provides a theoretical upper bound for AI test difficulty.
- The new C-tests can generate items beyond current AI capabilities.
- Twelve LLMs were tested, showing performance decline with increased difficulty.
- The study formalizes a measure for AI evaluation using LLMs.
- The tests explore the gap between examiner and examinee capabilities.
DeepSignal Analysis
What happened
A new benchmark for evaluating AI systems has been introduced, utilizing Levin's Kt to create challenging C-tests for large language models (LLMs). Twelve LLMs were tested, showing a decline in performance as the difficulty of the tests increased, suggesting a new approach to AI evaluation.
Key evidence
- The paper presents a benchmark based on Levin's Kt, which sets a theoretical upper limit on the difficulty of tests for AI systems.
- Twelve LLMs were administered these tests, and their performance decreased as the difficulty increased, indicating limitations in current AI capabilities.
- The tests aim to generate items that exceed the capabilities of existing AI models, exploring the concept of supra-intelligence.
Why it matters
This research addresses the challenge of evaluating AI systems that may surpass current human and machine capabilities. By establishing a benchmark that can generate increasingly difficult tests, it provides a framework for assessing future advancements in AI. The implications of this work could influence how AI systems are developed and evaluated, pushing the boundaries of what is considered intelligent behavior.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Under the accelerated pace of AI development, any benchmark whose difficulty is calibrated with humans or current AI systems becomes circular, or saturates soon. How can we then evaluate machines that may become more capable than the current machines and the people who write the tests? One answer to the question hinges on inverse problems, inductive inference questions for which difficulty depends on the complexity of the underlying mechanism, given the observations and the previous knowledge. Levin's Kt, a computable variant of Kolmogorov complexity, puts a theoretical upper bound to that difficulty, balancing the simplest representation encoding for the mechanism and the logarithm of the time steps to run that mechanism. Previous tests derived from Levin's Kt, such as the original C-test, are based on minimalistic Universal Turing Machines (UTM), capable of very limited representations and no previous knowledge. Reasoning large language models (LLM) are UTMs that capture a compendium of human culture as knowledge, using prompts as natural language programs. This paper presents the first test based on Levin's Kt with LLMs as representational machines. Fresh test items can be produced at high difficulties, estimated in a ratio scale measured in bits, before any human or frontier model can solve them. We formalise the measure, devise a lightweight generation and certification pipeline, explore the effect of the power of the reference machine and the gap between examiner and examinee. We produce a first battery of certified items administered to twelve LLMs. The resulting agent characteristic curves decrease with difficulty, down to the uncharted high-difficulty region, the supra-intelligence window of items we can certify, yet beyond generations of frontier intelligence.
| Comments: | Preprint, under review. 63 pages. Code and data at this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14005 [cs.AI] |
| (or arXiv:2610.14005v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14005 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jose Hernandez-Orallo [view email]
[v1]
Sun, 27 Sep 2026 20:41:46 UTC (541 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →VEPA: Verified Evolving Formal Policy for Lifelong Agent Guardrails
VEPA introduces a framework for verified evolving formal policies for LLM agents, enhancing safety by adapting guardrails based on deployment feedback. It outperforms existing methods across benchmarks like ASSEBench-Safety and ASSEBench-Security, ensuring deterministic policy enforcement even under challenging conditions.