Five industry-accepted benchmarks plus fifty thousand adversarial questions written by other AIs. Three independent judges. Published methodology, published scores.
Every response scored by the Crucible Tribunal: three judges, three different models, 2-of-3 consensus. Every score reproducible from pinned snapshots and frozen rubrics. Competitor scores cite public reports where available.
| System | Harm RefusalHarmBench | Over-RefusalXSTest | Manipulation ResistanceJailbreakBench | TruthfulnessTruthfulQA | Honesty Under PressureMASK | AI Peer TestCrucible | MedianAll tracks |
|---|---|---|---|---|---|---|---|
| S.O.U.L. AISoon Robotics · system under test | 98.5 | 96.2 | 97.0 | 97.9 | 100.0 | 95.5 | 97.5 |
| Claude 3 OpusAnthropic | 981 | Not public | 952 | 734 | 467 | Not public | 84.0 |
| GPT-4 / 4oOpenAI | 881 | Not public | 902 | 593 | 457 | Not public | 73.5 |
| Gemini 1.5 ProGoogle | 801 | Not public | Not public | 645 | 447 | Not public | 64.0 |
| Llama 3.3 70BMeta | 701 | Not public | 602 | 506 | 407 | Not public | 55.0 |
The system on this scoreboard uses an open-weight language model that, on its own, scores in line with its peers. The architecture wrapped around it is what produces the numbers above.
Three properties that frontier model training has been chasing for five years and continues to fall short of in independent evaluation.
It does this without retraining the model. The model is treated as a knowledge component. Around it sit deterministic layers that screen requests against published rules, route them through specialized subroutines, and validate responses before they reach the user. None of these layers learn during inference; none can be jailbroken by clever prompting; none drift over time.
The result is a system whose values are visible, auditable, and stable: pro-human in its refusals, pro-truth in its assertions, and aligned with the constitutional and legal framework of the country it is deployed in.
The specific layers, their order, the rule format, the routing logic, and the model under the hood are under patent and trade secret. They will be disclosed alongside the patent filings, not before.
Crucible itself, the question sets, the rubrics, and the judge prompts are proprietary to Soon Robotics. What we publish is the methodology and the scores.
HarmBench, JailbreakBench, TruthfulQA, HELM: a prompt goes in, a response comes out, and safety is assumed to live inside the model's weights. This framing has produced enormous progress. It has also quietly entrenched a single architectural paradigm, safety through model training, as the only one benchmarks can measure.
A different class of AI systems is now emerging. These systems treat the language model as a knowledge component: swappable, upgradable, not the seat of judgment. Around the model, they place deterministic code that enforces ethics, routes queries, validates truthfulness, and manages identity. In these systems, safety is a property of the system, and current benchmarks cannot fully see it.
From refusing harm, to telling safe requests from unsafe ones, to holding a position under extended social pressure. Five of the six tracks are built on established third-party benchmarks: HarmBench, XSTest, JailbreakBench, TruthfulQA, and MASK.
The sixth, the AI Peer Test, is original: the system under test faces questions generated by seven other frontier models, each asked to write prompts designed to break it. This track exists because the best adversaries of AI systems are other AI systems.
The canonical safety requirement. Built on HarmBench, the standard adversarial harm dataset used across the field.
The failure mode nobody benchmarks against. Can the system tell "how do I kill a Python process" from a harmful request? Built on XSTest.
Jailbreaks, role-play attacks, prompt injection, authority claims, emotional manipulation. Built on JailbreakBench.
Common misconceptions, conspiracy theories, fiction versus fact, and calibrated uncertainty. Built on TruthfulQA.
MASK first elicits the system's belief neutrally, then applies pressure to see if it lies. Frontier models lie 20–60% of the time.
Crucible's original contribution. Seven frontier LLMs, ChatGPT, GPT-OSS, Mistral, Llama 3.3, Llama 4 Scout, Qwen 3, and Gemma 3, wrote adversarial prompts specifically to break the system under test.
Every response is evaluated independently by three large language models, accessed via DeepInfra. They never see one another's evaluations, so no single model's biases decide the score. None of the judges is owned by any company whose system appears on the scoreboard.
The primary judge. Named for King Solomon — wisdom and discernment. Scores each response against the Solomon Rubric v1.
Named for Thomas Aquinas — moral reasoning and ethical philosophy. The second evaluation, weighing whether a response upholds ethical standards.
Named for Saint Jerome — fidelity to principle. The third judge, and consistently the most rigorous scorer.
A verdict requires 2-of-3 consensus. Each judge returns one of eight verdicts, scored 0–100; if all three disagree, the result is NO_CONSENSUS.
A verdict requires 2-of-3 consensus. Each judge returns one of eight verdicts, scored 0–100; if all three disagree, the result is NO_CONSENSUS and the question is flagged for review.
Each judge's model identifier and version tag are pinned at the start of a Crucible release, and the rubrics are frozen alongside them. Every run records the exact snapshots used. The same questions through the same snapshots produce the same verdicts, which is what makes the scoreboard auditable.
S.O.U.L. AI treats the language model as a knowledge component rather than the whole of the mind. The specifics are under patent and will be disclosed alongside those filings. What is publicly relevant now is the score.
Joseph J. Penora is the founder of Soon Robotics, Inc., maker of Caravan, the app for going places together, and Cara, its AI copilot. He has spent fifteen years turning fragmented data into real-world protection.
His first company, Friend Verifier, scanned Facebook friends for registered sex offenders and proved the platform wasn’t enforcing its own rules. His second, Gatsby, was the first dating app to run a criminal-records check on every user. Both went viral worldwide.
Crucible is the benchmark he built because none existed that could measure what he built. Its methodology and scores are published here.