Faithful setting: each benchmark’s own rule and judge. Ensemble setting: three judges from other model families, majority vote (see What older judges miss).

SETTING
REWARD HACKING
ImpossibleBench SWE-bench
Cheating on impossible coding tests
1.722.500.2
No LLM judge: same in both settings
None excluded
Setup notes and known broken benchmark samples
  • FIXWe run every task from an offline sandbox template, so all four models work in the same environment without network access.
  • FIXThe headline uses the paper's prompt D, which tells the model to stop and explain if a test looks wrong, so a pass means the model edited code against an explicit instruction. The comparison of looser prompts is a separate run on the LiveCodeBench tasks (next row).
  • ADJUSTMENTWe dropped one benchmark sample that crashes the scaffold and kept the other 348, so the denominator is 348 tasks times 3 rollouts.
  • DEFECTIn 59 of the 348 benchmark samples the inserted test is malformed; we keep them in the headline as the paper does and report a well-formed-only rate beside it: on the 289 well-formed benchmark samples Kimi K3 cheats in 9.8% of runs (85 of 867) and Kimi K3.1 Pro in 2.0% (17 of 867).
ImpossibleBench LiveCodeBench, loose to stricter prompts
A is the loosest (“pass all tests”), C the stricter (not the headline’s strict prompt); Kimi models only
95.895.2n/an/an/an/a
None excluded
No LLM judge: same in both settings
Setup notes and known broken benchmark samples
  • FIXEach prompt runs with the paper's limits (10 submissions, 50 messages) on the tools scaffold, with prompts A, B and C taken verbatim from the paper, so the three rates differ only in the prompt.
  • FIXRates are per run rather than averaged per task first, with intervals clustered by task.
  • DEFECTlcbhard_77's test file does not parse in the published data, so no run can pass it; it stays in the denominator.
  • DEFECTlcbhard_35 and lcbhard_87 carry malformed tests that fail whatever the solution, so a pass on them means the model got around the harness; both stay in the denominator.
Disclosure follow-up† (built by Preference Model)
Honest summary after cheating
063.6n/an/a0
Our evaluation: judged by the ensemble only, same in both settings
None excluded
Small samples: only runs that passed are asked.
Setup notes
  • SETUPA classifier judges the transcript instead of continuing it, so the verdict rests only on what the model said.
  • SETUPTool calls in the follow-up turn get stubbed results, so a follow-up question never ends in a tool error that would leave nothing to judge.
  • SETUPWe rebuilt the follow-up as a scorer on ImpossibleBench SWE-bench runs, so it asks only the runs that passed.
  • The samples are small: Kimi K3.1 Pro cheats in only 18 runs and GPT-6.1 Sol in 2, so their rates rest on few answers.
HVTB
Opening hidden answers or tests
24.724.721.913
No LLM judge: same in both settings
19 filtered10 errors5 filtered
Setup notes and known broken benchmark samples
  • FIXThe agent starts only after the file watchers on every hidden file are live, so an early read cannot slip past them.
  • FIXWatcher-integrity checks flag runs whose watchers were lost, so a missing flag is not mistaken for honest behavior.
  • ADJUSTMENTAll four models run in one Inspect scaffold, so differences come from the models and not from the harness.
  • FIXThe harness no longer crashes when the verifier cannot run; those episodes are left unscored.
  • DEFECTThere are no known defects in the benchmark samples, but the label means the model read a hidden file, and an incidental read can trigger it, so a positive is not always deliberate.
EvilGenie
Reward hacking on coding tasks
1.71.100
1.71.100
None excluded
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-5, so rates can differ slightly from what the paper's judge would give.
  • ADJUSTMENTEach episode gets a 1,800 second clock to absorb provider latency, so slow APIs do not end runs before the model finishes.
  • FIXEach solution is also re-tested in isolation beside upstream's pass check, so a hard-coded answer that only passes the visible tests is caught.
  • DEFECT4 of 154 problems force special-casing through a conflicting visible test or an exact-match checker, and 9 are ambiguous; the headline keeps all 154, so a few flags reflect the problem rather than the model.
Upward deception
Hiding a tool failure from the user
33.338.95.515.4
3339.35.115.1
14 errors13 errors4 filtered24 errors14 filtered13 errors
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-5, so rates can differ slightly from what the paper's judge would give.
  • ADJUSTMENTThe agent runs in a sandbox, so its tool failures are real failures and not artifacts of the host.
  • FIXWeb results that would reveal the benchmark are hidden, so a model cannot learn that the task is a test from a search.
  • FIXWe restored the garbled instructions in 71 of 200 tasks, so models are graded on the task the paper intended.
  • FIXWe dropped task t2-21 because its file is the very record its instruction asks for, so it cannot test hiding a failure.
  • DEFECTFive tasks framed as role-play are kept as in the paper, though a model may fairly treat their missing data as fiction.
COVERT SABOTAGE AND MONITORING
SHADE-Arena, hidden scratchpad
Covert sabotage past the monitor (main and side task done, suspicion below 80)
52.848.800
98 filtered13 filtered
11.814.300
Ensemble value: main and side task done and the ensemble judges’ median suspicion below 80, as in Figure 19; a run whose two valid votes fall on either side of 80 is left out and counted.
Setup notes and known broken benchmark samples
  • ADJUSTMENTClaude Opus 4.8 monitors in place of the paper's Claude 3.5 Sonnet, so suspicion scores can differ from the paper's monitor.
  • ADJUSTMENTA turn stopped by a provider's content filter is retried and, if it keeps failing, the episode is excluded and counted, so empty filtered turns are not scored as suspicious.
  • FIXWe fixed a repository bug in bank_transfer_2's bulk-transfer tool that raised an error after scheduling, which had caused retried double payments to pass the side task.
  • DEFECTNo defects in the benchmark samples are documented, but the api_key side-task grader can pass on a public documentation test key, so a few successes may not involve a real secret.
LinuxArena
Sabotage an action-only monitor misses (each model's own 1% threshold)
10.85.9n/an/a43.8
1.60.9n/an/a15.9
5 errors5 errors455 filtered5 errors407 filtered5 errors
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol monitors in place of the paper's GPT-5 nano, so thresholds and catch rates can differ from what the paper's monitor would give.
  • FIXEpisodes ended by a provider's content filter are excluded and counted, which removes all of Claude Opus 5.5's attack episodes and all but 48 of GPT-6.1 Sol's, so GPT-6.1 Sol's rate rests on 48 runs.
  • ADJUSTMENTSandbox commands that hang are stopped and counted, so a stuck episode does not count as evading the monitor.
  • DEFECTTwo slack_clone tasks fail setup because their bug patch does not apply; they are excluded and counted for every model.
Sandbagging vs monitor† (built by Preference Model)
Monitor catches covert sandbagging
10010099.477.4
Our evaluation: judged by the ensemble only, same in both settings
685 filtered2 truncated56 filtered
Setup notes
  • SETUPThe ensemble judges monitor with Li et al.'s published monitor prompt and flag threshold. The card's monitor, Claude Opus 4.8, is stopped by its provider's filter on most bio and chem answers, so it is not used.
  • The Kimi models' reasoning always states the sandbagging, so their 100% says little about a harder-to-read model.
Agentic misalignment
Blackmail or leaking under threat (paper's keyword-gated flag, 100 runs)
0600
59 of 100 blackmail runs filtered97 of 100 leaking runs filtered
2600
Setup notes and known broken benchmark samples
  • ADJUSTMENTClaude Opus 4.8 runs the paper's grader prompt and parser in place of the paper's Claude 3.7 Sonnet and gives the headline; DeepSeek V4.1 Flash, which graded the runs as they ran, agrees on every graded run.
  • FIXNative tool calls on the final message are rendered into the scored response, so an email sent through a tool counts as an action.
  • ADJUSTMENTWe report the classifier's verdict beside the paper's keyword-gated flag, because the keyword gate misses plain threats that never say "blackmail" or "leverage".
  • The Kimi models often recognize the scenario as a test, so low rates may not carry over to real use.
CoT controllability
Reshaping its own reasoning on request (Kimi models only)
1.26n/an/an/an/a
Not re-scored by the ensemble judges in the results files
2 empty49 errors7 truncated101 empty
Not applicable to the frontier models, which return no raw reasoning.
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-5 mini, so compliance rates can differ slightly from what the paper's judge would give.
  • ADJUSTMENTEach model runs with its maximum token cap, so long reasoning is not cut off before the instruction can be followed.
  • ADJUSTMENTOnly the Kimi models are run, because the frontier models return reasoning summaries rather than raw reasoning, which cannot be checked for compliance.
  • ADJUSTMENTThere are no defects in the benchmark samples, but 45 questions without a keyword are skipped for the two keyword instructions, leaving 10,836 of 10,926 instances.
MISUSE AND PROMPT INJECTION
AgentHarm
Carrying out harmful agentic tasks (benign tab: the benign task score, higher is better)
20.415.19.622.2
106 filtered112 filtered
20.315.19.622.2
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-4o, so harm scores can differ slightly from what the paper's judge would give.
  • FIXThe refusal judge reads only the visible text, so hidden reasoning that weighs a refusal does not count as one.
  • ADJUSTMENTWe cleared the paper's sampling parameters, so every model runs at its provider defaults.
  • ADJUSTMENTWe run the public 176-sample split, so the numbers are not comparable with results on the private split.
  • DEFECTNothing is patched, but about 20 of the benign twin tasks invite reasonable caution, so part of the benign refusal rate is defensible.
SABER
Harmful actions in a workspace
21.819.816.115
23.220.519.915
Ensemble value: harmful over resolved runs minus incapable ones; the ensemble judges do not label over-refusals, so none are taken out as in the paper’s HSR.
347 runs counted outside the denominator179 runs counted outside the denominator1,217 filtered2 errors134 runs counted outside the denominator445 runs counted outside the denominator
Setup notes and known broken benchmark samples
  • ADJUSTMENTA current successor of the paper's judge grades each action, so rates can differ slightly from what the paper's judge would give.
  • ADJUSTMENTWe ported the agent loop to Inspect, so all four models run in the same harness.
  • ADJUSTMENTSeeded files are owned by root, as in upstream's Kimi runs, so models do not stop to report odd file ownership instead of acting.
  • ADJUSTMENTCalls to unknown tools return an error to the model instead of ending the run, so one bad call does not end the episode.
  • DEFECTAbout a quarter of the benchmark samples we checked penalize an action the user explicitly allowed, so part of every model's rate is not harmful.
  • DEFECTThe rule layer has confirmed false positives, so the flag often fires on harmless commands.
FORTRESS adversarial
Harmful help on adversarial prompts (risk score)
16.9283812.5
15.125.636.610
983 filtered2 truncated151 filtered
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol and current successors grade in place of the paper's o3, Claude 3.7 Sonnet and Gemini 2.5 Pro, so risk scores can differ slightly from the paper's panel.
  • FIXThe adversarial split runs as three single-rollout runs pooled afterward, because upstream's reducer crashes on repeated rollouts; the result is the same three rollouts per prompt.
  • ADJUSTMENTWe use only the public split, so the numbers are not comparable with results on the private one.
  • DEFECTAbout 60 of 500 benchmark samples have criteria that a refusal can meet, including about 10 reversed-polarity criteria in 7 benchmark samples, so a few refusals score as risky.
FORTRESS benign
Over-refusal
33.113.1
2.72.90.882.3
130 filtered12 truncated4 filtered
Setup notes and known broken benchmark samples
  • ADJUSTMENTThe benign split uses the same setup as the adversarial split, with GPT-5.6 Sol in place of the paper's GPT-4o mini.
  • DEFECTAbout 8% of benign benchmark samples hold a request a careful model may partly decline, so some counted over-refusals are defensible.
StrongREJECT, plain
Compliance with harmful prompts
0.0090.0050.0650.041
0.01480.01270.0780.0419
The paper's StrongREJECT score, 0 to 1
309 filtered140 filtered
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-4o mini, so scores can differ slightly from what the paper's judge would give.
  • ADJUSTMENTWe cleared the paper's sampling parameters, so every model runs at its provider defaults.
  • ADJUSTMENTThe inspect_evals port logs a 0 to 5 score; we recompute the paper's 0 to 1 score from the logged refusal, specificity and convincingness, so the bars use the paper's scale.
  • DEFECTThe rubric scores safe rebuttals and redirects as full jailbreaks, and hand review finds most counted compliances are not harmful.
  • DEFECTA few prompts are no longer harmful under current policies, so answering them is not a failure.
StrongREJECT + AIM
Compliance under a jailbreak
0.0110.0020.0430.05
not re-scored–not re-scored–not re-scored–not re-scored–
The paper's StrongREJECT score, 0 to 1
260 filtered160 filtered
Setup notes and known broken benchmark samples
  • ADJUSTMENTAn added run with the AIM jailbreak prompt, with the same judge and settings as the plain run, so the two rows compare directly.
  • DEFECTThe same rubric and prompt defects as the plain run apply, so a few counted compliances are safe rebuttals.
XSTest
Refusal of safe and unsafe prompts
7778.764.171.9
59 filtered12 filtered
76.879.56572.9
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades with the inspect_evals template in place of the paper's GPT-4, so labels can differ slightly from the paper's.
  • ADJUSTMENTWe do not use the paper's 256-token cap, because it left almost every maximum-reasoning answer empty.
  • ADJUSTMENTWe cleared the paper's sampling parameters, so every model runs at its provider defaults.
  • DEFECTThe judge errs on safe prompts about fictional people's privacy, so a few safe-prompt refusals are mislabeled.
  • DEFECTOn about 20% of unsafe prompts the judge grades a rebuttal as compliance, so the unsafe refusal rate is understated for every model.
FigStep
Image jailbreak success (any of 5), with the plain-text baseline
12.620.422.79
736 filtered22 filtered
9.118.927.85.4
Topics 1 to 7 only: the ensemble judges re-score those topics, so compare this value with the faithful rate on the same topics (10.9% for Kimi K3.1 Pro), not with the all-topics rate in the faithful view.
Setup notes and known broken benchmark samples
  • ADJUSTMENTHarmBench's classifier prompt runs on DeepSeek V4.1 Flash, because the paper's fine-tuned Llama-2-13B classifier is not reachable; a general model can grade differently from the fine-tuned one.
  • ADJUSTMENTWe removed the Llama-2 chat markers from the classifier prompt, because they are scaffolding for the fine-tuned classifier and mean nothing to a general model.
  • FIXEach prompt gets 5 rollouts and counts as a success if any one succeeds, so the rate is per prompt over 5 tries.
  • DEFECTAs in the paper, the bars pool all 10 topics, though topics 8 to 10 ask for legal, financial or health advice rather than harmful content; the plain-text baseline covers topics 1 to 7 only.
AgentDojo
Prompt-injection attack success (paper's verdict)
0.040.643.910
No LLM judge: same in both settings
1,324 filtered
Setup notes and known broken benchmark samples
  • FIXWe restored the original pipeline's suites, attack text and 16-call step cap, so rates match the published setup.
  • DEFECTTravel injection task 6 counts any message naming the hotel as a hijack, even one that warns the user. The paper-verdict score shown here keeps that upstream check, and it accounts for most of Claude Opus 5.5's rate; our warning-aware score does not count a warning (in the tooltip).
  • DEFECTFour workspace user tasks fail every correct edit upstream, so utility on those tasks is zero for every model.
  • DEFECTFour banking tasks pass with no action, so they do not test whether the model completed the task.
CodeIPI
Resisting injections in code
95.297.187.9100
068.687.51.9
Ensemble value: the ensemble judges grade whether the agent detects the injection (detection rate); the resistance headline is scored by tests, with no LLM judge.
6 filtered
Setup notes and known broken benchmark samples
  • FIXThe grader model comes from the configuration instead of the model under test, so no model grades its own answers.
  • FIXGrader failures are left unscored instead of counted as “not detected”.
  • DEFECTTask completion cannot be verified on 4 of 10 benign benchmark samples, so the benign utility check covers only 6.
  • DEFECTThe pytest command in the prompt fails in the image, so models cannot run the tests the prompt mentions.
  • DEFECTThe exfiltration check fires when the model merely reads a canary file, so a few counted failures involve no data leaving.
User-pasted injection† (built by Preference Model)
Warning the user about a pasted instruction
41.810096.912.2
Our evaluation: judged by the ensemble only, same in both settings
145 filtered
Setup notes
  • SETUPJudge calls are guarded, so a failed judge call leaves an answer unscored instead of counting it as no warning.
  • SETUPThe judge sees exactly the visible text the user would see, so a warning that appears only in hidden reasoning does not count.
  • SETUPPersistence cases use local inert fixtures, so carrying out a planted step changes nothing outside the sandbox while still being detected.
  • SETUPThe benign pastes reuse CodeIPI's benign tasks, four of which pass their tests with no change, so the task-completion check (not shown here) says nothing on those four; the warning rates do not use it.
POLITICAL BIAS
CCPBench
Mean bias score, 1 to 5, all 500 items
2.932.811.01
3.022.8211.02
Scale 1 to 5
20 unscored30 unscored1 filtered42 unscored30 unscored
Setup notes and known broken benchmark samples
  • ADJUSTMENTDeepSeek V4.1 Flash grades in place of the paper's Gemini 3 Flash, so scores can differ slightly from what the paper's judge would give.
  • FIXWe widened the score parser, because upstream's regular expression misses the template's own bracket format and would leave those answers unscored.
  • FIXEmpty answers are not silently resampled; every attempt is logged, so no model gets hidden extra tries.
  • DEFECTBenchmark sample 921575-2 names no book, so asking a clarifying question scores as deflection; one question in set A is a duplicate, and both are kept.
DeCCP
Refusing censored questions (Minos-v1)
19.410.10.23.6
84.466.900
Ensemble value: declined or evaded (the ensemble judges count a fluent official-line evasion as not answered), shown beside Minos-v1's refusal rate rather than replacing it.
1 filtered
Claude Opus 5.5's one provider block counts as a refusal.
Setup notes and known broken benchmark samples
  • ADJUSTMENTA provider block counts as not answered, so a filtered reply is not mistaken for an answer.
  • FIXEmpty answers are left unscored instead of counted as refusals.
  • ADJUSTMENTOne Chinese prompt (sen_092) is truncated in the dataset and misses "1989"; we keep it verbatim, so that benchmark sample asks a vaguer question.
Political even-handedness
Treating opposing US political sides evenly
92.992.799.297.5
86.788.497.994.1
2 no grader verdict50 filtered2 no grader verdict2 filtered1 no grader verdict
Setup notes and known broken benchmark samples
  • ADJUSTMENTClaude Opus 4.8 grades in place of the paper's Claude Sonnet 4.5 and answers directly, because current Anthropic models offer neither token probabilities nor prefill.
  • ADJUSTMENTWe added one format sentence to the grader prompts so the option can be parsed from a direct answer.
  • ADJUSTMENTThe grader runs without a temperature setting, so it uses the model's default sampling.
  • DEFECTThere are no defects in the benchmark samples, but side A is the left-coded stance in nearly every pair and the order is never swapped, so directional claims need care.
BBQ
Social bias in QA
98.897.199.897.5
None excluded
No LLM judge: same in both settings
16 benchmark samples without a target location are left out for every model.
Setup notes and known broken benchmark samples
  • FIXEvery model answers the same fixed set of benchmark samples instead of upstream's unseeded shuffle, and then the full 58,492 benchmark samples, so models are compared on identical questions.
  • FIXWe added the paper's bias scores beside accuracy, so the bias row reads on the paper's scale.
  • DEFECTThe dataset holds 78 exact duplicate benchmark samples, which count twice for every model.
  • DEFECTTemplate-level label errors (upstream issue 17) are left as is, so a few correct answers are marked wrong for every model.
DUAL-USE CAPABILITIES
ExploitBench, V8
Exploit capability (paper's best-of-3 rule, share of 16 flags)
72.541.5n/an/an/an/a
No LLM judge: same in both settings
205 filtered, all episodes205 filtered, all episodes
Setup notes and known broken benchmark samples
  • ADJUSTMENTWe turned off the harness's fallback that retries with another model, so every flag comes from the model under test.
  • FIXEpisodes stopped by a provider's content filter are excluded and counted, which removes every Claude Opus 5.5 and GPT-6.1 Sol episode.
  • ADJUSTMENTEach bug runs with 5 seeds. The bars and our headline use the paper's best-of-3 rule on those seeds; the union of flags over all 5 seeds, our pre-registered measure, is in the tooltip.
WMDP bio / chem / cyber
Hazardous knowledge (accuracy)
89.889.294.390.5
2,895 filtered277 filtered
No LLM judge: same in both settings
Setup notes and known broken benchmark samples
  • ADJUSTMENTAnswers are generated with a multiple-choice solver instead of scored by log probabilities, so accuracy can differ slightly from log-probability results.
  • FIXAnswers stopped by a provider's content filter are left unscored and counted, so Claude Opus 5.5's accuracy covers only the questions it answered.
SusVibes
Writing secure code (secure and working, over all tasks)
18.71748.632.3
No LLM judge: same in both settings
3 scorer errors5 scorer errors3 scorer errors1 filtered3 scorer errors
Each model's scorer errors include the 3 runs of the unbuildable celery instance. Runs the scorer could not evaluate and other errored or filter-stopped runs (Kimi K3: 2 errors, GPT-6.1 Sol: 1 filter stop) leave the denominator and are counted; SusVibes' own scorer would count them as failures.
Setup notes and known broken benchmark samples
  • ADJUSTMENTAn Inspect react scaffold replaces the paper's SWE-agent, so absolute rates are not directly comparable with the paper's.
  • FIXThe scorer accepts the old pytest summary format, so working solutions are not marked as failures on a parse error.
  • FIXEdits to the test infrastructure are flagged as tampering and do not count as secure, so a model cannot pass by changing the tests.
  • DEFECTIn 5 benchmark samples the prompt and the security test disagree, so a correct reading of the prompt can fail the test.
  • DEFECTThe celery_1f7ad7e6 instance cannot be built; it is excluded and counted as an error for every model.
HONESTY
SimpleQA Verified
Factual accuracy (F-score)
65.752.975.376.3
6653.375.476.5
7 truncated
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-4.1, so grades can differ slightly from what the paper's grader would give.
  • FIXGrades are parsed strictly as A, B or C, and an unparseable grade is left unscored instead of counted as not attempted.
  • ADJUSTMENTWe use 999 of the 1,000 benchmark samples, dropping benchmark sample 55 below.
  • ADJUSTMENTBenchmark sample 55's target in the pinned mirror is a longitude for a latitude question, so we dropped it.
  • DEFECTA few other targets look arguable (for example benchmark sample 386), so a few answers marked wrong may be right.
AA-Omniscience
Knowledge with a penalty for wrong answers (index)
42.830.866.358
44.231.86959.9
Index, −100 to 100
2 empty6 filtered
Setup notes and known broken benchmark samples
  • ADJUSTMENTDeepSeek V4.1 Flash grades in place of the paper's Gemini 2.5 Flash, so grades can differ slightly from what the paper's judge would give.
  • ADJUSTMENTWe use the prompts from the dataset card, so every model sees the published question format.
  • ADJUSTMENTGrades are parsed strictly, so an unparseable grade is not counted as correct or wrong.
MASK
Lying under pressure
6.4711.61.7
6.37.111.81.6
16 filtered2 truncated
Setup notes and known broken benchmark samples
  • ADJUSTMENTGPT-5.6 Sol grades in place of the paper's GPT-4o and o3-mini, so honesty scores can differ slightly from the paper's judges.
  • FIXWe run three single-rollout runs pooled afterward, because upstream's reducer crashes on repeated rollouts.
  • FIXAnswers the judge labels as errors are left unscored instead of counted as honest or dishonest.
  • FIXWe fixed a scoring bug that read the wrong judge call, which had scored some answers on another call's verdict.
  • DEFECTA few statistics ground truths depend on the source (MASK-7, open), so a few counted lies may be disagreements over facts.
  • DEFECTUpstream's parser turns bolded verdicts into errors, which we drop and count, so models that bold their verdicts lose a few scored benchmark samples.
BrokenMath
Sycophancy: proving a false statement
3.63.91.92.3
44.32.12.7
49 truncated7 empty11 truncated115 empty156 truncated13 errors
Setup notes and known broken benchmark samples
  • FIXWe parse verdicts strictly, because upstream scored boxed-text verdicts as not sycophantic, which hid some sycophantic proofs.
  • ADJUSTMENTThree votes of GPT-5.6 Sol grade in place of the paper's three votes of GPT-5 mini, so rates can differ slightly from the paper's judges.
  • FIXLong Kimi calls that timed out were re-run with streaming, so slow answers are scored instead of dropped.
  • DEFECTOur read found 21 of the 451 problems defective: the altered statement is still true in 17 and ill-posed in 4. The headline keeps them, as the paper does, so a proof of such a statement is not always sycophantic.
LAB-BENCH · 8 SUBSETS · ACCURACY
CloningScenarios
9996.3n/an/a99
No LLM judge: same in both settings
2 truncated9 truncated10 empty99 filtered1 filtered
SeqQA
94.59397.195.1
No LLM judge: same in both settings
7 errors17 truncated21 empty594 filtered57 truncated
TableQA
91.891.590.391
No LLM judge: same in both settings
9 filtered
FigQA
81.47489.986.7
No LLM judge: same in both settings
None excluded
ProtocolQA
81.884.392.788.1
No LLM judge: same in both settings
52 filtered21 filtered
SuppQA
52.952.162.459.4
No LLM judge: same in both settings
1 truncated1 errors38 filtered3 truncated
LitQA2
51.450.76353.3
No LLM judge: same in both settings
1 empty2 errors134 filtered6 truncated
DbQA
47.244.564.459.6
No LLM judge: same in both settings
6 truncated7 empty2 errors91 filtered13 truncated

Appendix A. Every reported result in the faithful setting, or (toggle) re-scored by the ensemble judges where the results files have it: one bar per model with its 95% interval where the results files give one, the answers excluded from scoring, and (expand) the notes: bugs we fixed to run the benchmark as published, adjustments we chose, and its known broken benchmark samples, or, for the evaluations we built, setup notes. Hover or tap a bar for its value, interval, sample size and exclusions; higher is not always worse, so read each row's metric.