Clinic 24

Published Result Trust

A paper claims 94% AUC on a public benchmark. Your careful reproduction gets 81%. Before you email the authors, before you assume you are wrong, before you assume the paper is wrong — read the packet and write the defense.

Back To Clinics Open Splits Topic Beyond The Academy

Situation

94% Claimed, 81% Measured

Published number is 13 points above your reproduction. The question is not who is right. The question is which single diagnostic would move your belief.

Your Job

Trust, Partial-Trust, Or Reject

Read the packet. Commit to one call before the reveal. Name the one measurement that would change your mind — and say why it is that one.

Bad Habit To Avoid

Default Deference To The Paper

"They published it, so they are probably right" is not reasoning. "I could not reproduce it, so I must have missed something" is not reasoning either.

Situation

Six months ago a group published a short paper titled "DeepScreen-v2: Robust Anomaly Detection On SynthBench". The abstract, paraphrased:

We propose DeepScreen-v2, a two-stage transformer encoder trained with a class-balanced sampling regime. On the public benchmark synthbench-v1, DeepScreen-v2 achieves a test AUC of 0.942, outperforming the previous best reported result of 0.889 by a substantial margin. Code and weights are released.

You are evaluating whether to anchor your own work on this result. You cloned the code, downloaded the benchmark, and ran the released training script with default hyperparameters. Five seeds. Median test AUC: 0.813. Your reproduction is 13 points below the headline.

You are not preparing to email the authors yet. You are deciding, right now, whether to trust the published number enough to build on it.

Artifact Packet

Read every row before you write the defense.

Benchmark shape.

split	examples	class balance
synthbench-v1.train	2000	92% normal, 8% anomaly
synthbench-v1.val	200	92% normal, 8% anomaly
synthbench-v1.test	80	92% normal, 8% anomaly

The test set has 80 examples. Roughly 6 of them are anomalies.

Your five-seed reproduction of the released training script.

seed	test AUC
0	0.791
1	0.808
2	0.813
3	0.824
4	0.829

Median: 0.813. Range: 0.038. Paper number: 0.942. Gap from median to paper: 0.129. Gap from your best seed to paper: 0.113.

The class-balanced sampling detail.

The paper says "trained with a class-balanced sampling regime." The released code does not implement this. The training loop uses a standard RandomSampler. The authors' GitHub has two open issues asking about this discrepancy. Neither has been answered.

You tried adding class-balanced sampling yourself (upsample the minority class at batch time, 50/50 per batch). Five seeds with the added balancing:

seed	test AUC (balanced)
0	0.901
1	0.919
2	0.921
3	0.932
4	0.938

Median: 0.921. Paper number: 0.942. Gap: 0.021.

One more detail. The synthbench-v1 README includes this line under "Evaluation Rules":

The training distribution must be preserved during model training. Augmentation that alters class priors is not permitted for benchmark submission.

You flagged this to yourself while reading the repository. You are not sure whether class-balanced sampling counts as "alters class priors" under this rule.

Decision Prompt

Write a six-sentence defense that answers:

1. Do you trust, partially trust, or reject the published 0.942 result, and in one sentence, why?
2. What is the 95% confidence interval you would put on a test AUC measured from 80 examples with ~6 anomalies? (Rough answer is fine — this is a size judgment, not a calculation.)
3. Does the class-balanced sampling detail explain the gap on its own? How does your evidence support that answer?
4. Does the benchmark rule about preserving the training distribution force a different reading of the paper's number?
5. What is the one measurement you would ask for that would move your call from your current level of trust to a different one?
6. Would you anchor a week of your own work on this paper's result, this week, with what you now know?

Strong Reasoning Looks Like

• it notes that 80 test examples with ~6 anomalies gives a test AUC with a confidence interval on the order of ±0.05 to ±0.08 — the 13-point gap is not "noise," but the reported precision (0.942 vs. 0.889) was already inside an interval wider than the claimed margin
• it identifies that the class-balanced sampling detail quantitatively explains the gap: median without balancing is 0.813, median with balancing is 0.921, paper is 0.942 — the residual 0.021 is well within per-seed variance
• it flags that the released code and the published method do not match, and that this is not a reproducibility edge case — it is the paper's core method omission
• it reads the benchmark rule ("training distribution must be preserved") as at least ambiguous about class-balanced sampling, and resolves the ambiguity before citing the number
• it refuses the two easy postures: "trust because it is published" and "reject because my number is lower" — both skip the work
• it commits a single diagnostic as the move: either "ask the authors whether class-balanced sampling was the trick", or "email the benchmark maintainer to clarify the rule", or "accept 0.92 as the honest number and publish it with the method explicitly"
• it decides that anchoring a full week of downstream work on 0.942 is premature; anchoring on 0.81–0.92 with an explicit note about the sampling detail is fine

Common Wrong Moves

• treating the 80-example test set as if it were 8000 — a reported test AUC of 0.942 is not precise to three digits at that size
• running one seed of your reproduction, getting 0.81, and declaring the paper wrong — you have no variance estimate, and the paper's one number has no variance estimate either
• running the balanced version, getting 0.92, and declaring the paper right — you have changed the method, and maybe broken the benchmark rule in the process
• treating "code is released" as equivalent to "method is reproducible" — the released code is not the published method in this case
• emailing the authors with "your paper does not reproduce" instead of "your released code does not implement the class-balanced sampling described in section X — is the sampling code available somewhere else?"
• quietly adopting the balanced version for your own pipeline without resolving whether it is benchmark-legal — you will be in the same ambiguous position next month
• ignoring that "previous best 0.889" and "DeepScreen-v2 0.942" are both numbers measured on 80 examples — even the field's "previous best" is inside its own confidence interval

Sketch The Confidence Interval Yourself

Before reading the reveal, pick the bootstrap CI you would actually trust. The snippet below resamples the 80-example test set 2,000 times under the paper's prevalence (8 positives, 72 negatives) to give an order-of-magnitude width for the AUC interval.

import numpy as np
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n_pos, n_neg = 8, 72                 # paper's reported test split
y = np.concatenate([np.ones(n_pos), np.zeros(n_neg)])

# stylized score gap consistent with the paper's claimed AUC ≈ 0.94
scores = np.concatenate([rng.normal(2.0, 1.0, n_pos),
                         rng.normal(0.0, 1.0, n_neg)])
print("point estimate:", round(roc_auc_score(y, scores), 3))

aucs = []
for _ in range(2000):
    idx = rng.integers(0, len(y), len(y))
    if y[idx].sum() in (0, len(y)):  # bootstrap can produce all-positive / all-negative samples
        continue
    aucs.append(roc_auc_score(y[idx], scores[idx]))
lo, hi = np.percentile(aucs, [2.5, 97.5])
print(f"bootstrap 95% CI: [{lo:.3f}, {hi:.3f}]  (width ≈ {hi - lo:.2f})")

Run it. The CI width is wide enough to swallow the gap between "previous best 0.889" and "DeepScreen-v2 0.942" — that is the load-bearing fact this clinic exists to teach.

Reference Reveal

Open only after you write the defense

The reference call is **partial trust**. Neither "the paper is right" nor "the paper is wrong" fits the evidence. Why partial, not full trust: - the test set has 80 examples with about 6 anomalies. An AUC measured on that size has a rough confidence interval on the order of ±0.05 to ±0.08 (the exact interval depends on the score distribution, but the *size order* is what matters). The paper reports 0.942 with implied three-decimal precision. That precision is not supported by the measurement. The previous best of 0.889 is inside the same interval, so the field's claimed improvement was already fragile. - the published method says "class-balanced sampling." The released code does not implement it. With class-balanced sampling added locally, the median jumps from 0.813 to 0.921 — that is 85% of the gap to the paper's number, in a single change. This is not a mysterious reproducibility gap. It is a specific missing component. - the residual 0.02 between your balanced median (0.921) and the paper's number (0.942) is well inside the per-seed variance visible in your five-seed run. There is no need to invoke a second mystery to explain it. Why not full trust: - the benchmark evaluation rule says "training distribution must be preserved during model training. Augmentation that alters class priors is not permitted." Class-balanced sampling plausibly falls under this rule. If it does, the paper's 0.942 was obtained under conditions the benchmark forbids, and the number is not comparable to prior results. If it does not, the paper's 0.942 is still real — but the *method* is underspecified in the published code, which is a separate problem. - until that rule-interpretation is resolved, the correct anchor for your downstream work is **0.81 with the released code** or **0.92 with explicit balanced sampling and a footnote**. Not 0.942. Why not full rejection: - the paper's method idea (class-balanced sampling on a heavily imbalanced benchmark) is pedagogically clear and quantitatively effective. It is not fabricated — your own balanced reproduction reaches the same neighborhood. - the problem is not "the result is fake." The problem is "the released artifact does not match the published description" and "the test set is too small for the claimed precision." Both are fixable. The diagnostic that would move your call: - from the authors: did you use class-balanced sampling, yes or no, and does it comply with the benchmark's distribution-preservation rule? - from the benchmark maintainer: does class-balanced sampling at training time count as altering class priors for the purpose of submission? One answer from each flips this from "partial trust" into "trust with a caveat" or "reject with a counter-proposal." Neither answer requires more compute. Both require a short email. The practical lesson: **"I could not reproduce the paper" is not yet a conclusion**. It is a prompt to find the specific missing component. Often the component is stated in the paper and missing from the code. Sometimes the component breaks a benchmark rule. Sometimes the test set is too small for the gap to be meaningful. The clinic ends when you name which of these three is the case, not when you pick a number to quote.

What To Do Next

1. open Honest Splits and Baselines — for the underlying discipline on test-set size and variance
2. open Evaluation Metrics Deep Dive — for AUC confidence interval intuition on small test sets
3. open Public/Private Restraint — the sibling clinic on trusting a visible number when the evidence is thin
4. draft the two-line email to the authors using only the evidence in your own packet
5. if you are about to cite a published number in your own work, write down the test-set size and the variance estimate first. If either is missing, the citation is not yet ready.