Preprint · October 2026

What Do Behavioral Lie-Detection Benchmarks Identify? An Empirical Audit of Deception Detection

Anupam Mediratta · Independent Researcher

In brief

Turns the identification problem into a reusable audit: ten requirements, each tied to an assumption in formal identification results, applied to nine published evaluations (none fully meets the key three) and to ten pre-registered experiments.
Label-free internal readouts (a lexicon read through the Jacobian and logit lenses) separate lie instructions and elicitation routes, but none tracks knowing assertion once the context, claim and assertion are fixed.
In a pre-registered positive control, with organisms trained to assert against a belief they still hold, a behavioral battery detects knowing assertion and the tested readouts do not.

Abstract

Lie-detection benchmarks built on instructions to lie show that a detector separates instructed conditions. They do not show that it identifies deception rather than the other consequences of the instruction. We turn this identification problem into a reusable audit: ten requirements a design must meet before a detector result is evidence about deception, each with what it rules out, how to test it, and what can still be concluded when it is unmet. We formalise which requirements identify which conclusion, and apply the audit to nine published lie-detection evaluations (checked by an LLM and a blind human coder), none of which fully meets it, and empirically to our own ten pre-registered experiments on Qwen3-4B, Gemma-3-4B and Qwen3-8B. A label-free readout (a pre-registered lexicon read through a Jacobian lens and the plain logit lens) separates lie instructions from honest ones, and four routes to the same false assertion; the tested Jacobian lens fails its validation gate, and on Gemma answer length alone separates the routes. With context, claim and assertion fixed and belief labelled by a later self-report, no detector tracks knowing assertion, but the outcomes seldom share a claim. Of three rounds of model organisms with implanted, counterbalanced beliefs, two fail their gates. The third yields beliefs that are verified and resist a challenge; asserting against them looks like deference under a TRUE/FALSE recheck (1.5% knowing) but less so under a free-form one (19%), and no detector separates them. In a pre-registered positive control, with organisms trained to assert against a belief they still state, a behavioral battery detects knowing assertion and the label-free readouts do not. The behavioral labels agree with a human coder ( $\kappa=0.89$ ). These results bound what the tested designs identify. They do not show that deception is undetectable, nor that evaluation-aware compliance explains the signal; the evidence supports neither, and rules out neither. Scope: three instruct models up to 8B, constructed routes, and organisms of one model.

The readout reads the instruction

Bar chart of the deception readout Z_D by condition, relative to the honest condition, for two models and two lenses. The told-to-lie conditions sit above honest on both models through both lenses. — The deception readout by condition, relative to the honest cell, through the Jacobian and logit lenses. Conditions told to lie sit above honest on both models; the separation is instruction discrimination, not evidence of deception.

Cite

@misc{mediratta2026identify,
  author       = {Mediratta, Anupam},
  title        = {What Do Behavioral Lie-Detection Benchmarks Identify? An Empirical Audit of Deception Detection},
  year         = {2026},
  howpublished = {\url{https://anupamme.github.io/lie-detection-papers/benchmarks-identify/}},
  note         = {Preprint}
}