DetectiveBench

Introduction

A few of us here at Camulos are boardgame fans.¹ Hence when the opportunity arose we jumped at the chance to understand if AI could solve Sherlock Holmes: Consulting Detective (SH:CD) cases as well as a human could.

Inspired by CivBench, which evaluates how well agents can play Civ VI, we thought we would convert our investigations into a benchmark within Inspect.

The game turns out to be a very natural fit for a benchmark. Each case is a self-contained, long-horizon investigation: a lengthy rule book, a newspaper and a directory of Londoners to wade through, hundreds of possible addresses to visit, and a set of questions at the end that you don’t get to see until you’ve finished investigating. There is an official solution to mark against, and a built-in yardstick in Holmes himself, who always scores 100.

It also rewards something most benchmarks ignore: knowing when to stop. Every lead you follow beyond what Holmes needed costs you points, so a detective who reads everything is punished almost as harshly as one who guesses. That matters to us beyond the fun of it. A lot of real analytical work looks like this: sifting a large, mostly irrelevant pile of material for the handful of facts that matter, with limited time and resources, often without knowing in advance exactly what you’re looking for. An agent that can’t decide when it knows enough is expensive at best.

In this post we introduce the game, describe how we built the benchmark (and how we stop the models from peeking at the answers), look at how a range of models fare, and then describe an agentic harness we built to help the weaker ones.

Sherlock Holmes: Consulting Detective

Sherlock Holmes: Consulting Detective is a cooperative deduction game first published in 1981. Players take on the role of the Baker Street Irregulars: a gang of street children who assist the legendary detective.² Each case presents a mystery set in Victorian London, and players must solve it by following leads across a map of the city.

The game materials include:

• A map of central London with numbered locations across multiple districts
• A large London Directory listing residents and businesses with their addresses
• A daily newspaper (The Times) containing advertisements and news: some of which are relevant to the case
• Informant cards with specialised knowledge (police records, criminology analysis, underworld contacts)
• Case booklets containing the introduction, leads, the questions to answer, and the solution

Image of all the components. Image taken from BoardGameGeek.

If you are interested in trying the game yourself, a simplified demo case is provided online and can be found here.

Game mechanics

Players begin by reading the case introduction, which sets the scene and hints at the mystery ahead. They then follow leads: each lead corresponds to a location on the map (identified by a number and area code, e.g., 86 SW for the Colonial Institute in the demo case) or a directory entry. Each lead reveals narrative text containing clues, character dialogue, and sometimes red herrings. The scoring system is simple: players answer a set of questions (which are not known during the investigation) at the end, earning points for correct answers. However, they lose 5 points for every lead followed beyond the number Holmes needed, and gain 5 for every lead fewer. Sherlock Holmes himself always scores 100. To outperform him, players must either answer the core questions correctly while using fewer leads than he did, or correctly answer some of the extra questions that Holmes didn’t address without using too many extra leads.

Why this game is an informative test of reasoning

SH:CD is well suited to evaluating AI reasoning because it demands several distinct skills at once:

1. Semantic understanding: reading and comprehending rich narrative text, distinguishing relevant clues from irrelevant detail
2. Inductive reasoning: building hypotheses from partial information, recognising patterns across disparate leads
3. Quantitative reasoning: managing a limited lead budget and weighing the cost of each lead against what it might reveal
4. Spatial reasoning: navigating a map, understanding geographic relationships, and following descriptions of physical clues

No single capability is sufficient. Crucially, the score punishes two quite different failures: getting the answers wrong, and investigating wastefully. As we’ll see, these come apart in interesting ways.

DetectiveBench

To measure this properly we built DetectiveBench, an Inspect evaluation over the ten cases of The Thames Murders.³

Inspect is the UK AI Security Institute’s open-source framework for evaluating large language models. It gives you a standard way to describe an evaluation (a dataset of samples, a solver that has a go at each one, and a scorer that marks the result) and handles the plumbing around it: talking to different model providers, repeating attempts, enforcing limits on tokens and time, and logging every message and tool call so that runs can be picked apart afterwards. That last point proved invaluable whenever we needed to work out exactly where a model had wandered off into the fog.

In DetectiveBench, each sample is one case. The solver is Inspect’s built-in ReAct agent, which simply loops between the model and its tools until the model decides it is done, or runs out of road: each game is capped at 400 messages, 20 million tokens and three hours of working time (the time was only relevant for locally-running models, evals on remote models were much faster). When a quarter of that budget remains the model is told to wrap up and answer, so that slow models don’t lose everything to the clock. Each case is played several times, since the same model can crack a case in one game and flounder in the next. And because Inspect works with any provider, exactly the same eval runs against a local model on a laptop and against hosted frontier models. The only requirement is that the model can read images, as several cases can’t be solved without the map.

Game Server

To enforce the rules, the model has no direct file access. Instead, a small MCP server serves game material at the appropriate time through a handful of tools: start a new game, read the introduction and newspaper, view the map, search the directory, open a lead, fetch the questions (which seals the investigation) and submit answers (once). Each model also gets a to-do list to organise its investigation, and nothing else: no file system, no shell and no web search.

Opening a lead works just as it does at the table. In the board game you spot something promising (a name in the newspaper, say), look it up in the London Directory to find an address such as 86 SW, then turn to that entry in the case booklet and read it aloud. The model does the same: it searches the directory, finds the address and calls open_lead("SW86"). The server returns the booklet text for that address, if there is any. Visiting an address that has nothing to do with the case gets a polite “you find nothing of relevance” and, as at the table, isn’t counted as a lead. Going back to re-read a lead you’ve already opened is free, so models aren’t punished for a poor memory.

The rest of the material is served in the same spirit. The rules, introduction, newspaper and directory are long (the rule book alone runs to around 60,000 characters), so they are handed out a page at a time and the model has to read them itself; the server never summarises or filters anything on its behalf. The map comes back as an image, with the option to zoom into any of nine squares for street-level detail. When the model is ready, it asks for the questions, and the investigation is sealed: just as in the game, you can’t read the questions and then nip back out for another look. Finally it submits its answers, once. During the benchmark the submission is only recorded, not marked, so the model can’t learn anything from its score.

Behind the scenes, every game gets its own fresh copy of the server, so nothing leaks between games. The server keeps its own record of which leads were opened and what was submitted, and it is this record, not anything the model claims in its final message, that gets scored.

The DetectiveBench setup. The model only ever sees the case through the game server’s tools; the scorer works from the server’s own record of the game.

Scoring

Answers are marked by an examiner model: qwen3.8-27b, running locally. Whichever model is playing, the examiner stays the same, so every model’s answers are marked consistently. That does mean Qwen marks its own homework when it’s the one playing, but the examiner works in an entirely separate context from the player and isn’t told which model wrote the answers in front of it. Each case comes with an official scoring guide that splits every question into criteria worth a fixed number of points, so a question might be worth 0, 15 or 30 points depending on how much of it you get right. There is no half-credit for an answer that merely sounds like the right one.

The examiner marks each question separately, so a strong answer to one question can’t drag a weak answer to another along with it. Whenever it wants to give points for a criterion (we call this an award), it must quote both the part of the answer it is relying on and the part of the official solution it matches, and the server checks that both quotes really exist. Each award then gets a second, independent check: a fresh request reads the examiner’s evidence and reasoning, without being told its verdict, and confirms that the reasoning actually supports giving the points. This catches the examiner explaining at length why an answer is wrong and then marking it correct anyway, a habit we ran into more than once while building the scorer. Finally, the server, not the examiner, adds up the points and computes the final game score:

$$\text{final score} = \text{answer points} - 5 \times \left(\text{leads opened} - \text{Holmes's leads}\right)$$

Holmes’s lead count varies from case to case, from two to eight leads across the ten Thames cases.

Played vs zero-shot

Every case is run in two conditions:

• Played: the model investigates the case with the tools above, then answers.
• Zero-shot: the model sees only the questions, with no case material at all, and answers from guesswork or memory.

Zero-shot is a contamination control (has the model memorised the solutions?) and a floor. Because it opens no leads, a zero-shot run collects the full “fewer leads than Holmes” bonus, so a model that does nothing at all still scores 20 points on Case 1, and about 26 on average across the ten cases. Our headline metric, Investigation Gain, is played minus zero-shot: how much the investigation was actually worth.

Results

We began with qwen3.8-27b, which we’d informally assessed as the most capable model that runs well on a MacBook Pro M3 Max.

Qwen3.8-27B is an open-weight dense model, which runs well on the MacBook Pro M3 Max’s 128GB unified memory in 8-bit quantisation

Across all ten cases (three played games each) Qwen averaged -59. Its dominant failure was over-investigation: it opened 23 leads per case on average, far more than Holmes, and each surplus lead cost 5 points. It also earned only about a fifth (22%) of the available answer points, and rarely backed its answers with evidence. For comparison, answering the questions cold, without opening a single lead, Qwen averaged 34. Its Investigation Gain was therefore -93: on average, investigating cost Qwen 93 points compared with not investigating at all.

That raised an obvious question: was this just a small model out of its depth? To find out, we pointed exactly the same benchmark at three tiers of Anthropic’s Claude models: Haiku 5.5, Sonnet 5.5 and, at the frontier, Fable 5.1.

Model	Played Games	Mean Score	Range	Answer Points	Leads Opened	Cost per Game	Zero-shot Score	Investigation Gain
qwen3.8-27b	30	-59	-140 to 15	22%	23.0	$0.00	34	-93
Fable 5.1	30	48	-20 to 110	78%	16.8	$2.29	39	+8
Sonnet 5.5	30	27	-25 to 90	72%	19.3	$0.47	34	-7
Haiku 5.5	30	-44	-95 to 45	42%	25.1	$0.06	34	-77

Scores, lead counts and costs are means across played games, which is why they aren’t always multiples of five; every individual game’s score is. Answer points are the share of the available answer points earned.

Bigger models do better, but perhaps not in the way you’d expect. Fable comes closest to Holmes, averaging 48, but even it opens 17 leads per case where Holmes needed about 5. Sonnet earns 72% of the answer points, yet loses 70 points to the lead penalty, and Haiku manages to over-investigate and get many of its answers wrong (much like Qwen). Further up the scale, models get better at reaching the right answers, but not much better at knowing when they’ve found them: they’re better readers rather than better investigators.

The zero-shot column makes the point starkly. Answering cold, every model scores between 34 and 39, mostly from the bonus for opening no leads. Only Fable beats that by investigating, and only by 8 points; Sonnet’s investigation is worth slightly less than not investigating at all.

That’s an interesting result for anyone hoping to wait for the next model to solve the problem. Every tier we tested struggles with the same thing, and it’s less about raw capability than about how the model goes about the task. Which is exactly the kind of thing a harness should be able to help with.

An Investigative Agentic Harness

Given the mixed results on the benchmark, we asked if it was possible to design an agentic harness that allows the models to excel at the task. Thus we built task-specific assistance, delivered as three pieces:

• A gameplay skill: a written procedure for the investigation. Set up, draft the questions you expect to be asked, estimate your confidence, critique each proposed lead before opening it, stop when confident, then answer.
• A lead-budget MCP server: tools that normalise the model’s candidate questions, combine its per-question confidence into a lead-budget estimate and a stop/continue recommendation, and route proposed leads and draft answers to a critic.
• An LLM critic: a separate, small, hosted model (Gemini 3.5 Flash-Lite) that reviews the quoted evidence behind each proposed lead and, at the end, the coherence of the final answers. It sees only what the player has seen.

A protocol layer around the tools checks that the model follows the procedure (for example, that each lead was critiqued, and that the evidence quoted to the critic actually appears in material the model has read), using only public tool responses.

Model	Played Games	Mean Score	Range	Answer Points	Leads Opened	Cost per Game	Zero-shot Score	Investigation Gain
qwen3.8-27b	30	-15	-85 to 35	10%	10.8	$0.00	32	-46
Fable 5.1	30	56	-20 to 120	49%	7.3	$3.49	37	+19
Sonnet 5.5	30	32	-35 to 75	44%	10.6	$0.80	31	+1
Haiku 5.5	30	-3	-30 to 50	11%	8.7	$0.11	34	-38

To see what the harness actually changed, we compared each model’s assisted games with its unassisted games case by case:

Model	Score Change	95% Interval	Answer Points Change	Leads Change
qwen3.8-27b	+45	+26 to +63	-12 points	-12.2
Fable 5.1	+9	-11 to +27	-29 points	-9.5
Sonnet 5.5	+5	-10 to +20	-28 points	-8.7
Haiku 5.5	+40	+23 to +57	-31 points	-16.4

Assisted minus unassisted. Intervals come from resampling cases and games; an interval that includes zero means we can’t rule out that the harness made no difference.

The harness helps most where it was needed most. Qwen and Haiku each gain over 40 points⁴, well clear of the noise: with assistance, Haiku closes 57% of the distance to unassisted Sonnet, at around a quarter of the cost per game. Most of that gain comes from where we hoped it would: lead discipline. Assisted models opened about 9 leads per case on average, against about 21 without help, and in a game that charges 5 points per lead that adds up quickly.

But the harness overcorrects. Every model’s answers got worse with assistance, by around 30 percentage points for the three Claude models: having learned to stop, they now stop too early. Unassisted, Fable fully solved 15 of its 30 games; assisted, just 3. For the weaker models the lead penalty was so large that cutting it is a clear net win, but for Sonnet and Fable the leads saved and the answers lost roughly cancel out, and neither gain can be told apart from no effect. Nor is the harness cheap at the top of the range: following the procedure takes far more of the model’s own tokens, so assisted Fable costs about 50% more per game.

We still count that as a useful result. It shows that lead discipline can be taught: a written procedure, a few tools and a small critic cut over-investigation by more than half for every model we tested. What we’ve got wrong is the stopping rule, which is too eager, and that’s a much more tractable problem than “make the model a better detective”. Tellingly, the harness’s explicit “stop now” signal, which fires once the model’s own confidence estimates are high enough, went to Fable far more often than to any other model: 37 times, against 14 for Qwen and none at all for Sonnet or Haiku, which cut their leads through the rest of the procedure. Fable also lost more fully solved games than any other model: it felt confident enough to stop before it had the full picture.

We would also be interested to hear from people who have their own harness for this problem and want to see how it does!

Conclusion

Frontier models are the best readers we tested, but even they don’t know when to stop: Fable opens three times as many leads as Holmes and only just beats answering cold. Mid-tier and small models aren’t bad readers either, but over-investigation swamps everything else. In our full runs, the right tooling fixes that cheaply for small models, recovering 57% of Haiku’s gap to the next tier.

Unfortunately, it overcorrects, trading too many leads for too few correct answers, and for the strongest models the two roughly cancel out. Calibrating when to stop is the next problem to solve, and it’s where deep knowledge of both the problem space and the model’s capability gaps matters most.

As well as continuing to track how new frontier models perform on our benchmark, the big open question is can a better agentic harness help to close the gap. Reach out if you are interested in collaborating in this space.

Footnotes

1. Feel free to drop us an email to talk about boardgames and how they can be used to model real world issues. ↩

2. This is true for the demo case, in other sets you play as other characters. ↩

3. We won’t be releasing the full benchmark at this stage due to the SH:CD cases not being ours to release. If you want us to run a model through the benchmark please shout. If you are from Space Cowboys and want to give us permission to release the benchmark, even better! ↩

4. Running locally, the extra procedure takes Qwen a long time: 13 of its 30 assisted games hit a limit (ten the three-hour time limit, three the message limit), against 2 of 30 without assistance. Those games still count, scored as they stood when the limit was reached. ↩