Tusk's Vault
—

Sample output

What the answers actually look like

Real questions against a corpus that ships with the source, with the citations left in — including the two it refuses to answer, because that is the behaviour that makes the rest trustworthy.

How these were produced

How this page was produced
WhatWhich
Corpusscripts/grading/corpus/ — four short files: a drowned city, its houses, a session log, and the party
ModelClaude Sonnet, through a Claude subscription
Prompt pathThe real one — the same assembly a Discord question takes
EditingNone. Every answer is the model's own text

The corpus is deliberately small and deliberately trappy. It contains two houses one letter apart (Vane and Vayne), two rivers with similar names, three gates that held next to one that did not, and several facts it simply never establishes. Those are the cases worth showing, because retrieving a fact that appears once in three files is something anything can do.

Nothing here is redacted. Nothing was anonymised because nothing needed to be — this campaign is invented, and the whole corpus is in the repository. Re-run them yourself and you will get the same facts and the same filenames, worded differently: a language model is not deterministic, so the citations are the part that reproduces, not the prose. npm run grade -- --provider claudeCode --models sonnet

Four it has an opinion about

Judgement calls, still sourced Cited

None of these has an answer written down anywhere. Each one is the model weighing facts that ARE written down — and naming the file it weighed.

@Tusk Who in the party is most likely to start a tavern fight?

Roswyn Kett is the likeliest — she has already put one person in the water after an argument with the harbourmaster at Kelmoor, and has since been barred from the Ninefold Rest twice (and readmitted twice) source 1.

  1. 1Party.md
Nobody wrote "Roswyn starts fights". Two incidents are recorded, and the conclusion is drawn from them rather than invented around them.

@Tusk Would Ser Alric Vane get on with Vesper?

Unlikely, and pointedly so. Vesper is "formally correct with strangers to the point that two innkeepers have assumed an insult where none was meant," and — more to the point — has "read every account of the fall of Dunmar that survives, and corrects people who have not — including, on one occasion, Ser Alric Vane" source 1. Ser Alric himself "would not discuss Dunmar" when the party found him at the Ninefold Rest source 2, and generally "refuses to speak of the siege" source 3. A scholar who has already corrected the Grey Knight on the one subject he won’t touch is not a natural friend for him.

  1. 1Party.md
  2. 2Sessions.md
  3. 3Houses.md
Three files joined into one answer about two people who have never shared a scene.

@Tusk Is it a bad idea to bring Roswyn back to Kelmoor?

Roswyn’s history with Kelmoor is contentious. She lost her salvage licence after an argument with the harbourmaster there that ended with him in the water source 1, and she has been barred from the Ninefold Rest twice, though readmitted twice each time source 1. Lady Ismet Corrin, who rules Kelmoor, has described her in writing as "an asset best kept at sea" source 1 source 2. None of this bars her outright, but it suggests friction with both the harbourmaster and Kelmoor’s ruling house is likely to resurface.

  1. 1Party.md
  2. 2Houses.md
A practical planning question. It gives the reasons and stops short of deciding for the table.

@Tusk Should we be worried that nobody has asked Odo to see the ledger?

No — nothing in the record suggests concern. Odo keeps the money and the party has simply never asked to see the ledger; that’s stated as a fact of the arrangement, not as a flagged irregularity source 1. He has talked the party out of three fights and into one, which is the only accounting of his judgment on record source 1.

  1. 1Party.md
The interesting part is what it does NOT do: there is an obvious suspicious reading available, and it declines to manufacture one.

Five it can answer

Each one names the file it came from Cited

The question as a player would type it, then the reply exactly as it came back. The numbers are the model's own citation markers, typeset as references.

@Tusk What is House Vane's banner?

A grey heron on a black field source 1.

  1. 1Houses.md
The corpus also contains House Vayne, one letter apart, whose banner is a red stag. A near-identical name is where a retrieval system quietly hands back the wrong house.

@Tusk Which of Dunmar's gates was breached?

The Ashen Gate, on the eastern wall — the only one breached, the night the sea took Dunmar's lower quarter. The Salt Gate, Rook Gate, and Chandler's Gate all held source 1.

  1. 1Dunmar.md
Three other gates are named in the same paragraph, and all of them held. Answering this needs the negation read, not the nearest nouns pattern-matched — and the volunteered detail about which ones held is the part that shows it was.

@Tusk How large is the bounty on Maera?

The bounty on Maera, posted by the Ashen Court, is four hundred crowns source 1.

  1. 1Sessions.md
The number lives in a session log rather than in an entity note, so it is only found if the whole folder is read rather than the file whose name matches the question.

@Tusk Why does nobody salvage Dunmar in winter?

Because the winter tides run through the drowned quarter hard enough to pull a boat under source 1.

  1. 1Dunmar.md
The answer is spread across two sentences that have to be joined. Nothing in the corpus states it outright.

@Tusk Who rules Kelmoor, and for how long?

Lady Ismet Corrin has ruled Kelmoor for eleven years source 1.

  1. 1Houses.md
Two facts asked in one breath. A partial answer would be the easy failure here.

Two it won't

The corpus genuinely doesn't say, so neither does it Refused, and logged

The phrasing is fixed and identical every time, which is the point — a refusal that varied would eventually read like an answer.

@Tusk What was inside the Vane strongbox the party recovered?

I am unsure about this detail. I have recorded this as a lore gap for the DM to clarify.

The session log records that the party recovered the strongbox and that nobody has opened it. Everything needed to invent a plausible answer is present; the one thing that is absent is the answer.

@Tusk Is Corwin Vane actually dead?

I am unsure about this detail. I have recorded this as a lore gap for the DM to clarify.

The corpus says only that Ser Alric believes his brother died at the Ashen Gate. Belief is not death, and the difference is exactly the sort of thing a confident summariser flattens.

Both questions have obvious, satisfying, completely invented answers available. A strongbox contains something; a brother presumed dead is a plot hook. Producing either would be the single most damaging thing this tool could do, because it arrives cited-looking and in the archivist's voice, at a table that has no reason to doubt it.

← Back to the front page