Test what a discovery search missed
For discovery reviewers and counsel evaluating a document search.
If a search returns many relevant documents, how can reviewers tell what it left behind?
Test a set that contains documents the query did not retrieve. Count relevant misses as well as relevant hits. Precision describes the quality of the returned set. Recall asks how much of the relevant material the search found.
Write the relevance decision before running the search
Fix the request, sources, date limits and unit of review. An email, its attachments and its full conversation can produce three different counts. Write down which unit you judge and how you handle repeats. Give reviewers a positive example, a near miss and a rule for recording uncertainty.
Save the search exactly as run, including field names, phrase matching, time zone and exclusions. Keep a short list of documents you already know should appear as an early warning. Finding all of those documents is a useful diagnostic, but they were selected because someone already knew about them.
Read material on both sides of the cutoff
Randomly select records from the hits and from the much larger pool of non-hits. A reviewer can judge mixed samples without seeing whether each record was retrieved. Resolve disagreements under the written relevance rule. Keep the sample sizes, the pool sizes and unresolved calls.
If you sampled different proportions of the two pools, scale each pool’s sample rate to its own population before combining estimates. Report uncertainty and the sampling design. A small sample with zero responsive misses leaves uncertainty about the unreviewed pool. It does not establish perfect recall.
Use each miss to choose the next test
Classify missed documents by likely cause. The team may have used a project nickname, scanned a handwritten attachment without searchable text or stored the decision in a field the query skipped. Change one feature, save a new version and test on fresh material. Keep the first result so the improvement can be checked.
Worked example
Illustrative calculation using a fully judged teaching set. Of 200 documents, reviewers marked 60 relevant. A query returned 50, of which 42 were relevant. Precision is 42 ÷ 50 = 84%. Recall is 42 ÷ 60 = 70%. The 18 relevant documents outside the results are available for error analysis. Those exact percentages belong to this teaching set. A partially reviewed collection would require estimates.
Primary sources
Related work at Agentic Attorney
This search exercise belongs to the shared Agentic Attorney resource collection. The main site describes AI consulting and implementation for legal teams. Explore Agentic Attorney’s approach to AI implementation.