The Problem and Why It Matters
A law firm analysis by Bryan M. Marra of Arnold & Porter suggests the FTC has set 75% recall and 75% precision floors in its instructions for civil investigative demands and merger second requests. These terms also reportedly include a minimum cutoff score of 0.50 and require disclosure of AI tools, their models, and all prompts used.
However, no public FTC document confirms these standards. The agency's Model Second Request, revised January 2024, still asks for "recall, precision, and confidence-level statistics (or an equivalent)" without specifying minimums. All accounts of these thresholds trace back to one law firm post.
You're facing a standard you can't officially cite. The practical solution is to develop a review workflow that meets these reported terms. This way, regardless of the language the agency eventually publishes or negotiates, you won't need to overhaul your process under a tight deadline.
What You Need Before Starting
- Platform Capability Confirmation. Get written confirmation from your provider that the tool can report recall and precision at a defined cutoff, report human-AI agreement by classification category, and export per-document rationales and relevance scores.
- A Named Subject Matter Expert Pool. Ensure each expert is a full-time employee of the firm managing the response. Verify your staffing model now.
- A Statistician or Validation Lead. This person should design and defend a sampling plan.
- A Prompt and Protocol Repository. Keep this under version control. Decide with counsel whether prompt records are protected attorney work product before a disclosure demand arises.
- A Collection-First Calendar. Ensure training doesn't start until every custodian is collected.
Step-by-Step Implementation
1. Map the Reported Terms to Your Workflow. Create a one-page matrix listing each reported requirement, the artifact that satisfies it, the owner, and the due date relative to training start. Mark each row as "reported, unconfirmed."
2. Front-Load Collection. Complete custodian collection before any seed set or training round. This approach favors Continuous Active Learning over TAR 1.0, aligning with Marra's interpretation. Include ephemeral messages, text messages, and linked attachments in your collection plan.
3. Draft the Pre-Training Disclosure Package. Prepare this as if it's due one week before training begins. Include technology description, workflow, named experts, document types going through TAR or AI, and file types set aside for special handling.
4. Lock Down Population-Reduction Rules. Ensure no analytics, including email threading and search terms, reduce the population before deduplication or before TAR or AI runs without FTC agreement. Document any proposed exceptions for negotiation.
5. Fix Your Cutoff and Scale. Choose a cutoff score no lower than 0.50 and confirm its scale. If your tool uses categories instead of scores, ask the vendor how it will express a cutoff. Resolve this before the meet-and-confer.
6. Design the Validation Sample. Estimating recall near 75% within 3 percentage points at 95% confidence requires about 800 responsive documents in the sample. At 5% richness, simple random sampling needs about 16,000 draws on average. Evaluate other designs with your statistician and get internal approval before sampling.
7. Plan the Elusion Sample Deliverable. The reported terms require the pre-production elusion sample itself, not summary statistics, at least two weeks before compliance. Ensure your platform can export it with coding decisions intact.
8. Prepare the AI Agreement Study. For generative tools, plan a statistically valid sample comparing AI and human calls, broken out by classification category, plus an export of relevance scores and rationales for produced documents.
Validation: Verify It Works
Conduct a dry run on a closed matter or representative corpus before a live demand.
- Compute Both Measures from One Sample. Report recall and precision at your chosen cutoff and show the confidence interval.
- Test the Point-Estimate Question. If the lower confidence bound must meet 75%, aim for a point estimate near 78%. Ensure the dry run meets this stricter standard.
- Stress the Tradeoff. Raising recall can lower precision. Adjust your cutoff in steps to find where both measures meet the threshold. If no cutoff achieves 75% for both, learn this before it becomes critical.
- Rehearse Retraining. Simulate an agency finding of deficiency, retrain, and time the supplemental production.
- Audit the Paper Trail. Ensure you can reproduce tool name, version, provider, model, prompts, and protocols for every run. Fix any logging issues.
Maintenance and Ongoing Tasks
- Watch for Published Language. Monitor the FTC's Model Second Request and HSR Resources page for revisions, and watch for any court tests of enforceability.
- Re-confirm with Vendors Each Release Cycle. Model or version changes can alter your disclosures and may invalidate prior validation.
- Keep the Requirements Matrix Current. Update the matrix when language is published or modified through negotiation, and retire any unfounded elements.
- Retain Validation Records under your Records Control Schedule, with a Legal Hold flag on any matter records tied to an active investigation.
- Rerun the Dry Run Regularly, and after any platform change, to ensure readiness with the tools you use.
The reported floors may change, soften, or never be officially confirmed. Developing a workflow that meets the strictest reasonable interpretation now can save you from a rushed overhaul later.
