Should You Filter AI-Coded Documents Through Human Review?

You're considering generative AI for document review. The vendor claims 87.7% recall, while your current technology-assisted review achieves 64.4%. The choice seems clear.

It's not.

The real decision isn't just about AI finding more responsive documents. It's about whether those documents should go to production without human filtering and whether higher recall justifies lower precision. This choice impacts your costs, proportionality arguments, and defensibility.

The Real Question: Filtered or Unfiltered Workflow

Your decision hinges on one factor: do first-pass reviewers approve documents flagged by the system, or does the system's classification stand as final?

This isn't a technical preference. It's a workflow design choice with significant consequences. When Cormack, Grossman, and Pickens reanalyzed the Redgrave study comparing Relativity's aiR for Review to traditional continuous active learning, they found that workflow architecture mattered more than the technology itself. In filtered workflows where reviewers had the final say, traditional machine learning achieved 64% recall compared to 59.5% for generative AI. In unfiltered workflows where the system's determination stood, those figures jumped to 97.2% and 88% respectively.

Precision numbers moved in the opposite direction.

Key Factors That Drive Your Choice

Your proportionality threshold. Can you justify reviewing or producing the extra documents that come with higher recall? The doctrine of proportionality requires balancing review burden against document value. If you're retrieving one responsive document for every ten non-responsive ones, you've likely crossed into disproportionate territory.

Your ability to estimate the recall-precision curve. Traditional continuous active learning methods build gain curves incrementally, allowing you to assess utility as you proceed. Generative AI tools often offer only two cutoff points with no calibrated method to choose between them. If you can't explain why you stopped at 87% recall instead of pushing to 95%, you can't defend that choice under proportionality analysis.

The consistency of your responsiveness determinations. In the Redgrave study, first-pass reviewers classified 34% of documents as non-responsive that the independent expert considered responsive. The expert classified 61% of documents as non-responsive that reviewers considered responsive. When your reviewers, your prompt engineer, and your evaluating expert interpret responsiveness differently, you're comparing judgment calls, not technologies.

Your transparency requirements. If you can't explain your prompts, training methodology, or cutoff logic to opposing counsel or the court, your efficiency gains become defensibility risks. Confidential prompts limit your ability to demonstrate a defensible process.

Path A: Filtered Workflow (Human Final Say)

Choose this path when:

• You're working under tight meet-and-confer timelines where precision matters more than exhaustive recall
• Your matter involves nuanced privilege determinations that require human judgment
• You need to defend a lower recall figure by demonstrating human oversight at every decision point
• Your client's risk tolerance favors fewer false positives over potentially missing responsive documents

In this workflow, your AI or machine learning system ranks documents. First-pass reviewers code them. Only documents reviewers approve as responsive move forward. Your recall will be lower, but your precision will be higher, and you'll have a documented human decision for every production call.

The Redgrave reanalysis showed traditional continuous active learning slightly outperforming generative AI in this configuration, but the difference was marginal. Your real advantage is the documented review trail, not the technology choice.

Critical requirement: Your reviewers must apply consistent responsiveness standards. If your team's determinations vary widely from your subject matter expert's view, you're building a defensible process on an inconsistent foundation.

Path B: Unfiltered Workflow (System Final Say)

Choose this path when:

• You're facing document volumes where comprehensive human review is disproportionate
• Your recall target approaches or exceeds 90% based on specific case requirements
• You can articulate a principled cutoff point based on gain curve analysis, not arbitrary thresholds
• You have a transparent methodology you can defend in court, including training data selection and quality control sampling

In this workflow, documents the system classifies as responsive proceed to production without additional human coding. You're accepting that higher recall comes with lower precision and defending that trade-off as proportionate to the matter's stakes.

The Redgrave reanalysis found that traditional machine learning achieved 97.2% recall in unfiltered mode, but the authors noted this would "most likely be considered disproportionate because of review cost." You're not just defending the technology; you're defending the decision to review or produce that volume of documents.

Critical requirement: You must be able to demonstrate where diminishing returns set in. If you can't show the court that pushing from 85% to 95% recall required reviewing exponentially more non-responsive documents, you can't defend stopping at 85%.

Path C: Hybrid Approach with Calibrated Sampling

Consider this path when:

• You need flexibility to adjust recall-precision trade-offs as the matter evolves
• You're working with limited training data but need defensible results
• You want to demonstrate continuous quality control without full second-pass review

Use your technology to rank documents, but implement statistical sampling at intervals to validate performance and recalibrate cutoffs. The Redgrave reanalysis showed that Scalable Continuous Active Learning trained on only 2,000 first-pass coding decisions matched generative AI's performance while offering calibrated adjustment of the recall-precision trade-off.

This approach gives you the flexibility filtered workflows lack and the defensibility unfiltered workflows struggle to demonstrate.

Decision Matrix

Your choice isn't about which AI performs better. It's about which workflow you can defend when opposing counsel challenges your recall figures, or when the court asks why you stopped reviewing at your chosen cutoff point. The technology comparison matters less than your ability to articulate a proportionate, consistent, and transparent process.