Biederman and Burke: Is use of keywords in e-discovery a game of ‘Go Fish?’

Burke

Biederman

The keyword search is the elder statesman of the e-discovery world. Simple to understand and easy to implement, keyword searching has played a central role in e-discovery projects for more than a decade. But as advances in e-discovery expand the capabilities of technology-assisted review, or TAR, and introduce us to concepts such as predictive coding and visual analytics, the good old-fashioned keyword search has begun to lose its luster. Specifically, U.S. Magistrate Judge Andrew Peck in New York, the judge who first approved TAR, wrote that a legal team’s choice of keywords often “is the equivalent of the child’s game of ‘Go Fish.’” Accomplished e-discovery attorney, writer and speaker Ralph Losey notoriously opined that the “keyword search sucks.”

We agree that poorly thought-out keyword searches are time-consuming and expensive. As such, most of the criticism is defensible to a certain degree. After all, keyword searches behave like blunt instruments. Attorneys who are not familiar with proper search methodologies run the risk of creating lists of search terms that are both under- and over-inclusive. For example, searches conducted for the word “record” could yield results for a conversation about the new Kanye West record, a news article documenting Usain Bolt’s world record, a copy of a court record, a company’s safety record, or a request to record “Game of Thrones.” These so-called false positives decrease the percentage of relevant documents reviewed, increase client costs and accelerate document reviewer fatigue.

Inherent limitations aside, the question the legal community should be asking is not whether, but rather how, keyword searches should be used in e-discovery. In fact, the selective use of keyword searches can still be the cornerstone of an efficient and defensible search methodology. A properly deployed search takes advantage of a unique feature of electronic information — the ability to quickly and accurately find relevant people, words and concepts in large data sets. But successful keyword searches require foresight, communication and flexibility between litigators and ESI personnel.

One of the most frequently cited guidelines regarding e-discovery, the “Sedona Principles Addressing Electronic Document Production, Second Edition” (and its corresponding “Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery”) provides thoughtful commentary on the use of traditional keyword searches. It suggests the following primary components can be utilized in an effective and defensible search methodology.

1. Testing. Initially, keyword searches must be tested for accuracy and effectiveness, with particular emphasis on whether the search is producing over- or under-inclusive results. As stated by the “Principles”: “[m]ore advanced keyword searches using ‘Boolean’ operators and techniques borrowed from ‘fuzzy logic’ may increase the number of relevant documents and decrease the number of irrelevant documents retrieved.”

2. Sampling. After initial testing, the primary way to test the efficacy of keyword searches is through sampling. Statistical sampling can assist attorneys in strengthening cases, reducing costs, increasing confidence in relevance and improving defensibility.

3. Iterative feedback. The “Principles” also recommend creating an iterative feedback loop in order to reformulate and refine keyword searches. With collaboration between lawyers and e-discovery experts, searches can be conducted in stages to hone and validate results as the project continues.

Because actual data is being examined when testing, sampling and creating iterative feedback, the keyword search ceases to be the guessing game opined by Judge Peck and begins to increase coding precision and recall (i.e., reduces over- and under-inclusiveness).

In addition to aiding in search methodology, keyword searches can be used to aid document reviewers in their review. As it stands, most document reviews tend to employ a “let it ride” mentality to keyword searches. Once the initial keywords have been tested and sampled, the keywords remain static through the duration of the project. However, modifying and adding to the list of keyword terms during the course of a document review can pay huge dividends for reviewer accuracy and efficiency.

For example, assume all meeting invitations are non-responsive and that every meeting invitation contains the relatively unique term “standard time.” By adding “standard time” to the keyword search, one could filter out most results to meeting invitations with a fair amount of certainty. This will speed up review and provide a degree of clarity to the document reviewer.

Another example exists when a document reviewer discovers a group of attorneys from a law firm that was not included on the initial privilege list. By simply adding the law firm’s domain name to a keyword search, a reviewer will dramatically decrease the risk that privileged documents will unintentionally make their way into a production. Further, if the litigation team did not know the key defendant’s nickname was “Night Train” until four weeks into the review, adding the term to an extended keyword list could come in very handy.

The keyword search has unfairly found itself under fire by a number of judges, attorneys and e-discovery experts. Sure, keywords are not perfect. But their weaknesses often provide an opportunity for litigators and document review attorneys to communicate and provide each other insight on what is being seen in the documents. Reviewers should be trained on the intricacies of keyword lists and to be flexible in adding new terms to the keyword search lists. While e-discovery experts continue to develop new concepts and shiny new software, it’s important to recognize that the keyword search is still a useful — even essential — piece of equipment in the modern attorney’s tool box.•

__________

Ray Biederman and Sean Burke are founding partners and directors of the law firm of Mattingly Burke Cohen & Biederman LLP and ESI consultant Proteus Discovery Group LLC. Through both companies, Ray and Sean bring their experience with ESI to work with lawyers and clients in order to manage data protocols and discovery. Special thanks to Proteus project manager Scott Collins for his contributions to the article. The opinions expressed are those of the authors.