Literature Searches, Surveys and Reviews: When To Stop

The world of research is constantly evolving. Scholars are always generating new ideas based on past research, so there will never be a time when the research landscape is "complete." Knowing when to stop is subjective and is often based on time constraints and it is very easy to find yourself trapped in a never-ending loop of more and more searches.

A simple way to prevent this from happening is to break the entire process into phases. The "Searching" phase will be followed by organizing and synthesizing phases, all of which can be assigned an approximate date range to help you keep on track and budget your time accordingly. 

A few things to consider when deciding when a search is done (or done "enough")  are:

• The law of diminishing returns and Pareto Principle (sometimes called the 80–20 rule) should be considered. In the case of a literature search that means excessive searches in the same locations using the same techniques may not be time well spent, simply because most of the relevant citations have already been found.
   
• Example: A focused effort of 3-5 hours of careful searching may yield 80% of all relevant citations that can reasonably be located when generally searching in the right ways and generally searching in the right places. Spending another 10 to 20 hours on the search may yield a few more marginally useful citations, but possibly only another 5% to 10%. However, it is not a realistic expectation to find 100% of relevant research on a topic, regardless of the amount of time you spend. 
   
• Finding the same citations over and over in your search results, or finding new articles presenting concepts or findings similar to what you have already discovered suggests you are concluding the "searching" phase.
   
• There are always research projects in progress, and new articles, conference papers, and technical reports in the publication pipeline, some of which may never be published. It typically does not make sense to delay a literature search to wait for new content to be published. It is usually better to gather what is available at that moment in time. Some databases allow users to set up alerts notifying them when new results that match a saved search topic are published. 
   

Excerpted from Daly, Meier, Winter & Yu, "Literature Searches How to Search." in Literature Searches and Literature Reviews for Transportation Research Projects: How to Search, Where to Search, and How to Put It All Together: Current Practices.