Tips on Literature Surveys
CMSC 601 -- Spring 2012
Prof. Marie desJardins

What Makes a Good Survey Paper?

• Focused on a specific area/topic, and clear about exactly what subspace of the research landscape it covers (and what it doesn't cover).
• Comprehensive in its coverage of the topic, not omitting important subfields or perspectives. Many surveys cover 100 publications or more. Of course, this varies widely, depending on the topic, intended audience, and quality of the survey. (Of the three surveys I've assigned for you to look at, Jain et al. has six full pages of references (200+ entries); Fodor has 53; and Noy has 26.)
• Up to date, including the most recent work in the area. You might find a great survey on your topic, but if it's even 4-5 years old, and the area is still active, you still have a lot of work to do.
• Well organized, structuring the topic into subfields, relating them to each other, and characterizing the work within each subfield by type, method, assumption, etc.
• Clear, describing the most important methods or algorithms in sufficient detail and in a way that is accessible to the average reader, and summarizing the key aspects of the less important work.
• Analytical: that is, not just enumerating an extensive list of different approaches and methods, but analyzing their strenths, weaknesses, and problems for which they might be most appropriate.

  Finding Relevant Papers

  • Google is a useful tool, but should not be the only place you look! Also, as you get to know the field you're investigating, you can keep going back to google (and other sources) with new keywords (buzzwords) you've discovered.
  • CiteSeer, an online citation index and paper database originally developed by NEC, is a terrific resource. Many of the top google hits will likely be to CiteSeer papers. In CiteSeer, you can search by keyword/title/author, and can follow citation links forward and backward from important papers. There are also other nifty features like semantically similar papers, and "importance" measured based on number of citations. Many papers have associated (but possibly incomplete/incorrect) BibTeX entries.
  • Google Scholar is becoming very useful as well. The interface can be a bit difficult to use at times, and you can't always get the bibliographic information about articles here, and the article itself may or may not be available, but Google Scholar has fairly extensive citation counts, which can be quite informative. If a paper published in 2000 has only had 12 citations, it may be interesting, but it hasn't been super-influential. On the other hand, if any paper in your topic area has 500 or more citations, you definitely need to know what it's about!
  • For AI papers, if you really don't know where to start, AI Topics is sometimes helpful. It's a AAAI-maintained website that has some introductory material, a collection of links on various topics, news articles, and other interesting pointers. But the actual content is on the non-technical side (it's more at the high-school term-paper level), so it should just be one resource, not your primary one.
  • Identify a few important and relevant (recent or classic/seminal) papers, and work forwards and backwards through citation links (following references in the paper, and looking in CiteSeer to see who later cited this paper).
  • It's important to know what are the key publications (top journals and conferences) and researchers (most published and cited authors) in your field of interest. You may be able to find this out early in the process by asking somebody who's knowledgeable, or by stumbling on (or starting from) the key seminal paper(s). Or you may have to discover it more gradually. (If you see a citation or a name repeatedly, look it up! It's important! But also note: "important" is not synonymous with "good" -- sometimes everybody cites a paper just because everybody else does, not because it's actually a particularly good or useful paper.)
  • Also pay attention to institutions -- you'll quickly learn which places are doing important work in your area. (And my list of top institutions for AI won't be the same as your list of top institutions for graphics, or whatever. In fact, my list of top institutions for my particular subareas of AI might not be the same as that of some other AI researcher with a different focus. Of course, these things are subjective, so my list might not even be the same as that of a researcher with the same focus...)

  Locating and Reading Papers

  • If you can't find a paper online, try the library, other students, your advisor or outside reader. (For example, I have many of the older volumes of the Machine Learning Journal in my office. Also, as a AAAI member, I have online access to all AAAI publications.)
  • Be sure you use your time wisely, using the paper-reading tips we've already gone over to figure out which papers are worth reading closely, and which only deserve a cursory review.
  • Don't make the mistake of "depth-first search" of the paper space. Instead, look at papers closely enough to know how important they are, and to start creating clusters of "similar" papers (i.e., identifying themes, threads, or styles of research within the field). Then organize your reading by clusters. This is a much more efficient way to read than a scattershot approach. Also, doing this as you go along will help you to organize the literature survey itself.
  • As you read papers, make note of important citations to follow up on (and what cluster those citations seem to belong to).
  • Know when to stop. There will be more related fields than you can possibly follow up on. Be prepared to say "Researchers in statistics and physics also touch on these topics," perhaps (if you're lucky) with a pointer to relevant survey papers on those areas, and leave it at that.
  • Take notes and keep them in an organized system -- notebook, online, whatever. Don't just scribble on pieces of paper. I like to make notes in the margins of papers, but also to write short bulleted summaries of papers that are particularly relevant for an area that I'm trying to synthesize.