How to Read a Paper

by Rob Zinkov on 2010.12.02

When people first start reading research papers they go through them in the most completely wrong way. They read them from start to finish, and waste precious hours and days trying to understand them. This time is wasteful because usually these papers aren’t very useful. Whether it be during a literature review, recent conference proceedings, or skimming papers on arxiv, I am often confronted with hundreds of papers to possibly read. This is an approach I have found that helps me get through many of them.

Note this isn’t a guide to actually reading and understanding a paper. Many articles have been written about doing that. I found this paper by Michael Mitzenmacher to give great advice for understanding a paper. My advice is mostly about determining if a paper is seriously worth reading.

Skip the title

This seems rather counterintuitive. The title should in principle be a nice concise summary of the contents of the paper. In practice, the titles are so overloaded with insider jargon that unless we are working in that exact subfield there is no chance we will even comprehend what the title means.

As an example, here are some titles from NIPS 2007 papers:

A Spectral Regularization Framework for Multi-Task Structure Learning
Variational Inference for Diffusion Processes 
Fitted Q-iteration in continuous action-space MDPs
Robust Regression with Twinned Gaussian Processes

What is Structure Learning? What are Diffusion Processes? What is Q-Iteration? Even being ignorant of these terms, we might still be qualified to read these papers. This is not a problem unique to any particular conference. All academic papers suffer from this problem to some degree.

The titles though are not going to help us figure that out. At best, they serve as a unique identifier to aid in finding the paper later.

Read the abstract

The abstract on the other hand will help us decide if a paper is seriously worth checking out. The abstract is wonderful. It tells us what the paper is trying to accomplish. Since conferences usually print the abstract separately in proceedings, the abstract will lack pesky citations usually needed to understand the author. Best of all, the abstract is short.

When reading the abstract we are trying to determine two things. First, we determine the goal of this paper. Second, do we care. These are separate but related tasks. Where the title fails, the abstract does the heavy lifting. The abstract is where authors pitch their paper. It must concisely explain the problem they are trying to solve, why the problem is interesting, and why they have a good solution.

When we can’t figure out the goal of a paper from reading its abstract it might mean we don’t have the background to understand this paper, or worse the paper is badly written. A paper not solving a problem we care about isn’t worth reading. Set aside the paper if we think we might care about that problem later.

Examine the experimental results

Convinced we care about the paper, we politely skip to the end of the paper to see the results. While the authors may have claimed their findings are amazing, its important to get concrete. Figures allow us to see exactly how superior their solution is to the competition. The experimental section is also the easiest section to read. It’s all pictures with some text to elucidate elements that were hard to visually explain.

This is important, because the competition is usually simpler and has convienent incumbent status. The incumbent solution is likely already have implemented. The extra complexity needs to be justified in terms of obviously superior performance. We are looking for significant improvements. Why would we choose a complicated way to solve a problem if the simple way is just as good? Look to see where their solution performs well and where it performs poorly.

The results are also important to give a steady reminder of how existing techniques perform. Even not reading any other part of the paper, we are now informed in roughly how difficult the problem is and how far the current state-of-the-art goes to solving it.

Scan the bibliography

Personally, I don’t consider this an essential step, but it couldn’t hurt. Bibliographies provide context to a paper. More specifically, they provide the context the authors were in when they wrote the paper. This context is handy in figuring out what angle the author will attack the problem.

The angle they attack the problem also provides hints whether their approach will be interesting or successful. These hints will lose some value though if we review their results before reading the details of their paper. Reading the bibliography gives a sense of what to expect from the paper.

The bibliography is also handy for reviewing in which community the authors consider themselves. If we are familiar with this community of researchers our opinion of their work can form a reasonable prior for how good this work will be. Ideally all research would stand on its own merits, but heuristics can be handy.

Lastly, read the paper

At this point, we are reasonably sure this paper is worth reading. I don’t have much good advice for this section. There are plenty of guides to reading a paper and understanding it. Instead of completely reproducing what these other guides cover, I will instead give general approaches that should help.

First, try to understand what authors are actually doing. Spend time carefully reading and re-reading sections to confirm we understand the ideas being communicated. Take time to understand what assumptions they are making and why their solution works.

Second, try to understand the limitations of their solutions. See what problems they can’t solve. Where does their solution break down? Is the break down fundamental to the way the problem is defined? These questions let us evaluate the paper on the merits of what can be built on top of it.

Bonus: recreate the experiment

I list this as bonus, since at this point we finished reading. The point of this guide was to explain how to properly read a paper. However, there is more to life than reading papers. The next best step after reading a paper is to reproduce its results. I find I never really understand something until I have reproduced it. No amount of reading can replace that deep intuition only built while actually doing the work.

Re-read the paper with an eye for the details we need to re-create their experiments. We may be surprised to find that details are missing. Read the footnotes to see what details they reveal. The details we need might be buried in a cited paper. Insights missed on previous readings seem to show up now. The altered perspective almost makes it a whole new paper.

Conclusion

I hope this documents helps you read many more papers and save lots of time avoiding bad and useless papers.