How to read a psychology or cognitive neuroscience paper

Image source: Verisk Analytics.

[Revised 2016, crossposted from Medium].

Scientific papers can be hard to read. But it’s not only the jargon that makes them difficult. Scientific papers, like poetry, are meant to be read in special ways. Researchers know how to read studies efficiently, and you can learn, too.

I will focus on psychology and cognitive neuroscience papers because I know these fields best, but some of the principles will apply to other research areas, such as genetics or intervention studies.

Before you begin

The first step comes before you ever pick up the paper. Before you start, know why you’re reading. What kind of information do you want to find?

A student or researcher might want specific numbers and methodological details, so they can try doing the experiment themselves. An autistic person or a parent might also want a specific number (say, the rate of increase in autism diagnosis over the past ten years), or the details of a measurement technique (say, how demographic studies decide whether to count a person as having autism). However, their purpose differs. Usually, they want to better understand what the researchers did, and whether the results support their claims. This makes reading easier in some ways — you need fewer details, and the information might be easier to find. But it makes reading harder in other ways — you need a lot of critical thinking.

If you are new to autism research, you will probably want to know the current state of autism research in general. What are the major theories, and who are the important researchers? You might have specific questions about how autistic people develop or how to help them. For example, “What’s different about my child’s language development?”, “why do so many autistic people have anxiety,” or “are gastrointestinal problems really more common in autism?” Or you might have more general questions: what causes autism? Are the rates really increasing? How does it affect the way people perceive, move, think, feel, and learn? Why do autistic people have difficulty understanding neurotypical people’s behavior and forming relationships?

And, of course, you’re probably bringing theories to the table, from the media, previous reading, and your own experience. You might have strong opinions about the above questions, and be looking for evidence. (We all do this, but you may be disappointed to discover that research rarely perfectly supports one conclusion).

Your motivation will affect what type of paper you should choose, and which parts to focus on.

Know the Types of Research Papers

Scientific papers come in three basic flavors: experimental papers, review articles, and meta-analyses

.

Experimental Papers. Most people think of this most common type of paper when they imagine a research study. In these papers, the authors describe the results of one or more studies they performed, and draw conclusions about what the results mean. Most psychology and neuroscience papers fall into this category. Sometimes these involve new experimental techniques or flashy, gee-whiz findings that spawn new areas of research. More often, they occur within an ongoing conversation between researchers with different theories. They involve refinement to old procedures and correcting for seemingly tiny details (e.g., “X researcher argues that autism is explained by Weak Central Coherence theory, which predicts Q, and they report Y results. However, X overlooks Z methodological problem. We corrected for Z by doing A, and got result B, calling Weak Central Coherence Theory into question and supporting Enhanced Perceptual Functioning theory instead”).

Reviews. Whereas an experimental paper presents a statistical analysis of a single research group’s work, a review is an expert’s summary of the field as a whole.

You can identify review articles by the number of references they contain (often, hundreds). They also typically lack a Methods or Results section (more about these later), because the authors aren’t performing any experiments, just reporting on other people’s.

Some journals, such as Current Directions in Psychological Science or Frontiers in Neuroscience, publish only reviews.

Meta-Analyses. A meta-analysis is like a review paper, but quantitative instead of qualitative. A review paper tells you what patterns studies have generally found, and describes any areas of disagreement. A meta-analysis performs various statistical procedures to see if any results can be found across all the studies, and if so, how strong they are.

What kind of paper should I read? If you want to know the evidence for a particular claim, you’ll usually need to read an experimental paper or meta-analysis. If you want to learn about a new area of research, find out what “researchers in general” think about it, or discover what the big debates are, read a review paper. In general, if you’re just starting to read scientific papers, reviews are a good place to start.

Know the structure of scientific papers

The structure of a paper varies somewhat from journal to journal, and field to field. However, they all have the same basic parts, just arranged in different ways.

Abstract. The part of the paper that tells you what you’re going to read. Sometimes, busy researchers only read the abstract. Physicist Chad Orzel explains:

The Abstract of a scientific paper is a one-paragraph (usually) summary of the main points of the article. You can think of this as sort of like the “Attention conservation notice” Cosma Shalizi puts at the start of his longer posts: it’s there so you know what to expect the article to contain, and reading it should tell you whether you want to read the paper or not.

Introduction. Again, Chad Orzel describes it best:

Methods. This section explains what the researchers did and how they did it. It usually has several subsections with titles like “participants,” “materials” and “methods” or “procedure.”

Look at the participant section if you want to know more about how they chose the people in the study and what they were like. For example, you will want to know their sex, age, IQ, and if they had any other diagnoses or were taking any medication. Look at the materials section for a detailed description of what participants saw or heard. For example, a study comparing autistic and typically developing children’s reactions to animated shapes would describe the animated shapes. In a neuroimaging study, the materials section will also describe the model and settings of the scanner used, the methods for rejecting random noise, and similar information.

The methods or procedures section will tell you what happened during the experiment from start to finish, and sometimes how the data was analyzed.

In most psychology and neuroscience papers, the Methods section appears between the Introduction and the Results. However, some journals in these fields put the Methods section at the end.

Results. This section describes what the researchers observed. It can often be identified by its tables and graphs.

In psychology and neuroscience papers, this section is extremely dry. It’s a list of statistical analyses and their outcome; almost every sentence reads like this: “An ANOVA of the effect of caffeine revealed that the three groups differed significantly in their reports of alertness, F(2, 39)= 4.37, p < .05.” If you see lots of sentences filled with Greek letters, numbers, and < signs, often in parentheses, you’re dealing with the Results section.

Discussion. This is where you find out what the researchers think it all means. It differs from the Results section in two ways:

1. It uses plain English instead of statistical notation;
2. It places the results in the context of prior research and suggests implications for future research.

I’ve also seen a combined Research and Discussion section. These have some sentences that present the statistical analyses and others that talk about what the results mean and put them in context. Chad Orzel calls it a “Conclusion” and describes some of its functions:

Conclusion-type sections are where you discuss sources of error… and the possible implications. Conclusion sections are the third area with lots of citations, often repeating citations from the Introduction, but sometimes bringing in entirely new papers that are supported or contradicted by the current results. This is often where you find proposals of new measurements, as a way of staking a claim to a field.

Now that you have all that background knowledge, you’re ready to actually start reading the paper.

Read the paper out of order

Scientific papers are not like novels. You almost never want to read them straight through.

First, read the abstract. A good abstract will tell you, in the most general terms, what the researchers did, why they did it, and what they found. It will let you determine after a single paragraph whether the paper will be useful to you or not.

If you know the background research that inspired the study, or you don’t care because you’re looking for a specific piece of information, skip the introduction and go right to the discussion. Otherwise, read the introduction first, then the discussion. I recommend reading the introduction because it will explain what they think is important and how they frame the problem. A psychology researcher might have different ideas about, say, romantic love or attention than the average person. Knowing these can keep you from feeling confused or unpleasantly surprised when you reach the discussion section.

Next, check the results section to determine how much you can trust the results. Most studies include p-values (the letter in front of the < sign). These indicate the likelihood that the results of the study were due to random chance. By general consensus, results are considered “significant” when p is less than .05, or there is a less than 5% chance that the results were due to chance. Some papers will talk about trends that don’t quite reach significance (p=.051-.079, or a 5–8% chance that the results aren’t real) as if they were significant (p<.05). Even if the results are clearly significant, pay attention to how significant they are. An effect that was barely significant (e.g., p=.049) in a study with a lot of participants is a lot less convincing than an extremely significant (e.g., p< .0001) result in a study with few.

Notice that p-values do not tell you the size or strength of the effect. In a big sample, like the entire population of the United States, an effect too small to be practically meaningful can turn out significant. A food might turn be reported to increase cancer risk, but the effect might be so small that, say, only a few hundred people in the entire country get cancer who otherwise would not have. Most people who eat that food will never get cancer (and many people who don’t eat it will). On the other hand, a large effect (e.g., that men are taller than women) will not be significant in a very small sample.

To understand the strength of the effect, you need effect sizes. Only some papers report them. They will usually be reported as a d statistic where, as a rule of thumb, values of 0.2 or lower are considered small, those around 0.5 medium, and those 0.8 or higher large (see this visualization by Kristoffer Magnusson).

Finally, if you have any questions about the procedure, look them up in the methods. The methods are not meant to be read straight through. However, you will often have questions about what, exactly, the researchers did. Sometimes, these might affect whether you agree that the results mean what the researchers think they mean. For example, when the CDC reported an increased prevalence of autism in 2009, you could make an educated guess based on their procedures whether they underestimated or overestimated the actual rates.

That sounds like a lot, but don’t be discouraged. Reading scientific papers is a skill, which is rarely explicitly taught, takes time to develop, and challenges everyone. Do you have any questions, or tips for reading papers better? Share them in the comments!

Take Home Point

Know the parts of a research paper, and ALWAYS read out of order. Abstract first, discussion before results, and methods last. Know what you want from the paper before you start, and choose from experimental, review, and meta-analysis papers based on your needs.

Further Reading

This guide was inspired by Chad Orzel’s, which is ideal for college students in physics.