elitefts™ Sunday edition

Reading Research, Pt. I

As a lead-in to some science-focused articles I’ve been putting together, I thought it would be worthwhile to look at how I read peer-reviewed scientific articles and determine how (or if) they’re useful to strength athletes. While I’m lucky enough now to have a 9-to-5 job where I review academic writing, I started out reading scientific papers for my own benefit. To be blunt, I didn’t trust a lot of fitness writers and supplement companies to be honest when citing research, so I read the papers myself. What I learned positively influenced my diet and training, and saved me a lot of money.

A Primer on Papers

It’s important to know that scientific/medical papers are rarely “how-to guides.” Instead, they seek to answer very specific questions. “What’s the best way to put on muscle?” wouldn’t be a research topic anywhere, although figuring the actions of an enzyme or comparing two similar methods of weight lifting are common topics. Papers are structured in a fairly established manner and with similar stylistic features, both of which extend across various health/science disciplines, that help to infuse intellectual rigor into a researcher’s ideas and experiences.

The writers of papers may have personal objectives behind the publication—getting tenure, laying the groundwork for a grant request, boosting attention, helping others, keeping a job, etc.—which can flavor the article a little bit, but largely the formatting norms of these pieces keep them similar. It’s important to know that most papers are written by university professors. Academia is competitive, and being heavily-invested in one’s work is essential for success. “Publish or perish” is an exaggeration, but not by much. This demand can sometimes lead to a small degree of hyperbole or “reaching” within a piece. In my experience of working with professors, it's more common in grant applications (“my project on arboreal fungi in the Ozarks is essential to the survival of the human race, and so demands National Science Foundation funding for a man-portable, solar-powered scanning electron microscope”), although you’ll see it in published papers at times.

The conventions of form aren’t the only things keeping papers from straying into the rants, product placement ads, and bafflingly idiotic pieces that plague commercial info sources. The peer review process, in which experts in the article’s field of study will fact check and certify a piece for intellectual integrity before publication, is a strong force in ensuring that nearly all papers are honest and smart. Also important is that the wider audience of researchers will happily tear apart each other’s work after publication; the internet made this much easier.

Formal/pre-print and informal/post-print peer review aren’t perfect (peer review is conducted by humans, after all), so occasionally bad research slips through the cracks. It’s difficult to detect erroneous or falsified data in novel experiments and surveys; as evidence, the ongoing vaccination scare was prompted by a fraudulent paper that took several years to discredit through follow-up experimental reproduction. Finally, fringe elements of the scientific community can (and do) establish their own networks and journals in order to circulate ideas that are lacking in evidence and/or rigor.

For elitefts™ readers, useful pieces fall into a few different categories; as a caveat, this is all generalization using non-standard terms, but it should cover most of what you’ll come across:

Experiment Reports: These are the papers written by researchers after they’ve tested out an idea on a small scale. The idea could be seeing if Olympic lifts produce more force than powerlifts, or if casein proteins lead to greater muscle gains in athletes. Since experiments are run under tight restrictions and rely on eliminating variables, they’re often narrowly focused and most useful as clues on how to train, eat, supplement, etc.

Case Studies: A researcher (usually a medical doctor or trainer) writes about a single person he or she benefited. This benefit is usually treating an injury or illness, though you’ll also find coaches and trainers offering accounts of how they worked with an athlete or small group of athletes in a real-world environment (such as in/off-season training). Typically, either the technique used is new, or the case is unusual and of interest to the establishment. I’m pretty certain the first case study I read was on the dreaded candiru; fortunately (and I’ll spare you the details as to why) the paper likely was largely or wholly fraudulent, though it made for interesting reading. Because you’re basically trusting one person’s word on this, there’s a bigger chance for fibbed results than with other papers.

Surveys: These large studies exchange control over subjects for the benefit of a large number of subjects. Sometimes they involve tracking multiple variables over time, and then draw conclusions from commonalities and differences. For example, a large heart health study could survey 10,000 people at regular intervals and find that the people with coronary artery disease were the same people that ate fast food most frequently; it could even go a step further and hypothesize that there’s a connection between the condition and eating too many burgers and fries. It’s common for researchers to go into the large data sets created by surveys in order to perform their own research; in this instance, the original survey acts as a sort of control group.

Theory Papers: This is a little rarer in the scientific publications of interest to us, though pretty common in the general press. Basically, it involves a researcher writing about an idea and providing some circumstantial or anecdotal support for this idea. When a coach pontificates about a training technique (Mike Boyle’s single-leg ruckus comes to mind), it could be classified as a layman’s theoretical paper. Because theory papers don’t actually test anything, you won’t deal with the normal parts of a paper and there will still be some degree of review; additionally, a lot of rules for critiquing normal papers also apply to theory papers.

Reviews: This kind of report looks at the results of multiple papers and tries to draw a common conclusion from them. If you’re looking for a clear-cut answer to a problem, you’re more likely to get it in a review piece, provided your question is one an academic might ask.

Position Paper: These are the exception to the general rule of “how-to guides.” These papers explain how a particular organization views a topic, e.g., how best to train sprinters or rehab broken bones. You can find these in sport science journals, though don’t expect anything as detailed as a routine.

The Parts of a Paper

The Abstract: Thanks to the proliferation of online resources, you’ll find tons of study abstracts available online. Unfortunately, these brief overviews of papers won’t do much other than spark your interest in reading further. By their very nature, abstracts rarely tell you if a paper with interesting subject matter is worth your time or not. When I post abstracts on Twitter, it’s always as an alert for people to watch for the full text of a potentially interesting article in the coming months.

Introduction: If you’re relatively uninformed about a topic, this is useful for catching up. If you’re well-versed in a topic, the introduction can help you understand the researcher’s perspective, particularly in terms of what information sources they use, which can help you anticipate limitations in papers.

Procedure: Here’s where things get good, particularly with lab experiments. My main concern when reviewing a paper’s procedure section is applicability, which is just how the variables and environment in the paper apply to me and my lifestyle. This is easy to determine in a case study: if I have a torn ACL, a case study on ACL rehab on a man my age will likely be applicable.

The variation in lab experiments is a little less straightforward. In papers with control and experimental groups, it’s important to make sure both subject populations are relate to you. Here’s a look at common factors in health, diet, and exercise studies, as well as how I determine their usefulness to me as a 30-year-old male who lifts, is in decent health, of average physicality, and has been training consistently for about 15 years.

 

 

There have been a ton of test tube and rat studies that didn’t pan out in humans. Those examples are easier to spot, even though others aren’t, and can require digging down a little. Some common catches that can go unnoticed are:

No Relevance to Controls: When a researcher tests an interesting idea against a control group, it’s easy to think only about the experiment, while forgetting the control group. Don’t do that. If a new diet strategy is being tested against a control group that follows a typical American diet, the results won’t mean as much.

Newbie Gains: Since most untrained individuals will get bigger and stronger by regularly lifting just about anything in any manner, experimental results with these subjects aren’t very useful.

Unrealistic Routines: An experiment that uses an unrealistic exercise protocol or diet regimen has little value to most elitefts™ readers. You’ll see lots of papers with titles like “Lifting Weights with Prolonged Eccentric Portions Increases Muscle Mass.” Then, you read the paper and find out the subjects performed nothing but leg extensions twice-a-week using 30 percent of their one-rep maximum. This study might be a prompt for other scientists to dig deeper, but it probably won’t help you.

Fasted Subjects: It’s easy for a test subject to eat something weird before a diet study, which will then completely throw off their results. To get around this, researchers will often instruct their subjects to not eat for at least twelve hours before undergoing certain diet and training experiments. This is great for research, though the fact that the subjects are nutrient-deprived means the results don’t generally apply to a well-fed athlete.

Testing: This is another one to watch for. Did the study look for actual, measurable muscle growth from MRI or DEXA, or did it just use biopsy results that suggest increased muscle growth? Did the diet study accurately record body composition and control for varying hydration states, or did it only measure scale weight?

Small Subject Groups: With a small group of subjects, you increase the odds of one person with an unusual profile skewing the results. Imagine if you’re testing an endurance protocol on ten people, with four subjects in the control group. If one of the controls has undiagnosed anemia, it’ll throw off your results while being hard to adjust for as an outlier.

With all these catches, I will say that I’m not a big stickler for double-blind experiments when it comes to learning more about strength training and physique enhancement. A study is considered “double-blind” if the subjects don’t know if they’re part of the control or experimental group, and the administering researchers also don’t know which group is which. This prevents the subjects from reporting a placebo effect, and the researchers from letting their expectations cloud their analysis. This is extremely useful for medical studies, which often involve the patients reporting how they feel and doctors interpreting these reports, where personal bias and expectations can cloud results. With studies involving measurables, though, the collected data can generally speak for itself (though the conclusion may be arguable). A special note is that in surveys, the actual survey collection method is important. You’re usually looking for survey methods that sample about as random a population as possible.

In Part II we’ll finish looking at the paper itself, and then talk about getting your hands on the actual research.

For readers who are completely new to research, I recommend reading Chad Orzel’s post over at ScienceBlogs: http://scienceblogs.com/principles/2012/01/how_to_read_a_scientific_paper.php.