Is This Scientific Study Garbage? A Checklist

Worrying about whatever the latest scientific study has come up with is a seriously stressful parenting rite of passage.

It can start as early as pre-conception and really weighs heavily when you’re pregnant. “Did you read the latest pregnancy study?” is just about as shitty as a conversation opener as there is, and yet, almost all of us will have heard it at some point.

Food, drinks, medications, stress levels and the environment are just a few hot topics where new studies are coming out all the time. But interpreting those results can get tricky, especially if you aren’t familiar with what everything means.

If the latest scientific study has you twitching in your britches, fret not. Here's a checklist of things to consider when determining if this warrants action on your part.

✅ First – do you have the mental and emotional energy to think about this right now?

Is this study addressing something you can reasonably control for, or will knowing more potentially just add stress because it’s something you can’t really change (like air pollution)? Before you commit to the mental and emotional labor of trying to understand it, think about if it will do you any good before reading, or if it will just add extra stress.

✅ Is this study relevant to you?

There’s a big difference between drinking a single soda when you were 13 weeks pregnant and drinking 13 sodas a day for the duration of your pregnancy. Before you read something potentially upsetting or worrisome, really think about if you’re the target demographic.

✅ Read beyond the headline.

Scientific studies filtered through the media aren’t always the most accurate. Whether it’s intentional because they’re trying to get people to click through, or if it’s because they don’t fully understand the scientific jargon, it can be a bit of a game of telephone. If you can, track down the original publication and read through the abstract (usually this is free and doesn’t require a subscription to the journal) and conclusion. And if it doesn’t make sense, don’t be afraid to phone a science friend or your care provider.

✅ What does the big picture look like?

Zoom out and determine if this has larger implications than a single data point. Sometimes parallels can be drawn that suggest correlation between this data point and a future event (like low birth weight and increased risk of obesity later in life), but it’s important to keep in mind that correlation doesn’t equal causation, and this data point is only that – a single scientific finding that may or may not relate to another separate event.

This graph comparing the number of pirates and global temperatures is a great example. Yes, the decrease in pirate numbers lines up with increased global temperatures, but the decrease in pirates isn’t what’s causing global warming – there’s at least one other variable that isn’t being taken into account here.

✅ Is it peer-reviewed/published in a journal?

Is the latest pregnancy study you’re reading something that is listed as pre-print, or has it been peer-reviewed and published in a journal? Even though the information in pre-print articles can be reliable and statistically accurate, having something peer-reviewed lends credence to scientific claims being made.

✅ How was the study designed?

Studies can be run over a period of time (called a longitudinal study) or can be a single event (called a cross-sectional study). Both are invaluable tools when it comes to research, but are very different. Although this is really simplifying things, you typically need longitudinal studies to more thoroughly test for causation.

Here’s an example of a famous longitudinal study looking at 17,000 people born over the course of a week in England, Scotland, and Wales in 1970 and followed by researchers throughout their lives. Results from BCS70 continue to influence policy decisions regarding education and employment, as well as other common social issues like economic insecurity.

And here’s an example of a cross-sectional study looking at beer and obesity. Results from this study make me personally feel less bad about a beer here and there, but won’t likely be influencing any legislation any time soon.

✅ What are the controls?

Any published study will list what it controlled for – like age, weight, substance use, etc. – in the write up. When trying to decide how much weight to put into any one study, looking at the controls is a good factor to consider.

✅ How big is the sample size?

There are no hard and fast rules about how big a sample size has to be when running a study, but generally speaking, the bigger the better (okay, there’s also too big, but since this isn’t an upper level stats class, we’re gonna keep it basic). When analyzing the latest pregnancy study, consider how big the sample size is and if what they are finding can reasonably be applied to a larger population.

✅ Are there similar studies with different results?

A quick search can help determine if any other studies have been run that ask a similar question. How do the results measure up? Do they support each other, or do they paint a less conclusive picture?

And for those who want to take it a step further, you can search for a meta-analysis on whatever study you’re curious about. A meta-analysis looks at and summarizes many published studies that ask a similar question in order to tell a more comprehensive story.

✅ Be aware of the “file drawer” problem.

Think of this kind of like scientific clickbait, but without the intentional dishonesty. Essentially, results that show a significant association are more likely to be published in a journal than results that show a null finding (meaning no significant association). There may be several studies that don’t get published because they didn’t find anything statistically “exciting.”

✅ How are the results worded?

Words carry a tremendous amount of weight, and scientific writing is no different. In the results, carefully look at how they word the findings. There’s a big difference between “X causes Y” and “under these circumstances when controlling for A, B, and C, a moderate correlation was found between X and Y.”

What do you do now?

If the latest pregnancy study has you worried, you can always call your care provider. You won’t be the first, and you most certainly won’t be the last.

The next time some well-meaning associate comes up to you and asks, “did you see the latest study?” you can shut ’em up with “yeah, but correlation doesn’t equal causation.” Boom. You’re basically a scientist now.