BasicsYou always want to look at how many people participated in the study, and who they were. A study of 20 college students may not tell you much about the average 60-year old. (Though small, tightly-controlled studies can sometimes tell us as much as larger, less well-controlled ones.) The length of the study is important, as diet results over time are different than short-term ones. Also, look at the drop-out rates, which may skew the results. The funding organization (s) are important to know, but don't immediately discount an organization that looks like it might have an interest in the results. In the case of low-carb studies, it's actually quite difficult for researchers to find funding, and the Atkins Foundation is one of the few places they can go. That organization is scrupulous in its total non-interference with the researchers or the results.
Problems with Observational/Epidemiological StudiesA lot of the diet-related studies you'll see in the news simply follow people (sometimes very large groups of people) over a period of time. They track a bunch of the things they eat and a bunch of other things about them, and see which ones seem to occur together. These studies can be valuable for what scientists call "hypothesis generation". For example, we find out that the people in Fictional Study A who eat bananas are more likely to get hangnails. Does this mean that eating bananas causes hangnails? It does not. We could test that hypothesis if we wanted to in an experiment (give half the subjects bananas and see if they developed more hangnails than the other subjects that eat no bananas), but until that experiment is done, we have no idea. And then we'd have to make sure we figured out whether eating the bananas caused the hangnails, or simply peeling them.
One of the main problems with attempting to draw conclusions from observational studies is that the studies test a limited group of things and you don't know what other untested things may be the mix. For example, a 2012 study concluded that eating red meat is associated with dying (OK, "increased risk of death from all causes"). But people who eat more red meat have also been shown to: exercise less, smoke more, eat more sugar, eat more refined grains including white flour, eat more calories, weigh more, take fewer vitamins, eat fewer vegetables, eat less fish, and probably many more things. Some of these things were adjusted for in the "red meat will kill you study", and others weren't. So we basically have no idea why the people in that study who ate more red meat had an increased mortality rate.
Problems with Experimental Diet StudiesEven when it comes to real experiments, there are a number of problems that repeatedly crop up. In this section I'm going to talk mainly about diet studies where carbohydrates were a major component, but some of the principles apply to other diet studies as well. Be on the lookout for:
The Definition of a Low-Carb Diet - Diets everywhere from Atkins Induction (20 grams of net carb per day) to 45% of calories from carbohydrates have been called "low-carb" in the scientific literature. There is simply no consensus about this. Of course, this matters a lot! Sometimes two diets which turn out to be not very different from each other turn out to have outcomes that are, well, not very different from each other.
How Much Carbohydrate Did the Groups ACTUALLY eat? - Newsflash -- people don't usually stick to diets, and when they do, the usually need a lot of support to make that change. Not surprisingly, the studies where people get support are more apt to tell us about actual diet effects (as opposed to the effect of reading diet books), but even then, compliance tends to wane over time. Therefore, it's very important to pay attention to what people actually ate when assessing the results.
Food Frequency Questionnaires - These are often used as means to assess what people are eating, but they are actually one of the worst ways to do it. In the first place, they rely on accurate memory and a willingness to accurately report, sometimes over the course of a year. This has been documented to be a dubious assumption. Also, many of them are biased towards the "standard" diet, and will tend to be even less accurate for assessing foods commonly eaten on low-carb diets. I was part of a study once where I had to fill these out, and I was constantly trying to second-guess what would make the results more accurate. Does "muffin" include those made with almond meal and flax seed meal? Probably not.
Subjects On Calorie-Reduced Diets Often Also Reduce Carbs - The low-carb diet is often compared with a low-calorie low-fat diet. In reducing calories, the subjects in the low-calorie group usually also reduce the amount of carbohydrate they eat. Could this be part of the effect of the diet? Who knows? Researchers almost always ignore this.
You Can't Change Only One Thing - Are the subjects in the low-carb group eating more fat than they did before? More protein? Both? Neither? Are we measuring the lack of one component, or the increase of the other?
A Very Important Factor Usually Overlooked! - SubgroupsIt's become very clear by now that different people respond to different diets, based at least partly on genetics, and also probably based on the presence of conditions such as insulin resistance and other factors. In the famous "A-Z Diet Study", the original paper talked only about the averages of the groups when reporting the amount of weight loss (as is usual). That average difference was quite small. But it turned out there was a huge range of weight loss *within* each group (a 66 pound range!), and when they did another analysis later, they found out that the people who were insulin resistant or had metabolic syndrome responded best to low-carb. Of course, almost no one is aware of this finding (I heard the head researcher, Dr. Christopher Gardener, talk about it at a conference). To me, this means that when there is random assignment in an experiment and a broad range of responses to the intervention it's important to generate hypotheses about why different people had different outcomes. And then do new studies based on those hypotheses. (There are other studies, which show the same types of results, such as this one.)
Of course, not only do people respond differently to different diets, but different people are going to be willing to eat different diets. If you absolutely hate the food of the diet group you've been assigned to, what are you going to do? You may be game at first, but most people would not hold out for long.
Other Factors1) Support - I've already touched on the importance of support in diet studies. This is particularly important with low-carb diets, because as we know, the world is not set up to support the low-carb dieter. Furthermore, most of the people running the studies do not know how to properly support low-carb dieters, which could influence the outcome.
2) With cholesterol, measurements of particle size - Generally, a low-carb diet will improve triglycerides and HDL ("good") cholesterol. LDL ("bad") cholesterol tends to be more variable as far as response to carbohydrate reduction. The total LDL sometimes goes up, and sometimes goes down (though it often goes up for the first 3-4 months, and then declines). What is more consistent about LDL cholesterol response is that the particle size distribution usually shifts from the more dangerous pattern B (smaller, dense particles) to the safer pattern A (larger lighter particles). If the researchers don't measure particle size, they will miss this common and important outcome. (More about low-carb diets and cholesterol)
I hope I haven't discouraged you too much! I'm a science geek myself, and I find it very exciting. But I'm also aware that there are interests competing with the search for knowledge. The media creates dramatic headlines to garner attention. The researchers need funding for their next project, so they can over-hype results. But if we pay attention, we can gain fascinating new insights at the edge of human knowledge.