Evidence that there is still lots of potential for study researchers and their sponsors to only let you see what they want you to see

While some people put a lot of faith in published scientific and medical research, there are a number of reasons why we sometimes need to be a bit sceptical about what we read in scientific journals. One fundamental issue here concerns what is known as ‘publication bias’ ” the tendency for favourable results to be more likely to be published than less favourable findings. For example, let’s say a drug company funds a study into one of it products and this yields results which put the product in a favourable light. You can imagine that the drug company will be keen to have this study published, preferably in a prestigious or ‘high impact factor’ journal. However, if the results were unfavourable, however, you might imagine that the drug company may be none too keen for the study to see the light of day.

A few years back, I was having lunch after giving a lecture to a group of business consultants. While chatting one of my lunch-mates volunteered that one of her clients, a drug company, had some years before embarked on a study involving one of its products. The lead investigator had insisted before the study began that the trial be published on completion, whatever its findings. The drug company agreed to this proviso. However, the preliminary results were not good. According to my lunch-mate, the drug company committed to keep the study running so that, in essence, it was never concluded. The expense of this was viewed to be much more attractive than the loss of revenue the poor study results would likely have induced. You can read about publication bias here.

Some medical journals have attempted to reduce the risk of publication bias by encouraging that trials are officially registered (preferably prior to them starting, and hopefully before the study ends) as a precondition for publication. This way, it has become harder for the results of unknown studies to be buried. These new recommendations, set by the International Committee of Medical Journal Editors, came into force in 2005.

However, despite this step, it seems there is still potential for researchers and their sponsors muddy the waters. Here are some ways they can do this:

Don’t register the trial (as yet, it’s not mandatory to do so)

Register the trial, but wait till after it has been completed (this gives an opportunity to choose the ‘favourable’ outcomes to register)

Be unclear in the registration what outcomes are to be measured

Register certain outcomes, but present different outcomes in the published study

In a study published this week in the Journal of the American Medical Association, French, Canadian and British researchers attempted to quantify these problems exist in published medical literature [1]. They assessed this on the basis of 323 studies (in the areas of cardiology, rheumatology and gastroenterology) published in 2008 in the 10 medical journals with the highest impact factors.

Here, in summary, is what they discovered:

Less than half (45.5 per cent) of all trials had been adequately registered (i.e. registered prior to completion of the trial, with the measured outcomes clearly stated)

27.6 per cent of studies were not registered at all

13.9 per cent of studies were registered after they had been completed

12 per cent of studies were registered without a clear description of the outcomes being measured

Of the adequately registered trials, 31 per cent showed evidence of discrepancy between the outcomes that were registered and the outcomes the were published

In half of these studies it was possible to assess the influence these discrepancies had: in 82.6 per cent of these studies, there was evidence that the discrepancies favoured the publication of statistically significant results

So many trials remain to be adequately registered (which means that there is still much potential for publication bias). And of those studies that were adequately registered, there’s still a tendency for research (and likely, their sponsors) to shift the goalposts in terms of what the trial was seeking to show.

The authors of the JAMA study make some recommendations to help stamp out these sources of bias. They include:

The sponsor and principal investigator should ensure that the trial details are registered before enrolling participants.

The comprehensiveness of the registration should be routinely checked, especially regarding the reporting of important items such as the primary outcome.

Editors and, possibly, peer reviewers should systematically check the consistency between the registered protocol and the submitted manuscript to identify any discrepancies and, if necessary, require explanations from the authors.

Along with this, the authors point out that mandatory publication of results would also make it much harder for researchers and their sponsors to ‘hide information’.

References:

1. Mathieu S, et al. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302(9):977-984

3 Responses to Evidence that there is still lots of potential for study researchers and their sponsors to only let you see what they want you to see

  1. William Trumbower MD 5 September 2009 at 3:32 pm #

    Dr. Briffa Excellent Post. Currently, the only groups with enough money to do large scale research are the government or the drug companies. The gov. only researches what is politicly correct and when the results will support current gov. policy. (subsidies for grain production). The drug compancies have their obvious conflict of interest as you have shown. They also fund much of the journals thru their ads and they sponsor many of the medical mtgs held for physicians (along with surgical equip companies). Thus the information stream that good physicians use to learn how to be the best they can be has been tainted!

  2. Margaret Wilde 6 September 2009 at 3:04 pm #

    Many years ago I wrote to a psychologist working in a famous dental hospital in London. I told her about how the dental hospital near to where I live was labelling patients depressed or mentally ill after incompetent/negligent dental treatment had left them with chronic, severe toothache. In other words, these patients were being told that they were not really in pain, they were just depressed! I had hoped that she would do something to prevent this devious, dishonest practice.

    In her reply, however, she claimed that a research project proved that chronic toothache was really depression and sent me a copy of the paper. In the preamble to the paper it was stated that the purpose of the study was to prove that chronic toothache is really depression! And where it was found that subjects who were intended to be part of the research actually did have painful dental problems causing their pain, they were excluded from the research! – ‘Nuff said!

  3. Trinkwasser 11 September 2009 at 5:41 pm #

    Couldn’t agree more: “evidence based” medicine falls down when the evidence is little more than infomercials for the current profitable drug of choice.

    “Absence of evidence is not evidence of absence” – for example where are the studies on well controlled diabetics? We are legion on forums, blogs and newsgroups but routinely turned down for studies for being “too well controlled”, thus doctors have no “evidence” that it is possible to improve significantly on Conventional Wisdom.

    Do you know what became of a recent UK dietary study? Did it come up with the “wrong” results and get sat on, or are they still tweaking the statistics to “prove” that low carb doesn’t work?

Leave a Reply to William Trumbower MD Click here to cancel reply.