EDCs, Scientific Review & the “Manufacture of Uncertainty”

October 30, 2015 at 3:30 pm | Posted in Feature Articles | Leave a comment

EDCs, Scientific Review & the “Manufacture of Uncertainty”

What future for narrative reviews in chemical risk assessment, amidst calls for systematic approaches to synthesising evidence of risks?

Click here to download a printer-friendly PDF of this article.

Picture by tanakawho / flickr

Studies which investigate potential risks to health posed by chemicals vary in design, methodological quality, populations studied, and exposure and health outcomes considered. Even very large, highly-cited studies will be challenged or refuted over time, which is why decision-making (if it is to be based on the best available evidence) requires on-going reconciliation of studies which give different answers to the same question. Since it is impractical for individuals to conduct these reconciliations themselves, there is always demand for researchers to produce reviews of what the existing evidence says about risks to health posed by chemical compounds.

The problem is, these reviews themselves can be conflicting and controversial, and in these circumstances it can be difficult to know which interpretation of the evidence (if any) ought to be believed.

Highly topical at the moment is the debate about risks to health posed by endocrine disruptors. A major “state-of-the-science” review of the evidence relating to endocrine disruption, commissioned by the United Nations Environment Programme and World Health Organisation, was completed in 2012 (Bergman et al. eds. 2013). It is a lengthy, wide-ranging document which is difficult to summarise, but the story it tells seems to go more-or-less like this:

  • Many endocrine-related diseases and disorders are increasing in incidence;
  • 800 chemicals are either known or suspected to be capable of interfering with hormone receptors;
  • Exposure of human and wildlife populations to these chemicals are widespread;
  • There is some evidence associating exposure to these chemicals with increased incidence of endocrine-related diseases and disorders in humans;
  • Laboratory studies support the idea that EDC exposure could be contributing to incidence of endocrine-related diseases and disorders in people and wildlife;
  • Lack of research and appropriate test methods for identifying EDCs means the risks they pose to health are not adequately characterised and may be significantly underestimated;
  • Therefore a coordinated international effort is needed to define the role EDC exposure plays in current declines in human and wildlife health.

The report was fiercely criticised in a commentary published in early 2014 in the journal Regulatory Toxicology and Pharmacology (Lamb et al. 2014). It was argued as failing on five main grounds: for having unclear scope, in that the sense that the document seemed to be unclear in what it was trying to achieve as a review of the evidence of endocrine disruption; for being selective in its choice of data for developing its main claims about environmental health risks posed by EDCs; for being biased in how it interpreted the included data; for not distinguishing more relevant, better-conducted studies from less relevant, weaker ones; and for not having a method for consistently putting more weight on stronger evidence when drawing conclusions from the included data.

Lamb et al. concluded that “the 2012 report does not provide a balanced perspective, nor does it accurately reflect the state of the science on endocrine disruption”.

As one would expect the UNEP-WHO report authors are hitting back, publishing a forthcoming commentary which accuses Lamb et al. of “manufacturing doubt about endocrine disruptor science” (Bergman et al. 2015). They argue that Lamb et al. repeatedly misrepresent the content of their report through misleading summarisation and selective quotation of their methods and findings, and exaggerate the report’s methodological shortcomings via comparison with an arbitrarily high standard for conduct of evidence synthesis which no state-of-the-science report can be expected to meet. The upshot is a deconstruction which is “not particularly erudite” and “not intended to be convincing to the scientific community, but to confuse the scientific data”.

Stirring stuff. The problem is that, regardless of their motivation, the issues raised by Lamb et al. are generic and easy to make because they affect any review of the literature, while narrative reviews such as the UNEP-WHO report are particularly vulnerable to these criticisms (even without selective quotation). The principle challenges in interpreting narrative reviews are neatly summarised by Garg et al. (2008):

  • To what extent is the result of the review determined by the data collected, as opposed to the experience and beliefs of the reviewers?
  • How exhaustively has the available literature been identified and compiled?
  • Why are some studies given more emphasis than others?

It is to address these generic shortcomings in narrative reviews that systematic review methods for synthesising evidence have been developed in various fields, including social science, medicine and education. They are catching on in environmental health and chemical risk research and are acknowledged as desirable both by Lamb et al. and by the UNEP-WHO report authors – though there does seem to be some confusion as to the extent to which systematic review in a state-of-the-science review is feasible or desirable.

Lamb et al. themselves send mixed messages in their critical commentary, saying (p24): “The literature published on the potential endocrine activity of specific chemicals is extensive and beyond the scope of either the 2002 or 2012 reports. Neither report could be expected to undertake complete reviews for even a small subset of chemicals, but a systematic methodology would have ensured that a representative spectrum of the available literature was captured in the review.”

However, they also say (p28): “Any state-of-the science review should be a balanced and objective review of all of the available literature with identification of data gaps along with clear statements of the conclusions supported by the data and the inferences that cannot be supported.” And again: “Overall, any claims of trends indicating an endocrine cause must be supported by a systematic review of all relevant data regarding disease trends, exposure trends, and alternative explanations for observed statistical correlations.”

On the one hand, Lamb et al. appear to accept that some selectivity is going to be inevitable; on the other, they seem to be arguing that any selectivity at all, even in terms of statements of the inferences that cannot be supported, is intolerable. It is worth noting that the standard they propose is one which nobody has met: there is as yet no accepted systematic approach to synthesising evidence of the endocrine disrupting potential of a chemical; and the reviews which Lamb et al. highlight as examples of good practice in comparison to the UNEP-WHO report (such as the US Environmental Protection Agency’s draft review of issues surrounding non-monotonic dose responses, the European Food Safety Authority’s 2010 risk assessment of bisphenol-A, and the National Research Council’s review of adverse health effects of perchlorate ingestion) all fall very short of the systematic standard which Lamb et al. demand.

This confusion about what is desirable, what is necessary and what is achievable when it comes to making a state-of-the-science review fully systematic is partially echoed by some of the authors of the UNEP-WHO report in a useful 2014 “path forward” paper which advocates that “once developed and shown to produce non biased assessments, systematic review methods should be used to assess the EDC literature” (Zoeller et al. 2014). However, the paper also says that in light of the “impossibility of using [systematic review methods] for such a large undertaking, state-of-the-science reviews are likely to always require … narrative reviews”. (The Bergman et al. response paper is much less equivocal about the need for systematic review methods, stating: “We explicitly recommended [in the UNEP-WHO report] that ‘Efforts are needed to develop systematic and transparent approaches to identifying, evaluating and synthesizing the scientific evidence for endocrine disrupters that consider the science of endocrine action.’”)

This confusion is worth some comment. While systematic review might appear to be an arbitrarily high standard, it is a real standard in many research fields outside toxicology. It is therefore difficult to argue that it need not be adhered to. Furthermore, in comparison to systematic reviews, narrative reviews will always have a soft underbelly whereby the rigour of methods (in particular, the possibility of bias in selection and weighting of studies) has to be taken on trust rather than demonstrated. This is not to say that systematic reviews are a panacea for preventing controversy; the difference is that when debates get heated, stakes are high and vested interests seek to discredit the findings of evidence reviews, narrative reviews are simply much less equipped for preventing or resolving controversy than their systematic counterparts.

Where does that leave the UNEP-WHO report? It is difficult: dependent as it is on narrative methods, it is hard to see how (if it was their goal) Lamb et al. have not been successful in undermining confidence in the report – regardless of whether or not their critique is justified. On the other hand, if the conclusion of the UNEP-WHO report is that EDC risks are plausible but under-researched and potentially under-estimated (which appears to be the bottom-line), then as a reiteration of expert opinion in the absence of systematic review, it seems to do an important job: knowing what a group of experts think is useful, even if it is not necessarily decisive.

That the relatively modest conclusions of the report should be so threatening that they require lengthy deconstruction is perhaps interesting to note. It is also possible that the report is not in fact the target, rather the supposed consensus positions which are imputed to the report but which the authors of the report say are absent. Perhaps the ease of the deconstruction should be a worry: if generic criticisms of narrative reviews are legitimate and easy to make, what then is the value of narrative, state-of-the-science reports? If they are so vulnerable to generic critique that they can be dismantled as soon as articulated, is there a danger that with the advent of systematic review techniques, a less valuable but not valueless approach to synthesising science around endocrine disruption could be lost because of the politics of chemical regulation? That would be a shame: while SR methods are desirable, something presumably should be done to extract what value we can from less rigorous methods.

That said, it is not clear how we can just accept that Zoeller et al.’s view that, when data sets get big and issues get complicated, narrative review will be essential. While it is true that systematic review processes can currently only handle relatively small amounts of data, it is not clear that a mere increase in size of data set obviates the demand for transparency and rigour of methods in synthesising it.

Resolving this tension needs careful thought, but in primary research it would never be advocated that an experiment requiring a large budget should be done badly on a small budget; instead, the research problem would be broken down into manageable chunks and progress made by conducting smaller experiments in line with available resource and capacity. The size and complexity of a research project has no bearing on requirements of rigour in primary research (if anything, the more money being spent the more rigour is required); why should that be any different in secondary research? The intuition here is that the solution lies in intelligent breakdown of research synthesis problems into manageable chunks, not in throwing out the requirement for validity of method.

Overall, the lesson to learn (at least from the perspective of evidence synthesis) from the debate about the relative merits of the UNEP-WHO report on EDCs, is that it reveals a shared recognition of the need for systematic review methods for synthesising evidence of environmental harm in chemical risk assessment, but at the same time demonstrates a lack of clarity of vision about how this is to be implemented, particularly for complex research problems.

References and Recommended Reading

Bergman, Å., Becher, G., Blumberg, B., Bjerregaard, P., Bornman, R., Brandt, I., … Zoeller, R. T. (2015). Manufacturing doubt about endocrine disrupter science – A rebuttal of industry-sponsored critical comments on the UNEP/WHO report “State of the Science of Endocrine Disrupting Chemicals 2012.” Regulatory Toxicology and Pharmacology, 1–11. doi:10.1016/j.yrtph.2015.07.026

Bergman, Å., Heindel, J. J., Jobling, S., Kidd, K. A., & Zoeller, R. T. (eds) (2012). State of the Science of Endocrine Disrupting Chemicals – 2012. World Health Organization: Geneva, …. Retrieved from http://www.who.int/ceh/publications/endocrine/en/

Lamb, J. C., Boffetta, P., Foster, W. G., Goodman, J. E., Hentz, K. L., Rhomberg, L. R., … Williams, A. L. (2014). Critical comments on the WHO-UNEP State of the Science of Endocrine Disrupting Chemicals – 2012. Regulatory Toxicology and Pharmacology : RTP, 69(1), 22–40. doi:10.1016/j.yrtph.2014.02.002

Ulucanlar, S., Fooks, G. J., Hatchard, J. L., & Gilmore, A. B. (2014). Representation and Misrepresentation of Scientific Evidence in Contemporary Tobacco Regulation: A Review of Tobacco Industry Submissions to the UK Government Consultation on Standardised Packaging. PLoS Medicine, 11(3), e1001629. doi:10.1371/journal.pmed.1001629

Zoeller, R. T., Bergman, \AAke, Becher, G., Bjerregaard, P., Bornman, R., Brandt, I., … others. (2014). A path forward in the debate over health impacts of endocrine disrupting chemicals. Environmental Health, 13(1), 118. Retrieved from http://www.ehjournal.net/content/13/1/118/abstract

Leave a Comment »

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: