Research practice and chemicals policy: how science makes life difficult for regulators

April 29, 2011 at 9:02 am | Posted in Feature Articles | 6 Comments

Click here to download printer-friendly PDF version of this article.

There is a great deal of controversy and argument around whether or not the way chemicals are assessed for safety in the EU is adequately responsive to evidence that they may be causing harm. Leaving to one side lobbying by commercial and public-interest organisations, here we look at whether or not scientific practice produces the data regulators feel they need in order to make decisions about restricting the use of chemicals – and if not, then what can be done about it.

Chemicals testing flow chart

As an alternative to accepting into the risk assessment process peer-reviewed studies, the EU could solve its data issues by funding independent laboratories to produce data useful for risk assessment, regularly revisiting the process to make sure it reflects state-of-the-art research. Click to enlarge.

Decisions about which chemicals are safe rarely fail to attract controversy, with regulators under constant attack for either giving too much or too little credence to studies which suggest a chemical may be harmful. Dr Ruth Alcock of Lancaster University’s Environment Centre (UK) argues in a recent paper that one reason for this is that scientific research practices are poorly suited to the needs of regulators, leaving regulators unable to interpret new findings into the risk assessments on which EU chemicals regulation is based (Alcock et al. 2011).

Alcock cites the example of the flame-retardant deca-BDE as being no exception. Although deca-BDE is currently given the green light by EU safety assessment standards, Alcock reports polarised expert opinion about the safety of the substance, with toxicologists, regulators and chemists expressing attitudes about the risks it poses to health ranging from “obvious impacts […] on neurodevelopment” to “no direct evidence of harm at all”.

Risk assessment is based on two principal factors: estimates of human exposure to a chemical agent, and an assessment of the toxicity of the chemical at the likely level of exposure. The problem regulators face with deca-BDE is precisely a lack of consistent data in either area.

On exposure, different laboratories diverge greatly in their assessments of levels of deca-BDE in identical material samples – partly because the amounts are very small, and partly because deca-BDE breaks down rapidly under analysis, making it very difficult to obtain a reliable measure of its presence in the environment.

Understanding deca-BDE’s toxicity is similarly complex, with concern focusing on its ability to interfere with the healthy development of the brain. A substantial body of research produced by Professor Per Eriksson of Uppsala University (Sweden), shows strong evidence of neurotoxicity. However, doubts about his methodology has allowed enough uncertainty to persist that deca-BDE is not, from a regulatory perspective, considered as neurotoxic.

Without consistent, reliable data regulators are loathe to restrict the use of a substance, which is why legislators have yet to act unequivocally on deca-BDE, commissioning further assessments of exposure and toxicity without restricting the use of the substance.

The trouble is, where regulators want consistency and reliable protocols to give them definite answers about a substance’s potential for harm, researchers such as Eriksson need cost-effective methods for exploring new ways in which a substance may be harmful, understanding why that is the case, and for improving the predictive capacity of their models. To put it another way, university scientists tend to be engaged in exploratory research, whereas regulators want confirmatory research.

Exploratory research methods are rarely static, with the science continuously evolving, producing new knowledge and triggering new research programmes. At the same time, this constant flux limits the opportunity to judge overall reliability of results: two laboratories might produce the same results but with different methods, leaving the question as to which method, if either, is reliable or if both methods produce a false positive. Worse still, two methods may produce two different results, leaving regulators to ponder which experimental result to believe.

There is therefore an obvious mismatch between the needs of regulators and the practices of researchers. If regulators are trying to use exploratory research for confirmatory purposes, it is little wonder that so many of their decisions attract controversy and are open to accusations from both environment groups and commercial interests of not being sufficiently grounded in scientific evidence.

Reliability of data

One thing to which regulators are looking as a mark of reliable data is Good Laboratory Practice (GLP). GLP was established in 1978 by the US Environmental Protection Agency after a series of fraudulent chemical safety tests at commercial laboratories showed the need for a standard for data reporting and management. The standards ensure that outside auditors can evaluate any particular piece of work the laboratory does.

Compliance with GLP carries a great deal of weight with regulators, to the extent that both the US Food and Drug Administration and the European Food Safety Authority (EFSA) have treated two GLP studies as providing definitive proof that the controversial chemical bisphenol-A is safe, refuting a large body of peer-reviewed, non-GLP evidence that it is a reproductive toxicant (Myers et al. 2009).

Although it is true that detailed reporting means GLP tests easily replicated, regulators’ acceptance of GLP is not uncontroversial and may ultimately ride on a misunderstanding of what the standard guarantees.

The quality of a study is a product of its reliability and validity. Reliability is determined by the success with which independent research teams can produce the same results using the study’s techniques (something lacking in deca-BDE detection studies at the moment). Validity comes from study design and competence of execution. Both are essential for a study to be considered to reveal facts.

In order to be repeated and proven reliable and valid, a study has to be sufficiently well documented to allow a second independent laboratory to repeat the experiment and produce the same results. It is this sufficiency of documentation which GLP guarantees. Precise documentation, however, counts for nothing towards quality if a GLP study uses the wrong sort of animal, measures the wrong end-points for detecting an effect, or technicians make errors in e.g. removing organs from animals for examination. In all these cases, the study would be invalid even though it meets GLP standards.

At least one of the two GLP studies taken by the FDA and EFSA as exonerating BPA, Tyl et al. 2008, was heavily criticised for, amongst other issues, obviously over-heavy prostates retrieved from mice in the study, indicating either improper dissection or that the mice in the study were much older than reported. Since this means another laboratory would likely produce different results despite following the same protocols, the reliability of the Tyl study is in doubt even though it meets GLP standards.

Is peer-review a viable alternative to GLP?

There is more to the peer-review process than one researcher submitting their research to the scrutiny of others. In order to secure research funding, researchers have to demonstrate competence in the proposed area of study, use state-of-the-art experimental techniques, and submit their work for evaluation by independent experts before publishing in journals. On top of this, independent efforts are made to replicate findings are made, the possibility of refutation by these further encouraging honest and effective research practice.

Peer-review therefore functions as a set of safeguards which helps ensure that an overall body of research is more likely to reveal facts than fail to do so. Individual studies may be invalid and some research avenues may be red herrings, but these are normally identified and discarded by the system. The acceptance of invalid studies as valid and massively false bodies of research as true are aberrations rather than the norm.

Peer-review does not, however, amount to a formal process of validation. As a system it works because the results are generally reliable – however, there is no guarantee that any particular study within the system is itself reliable. Since regulators seem to want individual studies, not just the system as a whole, to produce reliable results, peer-review may not be a viable alternative to the use of standardised protocols.


Peer review was never designed with risk assessment in mind and so will never produce studies reliable enough for the existing demands of risk assessors. Unfortunately, standardised protocols such as GLP are not a short-cut to determining reliability of a study. What, then, might be the best way to deal with the complex evidence base produced by academic researchers? There appear to be several choices.

1.  Stop using peer-reviewed studies in risk assessment. Risk managers already receive a lot of criticism for not using enough peer-reviewed data; to make it a policy not to use it could be interpreted as perverse, failing even to address the problems with how science feeds into policy.

2.  Insist that academic laboratories become GLP-certified. GLP studies are 2-10 times more expensive to run than non-GLP studies while the extra cost adds little value to exploratory research (Apredica, retrieved 2011). The massive increase in cost of research funding which this would entail makes this financially unrealistic.

3. Encourage academic laboratories to do more corroborative studies. Few laboratories are equipped for corroborating another’s findings with new techniques and equipment: this has to be bought, staff have to be trained, and few laboratories will want to go to the expense of doing this while a technique is unproven. This option is also financially unrealistic.

4. Soften the demand for reliability of data in risk assessment. This would allow more peer-reviewed evidence to be introduced and be consistent with a precautionary approach to risk management, effective for preventing potential harm but at the likely cost of some unnecessary restrictions being placed on some chemicals. From an environmental health perspective this would make sense, though in the current climate is probably politically unrealistic.

5. Fund laboratories to replicate academic findings under standardised protocols. This amounts to a combination of (2) and (3): rather than pay for all laboratories to become GLP-certified and capable of corroborating new findings, the EU could fund the establishment of independent laboratories whose purpose is replicate the findings of academic studies under standardised conditions.

Option (5) would secure corroborative studies to standards amenable to risk assessment. Furthermore, if the funding also supported the development and validation of new assays, then testing procedures could keep pace with scientific knowledge. It is analogous to how the pharmaceutical industry moves from exploratory to confirmatory research, and arguably the most realistic option because it is the cheapest means for meeting the established needs of an existing risk management process.


RSS feed for comments on this post. TrackBack URI

  1. […] compared with those of risk assessors trying to draw firm conclusions about a chemical’s safety (H&E #37). The mismatch helps explain why so few studies by academic researchers are included in risk […]

  2. […] compared with those of risk assessors trying to draw firm conclusions about a chemical’s safety (H&E #37). The mismatch helps explain why so few studies by academic researchers are included in risk […]

  3. Thanks for an interesting and highly topical blog post. I have one comment regarding the suggested choices. One less costly, and perhaps more feasible approach to increase the use of data from the open scientific literature in regulatory risk assessment is to improve the reporting of the data so that it fulfill the requirement regulators have.
    We have conducted two studies regarding reliability and relevance of data. In the first study we evaluate existing reliability evaluation criteria and in the second we propose a new set of criteria that can be used both for reporting and evaluation of data.
    Ågerstrand M, Breitholtz M, Rudén C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data – A case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environmental Sciences Europe 23:17.
    Ågerstrand M, Küster A, Bachmann J, Breitholtz M, Ebert I, Rechenberg B, Rudén C. 2011. Reporting and evaluation criteria as means towards transarent use of ecotoxicity data for environmental risk assessment of pharmaceuticals. Accepted for publication in Environmental Pollution.

  4. Thank you for these references.
    I believe there is some concern that regulators underestimate the reliability or usefulness of the academic literature, while the demand for “relevance to risk assessment” obscures deep problems in the risk assessment process itself (such as not using up-to-date assays, or discarding evidence of harm because the evidence is not conducive to calculating a TDI).
    If this is the case, then making academic research more relevant to regulators’ needs can only be part of the solution – the regulators need to make changes to what they are demanding.
    I suppose what we will see is a compromise, so I’m looking forward to reading your papers – Ruth Alcock presents a strong case for changing research practices to meet the needs of regulators, so the compromise must to some extent lie in that direction.

  5. […] Research practice and chemicals policy: how science makes life difficult for regulators […]

  6. […] 3. Diversity of literature. The differences between academic research and regulatory findings has been discussed in the academic literature (see, for example, Alcock 2011, and a relatively early piece of writing by this author). […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at
Entries and comments feeds.

%d bloggers like this: