NHMRC Public Consultations

Skip Navigation and go to Content
Visit NHMRC website

NHMRC Draft Information Paper: Evidence on the effectiveness of homeopathy for treating health conditions submission

ID: 
22
Personal Details
First Name: 
Jenny
Last Name: 
Heywood
Questions
Q1. Is the draft Information Paper presented and written in a manner that is easy to understand?: 

I am an information specialist and educator who has been researching medical issues for a number of years. My sources are official publications, medical journals, and other publications by experts in the field of medicine.  I am not a homeopathist, but I am interested in homeopathy as an alternative therapy that has widespread use throughout the world, and I belong to the growing group who use alternative therapies.

Q1. Is the draft Information Paper presented and written in a manner that is easy to understand?

As it is written, this consultation draft brings NHMRC/HWC expertise or ethics or both into question.  Either that or it is poorly written, and widely mis-understood, as shown in media reports. It is not easy to understand for the following reasons:

1.1. It is CONFUSING in its use of the principle of a null hypothesis, breaching the principles of scientific method:

“The Homeopathic Working Committee (HWC) decided that the null hypothesis would be assumed, unless there is sufficient reliable evidence to demonstrate otherwise.” (App. C p298)  ”.  Instead of starting with a null hypothesis, the committee started with an assumption. This is in contravention of core scientific principles.[i] To make this assumption is neither ethical nor logical. “Not enough evidence to prove innocent” does not mean “guilty”.  “Not enough evidence to prove effective” does not mean “ineffective.” 

1.2.  It is CONFUSING because the review itself breaches ethical standards for analysis of data. “For each clinical condition, the null hypothesis was that homeopathy has no effect as a treatment for that condition.” (App. C p298)  This would be an unwarranted extrapolation of the data, claiming that homeopathy in total had no effect, when it has only been shown that the evidence could not be found to prove that a particular intervention had an effect.  This would be like saying that because the drug Fenofibrate was not associated with improved cardiovascular outcomes among patients with diabetes, no conventional intervention was.[ii]

1.2.1 It is confusing because the report itself fails a number of the criteria used to exclude trials of homeopathy from the review.  It does not meet the AMSTAR criteria 1, 2, 3, 5, 8, 9, 10, 11:

(1) It was not an “a priori” design; (2) Independence of data extractors is open to question; (3) The literature search and supplementation were too limited; (5) The list of reviews could have been included in the draft Information Paper, but was not; (8) The recommendations do not accurately reflect the data found in the reviews; (9) The methodology was inappropriate for combining a wide range of conditions; (10) Possibility of publication bias appears to have been ignored;    (11) Statements of conflicts of interest were not available.

1.3. It is confusing because it sets out to inform the Australian people, yet the only way they can access the critical information about the reviews and trials is to trawl through the data for themselves. The summary fails to provide this, in fact, it barely acknowledges any positive evidence for homeopathy. It fails to communicate how 343 articles provided by the Australian homeopathic community did not result in any positive statements about homeopathy in the review. It is hard to believe this.

If people want to “understand any possible benefits and risks,” which the Homeopathy Working Committee claims to be facilitating (p.4), they will have to re-examine the trials for themselves unless the committee undertakes to release more information. It would be in the public interest if these articles and the titles of the reviews at least were made freely available.

1.4. It is NOT EASY TO UNDERSTAND because the process is described in a way that makes it look deliberately biased from the start. NHMRC/HWC started from the null hypothesis that “homeopathy has no effect as a treatment...” (Appendix C p.298).  A selected working committee then employed an unnamed reviewer and a checker from a commercial research organization Optum, and took advice on methodology from the Australasian Cochrane Centre to perform a review of reviews of homeopathy, even though this review did not fit any of the Cochrane categories for an overview. (http://handbook.cochrane.org/ [accessed 16.5.14]) The Homeopathy Working Committee developed its own criteria for use exclusively with this review. Members of a final expert panel have not yet been named, but none of those involved to date have had expertise in homeopathy even though that is one of the criteria in the AMSTAR Toolkit.  

1.5. It is not easy to understand because DEFINITIONS have been changed and this is not adequately explained. The chief example is the word ‘homeopathy’. Because of the importance of the re-definition, it is ethically incumbent on NHMRC/HWC to state this, and re-state it frequently, particularly in its final assessment. The statement that this overview failed to find research of a high enough standard to prove that ‘homeopathy’ was effective in the clinical setting needs qualification - because there were extensive limitations on inclusion -  reviews had to be written in English, not address safety alone, not be about preventative homeopathy, use comparators that the reviewer was familiar with, must have more than 150 participants, and be random controlled trails published in a medical journal and included in a review  ....  It may be unwieldy, but it is essential, to qualify the word, “homeopathy” whenever it is used: ‘Homeopathy in a clinical setting, (this excludes homeopathic “vaccines”, preventative treatment, and issues of safety and cost-effectiveness, ...)’ And to state it prominently as part of the final assessment..

1.6. It is not easy to understand because the DEFINITION of “effectiveness” is very specific at times and not at others. On page 20 it is defined as “the extent to which a treatment works or not...” but in the interpretation on page 19 it is given a very specific meaning of “whether or not the NHMRC has found evidence, in a review that meets stringent conditions, that the treatment works.” This is not the same.

1.7. Not easy to understand because the DEFINITION of “independent” appears unique, for example, “independent” reviewers. Optum do not appear to regard themselves as independent.  They are a commercial enterprise that promises to deliver “competitive advantage”.  They contract to further the interests of the entity employing them, in this case NHMRC/HWC.  (http://www.optum.com/about.html [accessed 7.5.14]). The potential for bias is greatly magnified by this arrangement. (More detail in Ben Goldacre’s book Bad Pharma). To describe the Optum employee as “independent” without any evidence that he or she was not influenced by this arrangement stretches credibility.

1.8. Transparency: There appears to be a lack of transparency in the process based on a variable definition of what the word, “homeopathy” means, and what qualifies as acceptable review practice.  There is significant opportunity to bury unwanted evidence by ensuring it is excluded by the criteria specifically developed for selection in this review.

1.9  It is CONFUSING because of the different standards applied to this alternative form of medicine and pharmaceutical medicine. Having failed to find evidence that met its criteria, the committee appears to have concluded that this means that ‘homeopathy’ is not effective.  If so this is a fallacy.[iii]  If not, the draft is being seriously misunderstood. 

1.10. It is NOT EASY TO UNDERSTAND because, without doing further primary research, the consultation draft appears to claim to have discoveredconsensus in the medical literature on the value of homeopathy, when there is clear evidence that there is none – reviews and reports “have reached different conclusions.” (Consultation draft p.4)  It fails to reflect that  “... consensus across various government reports and systematic reviews has not been reached” or to reflect the “inconsistency among published reviews” (Optum p.9, 10)  The Consultation draft fails to convey the contradictions in the information assessed, or to show that there is both positive and negative evidence. Instead it implies strongly that it has examined all the evidence, and interprets what it failed to find to mean that homeopathy is not effective.  

1.11. It is NOT EASY TO UNDERSTAND because the claim is made that there was good reason to do this overview to take in the Australian perspective. This is not borne out by the review itself: it is essentially a review of already-published reviews from around the world but written in English. This does not add significant Australian perspective.

1.12. Wording is misleading: Media has reached the conclusion that all homeopathy is no better than placebo, and proves this with headlines like: “A waste of your money”, “Useless to human health”.  It is apparent that the interpretation on page 19 has been taken at face value without anyone checking if it reflects what the review found.

Summary for Q1:

The draft Information Paper is not presented and written in a manner that is easy to understand.  It fails to communicate even a need for its investigation.   It fails to communicate acceptable use of scientific method as required in random controlled trials, it fails to communicate how it could expect to produce a consensus statement when there is no consensus in medical publishing, it does not explain how it avoided bias in a process noted for producing biased results, and it does not give a statement of conflicts of interest. It has created an impression that no homeopathic treatment is effective, but the process does not follow established criteria for objectivity or reliability.  

The message that evidence has not been found to prove homeopathy effective, and that no conclusion could be drawn from this about homeopathy’s effectiveness is not getting through.  This message seems to have been obliterated by the interpretation on page 19, which goes far beyond the findings of the review itself.




[i]  If the null hypothesis is not rejected, then we must be careful to say what this means. The thinking on this is similar to a legal verdict. Just because a person has been declared "not guilty", it does not mean that he is innocent. In the same way, just because a null hypothesis is not rejected does not mean that the statement is true.http://statistics.about.com/od/Inferential-Statistics/a/The-Difference-Between-The-Null-Hypothesis-And-Alternative-Hypothesis.htm [accessed 8.5.14]

 

[ii]JAMA Internal Medicine. 05.05.2014.  doi:10.1001/jamainternmed.2014.1371

 

[iii]  If the null hypothesis is not rejected, then we must be careful to say what this means. The thinking on this is similar to a legal verdict. Just because a person has been declared "not guilty", it does not mean that he is innocent. In the same way, just because a null hypothesis is not rejected does not mean that the statement is true.http://statistics.about.com/od/Inferential-Statistics/a/The-Difference-Between-The-Null-Hypothesis-And-Alternative-Hypothesis.htm [accessed 8.5.14]

 

Q2. Does the draft Information Paper clearly outline how the evidence was reviewed and interpreted by the Homeopathy Working Committee?: 

2.1.  How data was really reviewed is not clear because the draft indicates an attempt to perform an overview when an overview is not appropriate - virtually impossible, in fact. The Cochrane Handbook describes an overview as “intended primarily to summarize multiple [Cochrane Intervention] reviews addressing the effects of two or more potential interventions for a single condition or health problem.” (italics added).  It can also be applied to non-Cochrane reviews, but a number of overviews are specified for multiple health problems.  There is no precendent[i] for this over-ambitious exercise, to cover 61 health conditions in one review. Criteria had to be created specifically and only for this investigation, with a warning not to apply it elsewhere. (Optum overview report p. 291)  The HWC chose to do an overview when it was not appropriate, and subsequently created its own method of assessment., after the overview was done.  This flawed process cannot provide answers scientifically, logically or ethically sound.  

The overview does not fit any of the categories in the Cochrane Handbook, and does not appear to have followed its principles.  

2.2  A review of reviews can make things clearer if it follows the Cochrane pattern, and this does not. It permits further confounding of outcomes because it depends on the quality, accuracy and comprehensiveness of the original reviews. Cochrane has procedures to avoid this but these were not followed.

2.3  The draft is not clear about methodology: The Cochrane Overview of reviews aims to serve as a ‘friendly front end’ to The Cochrane Library, and is essentially to bring together reviews for a single condition or health problem.[ii] Table 22.1a indicates that the HWC overview does not fit with any of the reasons or criteria for overviewing reviews.  Overviewing more than one condition with a range of interventions is not in the table.  To find them allineffective because conventional evidence cannot be found for some, and claim to have followed Cochrane methodology, is unethical.   This HWC overview is not compatible with Cochrane methods, and the involvement of the Australasian Cochrane Centre raises more questions than it answers.

2.4  How bias was dealt with is not made clear.  The Cochrane Collaboration provides guidelines on “Potential biases in the overview process: State the strengths and limitations of the Overview with regard to preventing bias. These may be factors within, or outside, the control of the Overview authors. The discussion might include whether all relevant reviews were identified and included in the Overview, whether all relevant data could be obtained, or whether the methods used (for example, searching, study selection, data collection and analysis) could have introduced bias.” These guidelines were not followed.

2.5   How the review established that medical publishing, and random controlled trials were the gold standard for comparison is not clear:

The exploitation and debasement of random controlled trials and of medical publishing have become general knowledge, confirmed recently by Bad Pharma and The trouble with medical journals, written by ‘insiders’. It is no longer defensible scientifically, logically or ethically to treat medical publishing, and random controlled trials in particular, as intrinsically reliable or the standard against which to measure all other evidence.

The consultation draft appears to be ignorant of the fact that much research which conventional medicine relies on is not of the standard the Homeopathy Working Committee is demanding for reviews of homeopathy. Trials purported to be random controlled trials frequently fail to reach these standards.[iii] Nor are reviews of published trials any more reliable.

2.6  The review is not clear about why unquestioning reliance is placed on reviewers. The introduction of error by reviewers is confirmed in a recent paper in the Journal of the American Medical Association JAMA:

After it was found in the ACCORD-Lipid study that Fenofibrate was not associated with improved cardiovascular outcomes among patients with diabetes mellitus:  nearly 20%  of biomedical journal articles [still] concluded it was effective , and 67% supported continued fibrate use.  JAMA Internal Medicine. 05.05.2014.  doi:10.1001/jamainternmed.2014.1371

Any review of these published papers, with 67% wrongly supporting continued use, would compound the error that has found its way into the journals.  And the accuracy of findings from a review of such reviews can only be guessed at.  Dependence on medical publishing and reviews of the literature can increase error. Reliable reporting cannot be assumed.  Medical journals are being used for a purpose other than to convey the findings accurately.  In fact, newspaper journalists were more accurate. It is clearly not scientifically sustainable to place such faith in one reviewer with one checker, particularly when they are not named or conflicts of interest divulged.

2.7  The draft is not clear because of apparent double standards with respect to trials. There is a double standard, one for alternative therapies, one for conventional pharmaceutical medicine. Much evidence in pharmaceutical medicine is not up to standard. For example, 90% of random-controlled-trial evidence considered by the Cochrane Review of flu vaccination fails the test of sound methodology, yet the NHMRC has not investigated this, and does not appear to have any issue with the finding that flu vaccination is no better than placebo. If NHMRC was consistent, it would follow through on this ready evidence of a huge cost for no benefit, before embarking on a long, complex, costly investigation into a therapy of which it has little understanding or expertise.

The 2014 Cochrane review of influenza vaccination found that “About 20% of the included studies (mainly cohorts) had a high risk of bias. Just under 10% had good methodological quality.” “Vaccination shows no appreciable effect on working days lost or hospitalisation.” and “At best, vaccines may only be effective against influenza A and B, which represent about 10% of all circulating viruses.”[iv]It is no longer true that random controlled trials published in medical journals are intrinsically reliable sources of information, and they are certainly not the “gold standard” by which to judge other forms of medicine. Bias contaminates results and burying or omitting data is commonplace.  In the case of this investigation into homeopathy, bias and the burying of relevant data have been given lip-service only, not seriously addressed, and this throws doubt on the findings. You can prove the sun shines 24/7 if you discard or ‘bury’ half the data.

2.8   How the evidence was reviewed and interpreted is not clear because the potential for bias is ignored and conflicts of interest are not stated. Cochrane Handbook chapter 22 (http://handbook.cochrane.org/ [accessed 16.5.14]) spells out what is reasonable for an overview:

Authors should report any present or past affiliations or other involvement in any organization or entity with an interest in the review that might lead to a real or perceived conflict of interest: see Chapter 4 (Section 4.5). “

I found no evidence that NHMRC considered this relevant. There is no statement of its well-documented history of antagonism to homeopathy, no statement of conflicts of interest for the Homeopathy Working Committee.  Without this, the value of the investigation and particularly of NHMRC’s interpretation of the assessment of the evidence, is wide open to charges of biased judgement and flawed results.

2.8.1 Conflict of interest does affect outcomes: In 2005 Marcia Angell MD, who had been chief-editor of the New England Journal of Medicine for 20 years said "The Medical establishment is complicit in the deception of the public and they have abdicated their duty of care to the public" (p.154).  In 2010 the Baxter report found so much conflict of interest in the Australian Commonwealth Health Department that policies are no longer being made in the public interest.   

2.9  It is not clear how criteria can be developed post-overview and meet criteria for transparency.  The criteria in this document were not developed a priori [as required of the homeopathy reviews by AMSTAR (App. B p.288], but rather were developed by the Homeopathy Working Committee with the assistance of the evidence reviewer over a number of months following the completion of the overview.  The criteria reflect the judgement of the Homeopathy Working Committee members about the key features of the evidence base that should be captured in each evidence statement.  The committee notes that: These criteria should not be treated as universal rules or principles that are applicable to all clinical contexts.  The criteria were developed in response to a specific activity– NHMRC’s overview of the effectiveness of homeopathy for treating clinical conditions in humans. This situation is obviously open to bias – creating criteria after the event - but this is not addressed in the documents.     

2.10  It is not clear how a process described as “not ideal”, and “subject to bias” became the basis for a blanket claim that homeopathy was “ineffective”.   The Optum review states that “... the assessment of ‘effectiveness’ based on statistical significance and not clinical significance is not ideal. This was, however, necessary due to the poor reporting (e.g. no reporting of primary outcomes, effect estimates or confidence intervals) and lack of analyses by the included systematic reviews and primary studies. Further, it was not possible to create a hierarchy of clinically relevant outcomes prior to conducting the overview (due to the number of conditions and systematic reviews included in the overview), and making post hoc decisions about the importance of outcomes is likely to be subject to bias.” (App. C p.288)  

2.10.1  This quote emphasises that the original choice of methodology was inappropriate. (2.1).  An overview is not appropriate for 61 different conditions. A number of systematic reviews are needed. The decision to do an overview compounded the difficulties, and meant that normal protocols could not be followed, and that there was increased liklihood of bias.

2.10.2 Bias can confound the outcome no matter what the quality of the process or the data, and  subjective assessments were embedded in the process - designed to depend on judgement (App. C p.298). What features of the evidence to capture was a judgement of the Homeopathy Working Committee, not established criteria.  Assigning a level of confidence was based on judgement (App. C p.296). This is the largely the judgement of a single reviewer working for a company which aims to provide the results wanted by whoever engages it, in this case, NHMRC/HWC. The bias of the NHMRC is a matter of record.

Summary for Q2: The draft fails to communicate clearly an appropriate methodology.  An overview is not appropriate for 61 different conditions. A number of systematic reviews are needed.  This choice led to difficulties, and meant that normal protocols could not be followed, and that there was increased liklihood of bias.

The ad-hoc criteria allowed “reliable (good-quality, well-designed)” studies to be excluded if they did not make up the majority of the studies (reviews?) for that condition. The reliability of random controlled trials as the gold standard was assumed despite increasing concerns about their quality, recent notable examples being Cochrane reviews into Tamiflu and flu vaccination. Conflicts of interest are not divulged, there was failure to consult widely with the homeopathic community.  The Homeopathy Working Committee has apparently endorsed an NHMRC “interpretation of the assessment” on page 19 that does not reflect the findings in the rest of the draft information paper. What it is based on is not divulged.




[i] There is no relevant standard relating to the development and content of evidence statements for a large number of clinical conditions. Cochrane Handbook does not allow for this. Criteria were developed by NHMRC specifically for this overview of homeopathy, and should not be treated as applicable elsewhere. (App. C p.291)

 

[ii]  (Cochrane Handbook of Systematic Reviews of Interventions http://handbook.cochrane.org/ [accessed 16.5.14]). 

 

[iii]Vaccines to prevent influenza in healthy adults Updated Demicheli V, Jefferson T, Al-Ansary LA, Ferroni E, Rivetti A, Di Pietrantonj C Published Online: March 13, 2014 http://summaries.cochrane.org/CD001269/vaccines-to-prevent-influenza-in-healthy-adults [24.3.14]

 

[iv]Vaccines to prevent influenza in healthy adults Updated Demicheli V, Jefferson T, Al-Ansary LA, Ferroni E, Rivetti A, Di Pietrantonj C Published Online: March 13, 2014 http://summaries.cochrane.org/CD001269/vaccines-to-prevent-influenza-in-healthy-adults [24.3.14]

 

 

Q3. Is there additional evidence on the effectiveness of homeopathy for the treatment of clinical conditions in humans that needs to be considered? To be considered in public consultation any additional evidence must:: 

3.1  It is not that additional evidence needs to be considered but rather that the evidence already to hand needs to be expressed in a form that is unbiased, and readily accessible to the Australian public, neither of which can be said of the present report.  

3.2  If the NHMRC did not have the resources to do a genuine review of homeopathy involving the Australian profession, I suggest it should not have embarked on this fruitless exercise, doomed by wrong choice of methodology. But having failed to find the necessary evidence, the logical next step is to get the necessary studies done.

3.3  I ask that the NHMRC undertake to make evidence on homeopathic treatments widely accessible so people can make informed decisions about specific conditions  -  not only the trials described as “good quality, well-designed and with enough participants for a meaningful result” in the present review (Consultation draft p.14),  but the whole body of evidence so Australians can judge for themselves.

3.4  I askthat a further review be done of homeopathic ‘vaccines’ as this was excluded from consideration, and, from my observations, it is of interest to an increasing number of Australians.  

SUMMARY

This review was carried out according to scientific principles .................................. .WRONG

Instead of a null hypothesis, it started with an assumption. Failing to find enough evidence that met its stringent requirements, it concluded that the assumption must be correct. A genuinely scientific conclusion would have been that evidence could not be found to prove that homeopathy was effective, and more evidence was needed.

An overview was appropriate .......................................................................................WRONG

It does not fit the criteria for an overview principally because it deals with 61 health conditions and an overview (as described in the Cochrane Handbook) is designed to deal with one. This meant that established protocols could not be used and untested new ones had to be developed. Significant scope for bias resulted.

Established criteria and processes were followed .............................................................WRONG

Decisions about the process were not made ‘a priori’, but during the process, and were created specifically for this review. 

The homeopathic community was fully involved in deliberations.......................................WRONG

Involvement of the homeopathic community in decision-making was minimal.

Conflicts of interest of committee members were published ..............................................WRONG

This omission casts doubt on the impartiality of the committee which had oversight of the entire process, and was involved in developing criteria for exclusion of data.

All relevant evidence was included.....................................................................................WRONG

Major aspects of homeopathy such as preventative treatment, and homeopathic ‘vaccines’ were excluded. As were any reviews not in English, and trials of less than150 participants. These exclusions cannot be justified on scientific, logical or ethical grounds.

The “interpretation of the assessment” was based on the findings of the review...............WRONG

The NHMRC interpretation of the review did not reflect the findings published in the draft information paper, nor did it provide a useful summary of the wealth of data reviewed.  

Page reviewed: 11 March, 2015