NHMRC Public Consultations

Overall finding (p.10)

‘NHMRC concludes that the assessment of the evidence from research in humans does not show that homeopathy is effective for treating the range of health conditions considered.’

This sentence – the most important of the entire Information Paper – is not easy to understand. It is highly likely that the average reader will misunderstand it. To academics it is clear that the NHMRC is drawing the conclusion that there is an ‘absence of evidence’, but the average reader (at whom this document is targeted) is likely to misinterpret this statement as saying that the NHMRC has found ‘evidence of absence’ of a clinical effect – an entirely different finding.

This is not just supposition. The difficulty most people have in understanding this conclusion has already been demonstrated by the flurry of inaccurate media reports which followed release of the draft Information Paper. The NHMRC’s overall finding has already been widely misinterpreted in headlines such as ‘Homeopathy is no more effective than a placebo’ (The Guardian, UK, 8 April 2014) and ‘ Government researchers conclude that homeopathic therapies do not work.’ (The Scientist, 14 April 2014).

If a single sentence summary such as this is going to be provided in the final Information Paper, it must accurately reflect the entirety of the NHMRC’s overall findings, as described in the following bullet points (p.10):

• ‘For some health conditions, homeopathy was found to be not more effective than placebo.'
• For other health conditions, some studies reported that homeopathy was more effective 
than placebo, or as effective as another treatment, but those studies were not reliable.
• For the remaining health conditions it was not possible to make any conclusion about whether homeopathy was effective or not, because there was not enough evidence.’


It is always a challenge to communicate scientific findings clearly and accurately to the media and general public, so perhaps the authors have over-reached here in attempting to capture all of the above in a single sentence, which the general public can understand and will not be open to misinterpretation?

The Information Paper’s overall finding summary statement will be the ‘take home message’ from the NHMRC’s 600 page-plus evidence review and associated documents. It is therefore essential that this summary is amended to prevent further misinterpretation of the NHMRC’s work.

Fortunately this consultation process provides the perfect opportunity to rectify this problem, which stems from the fact that several important points are not clearly reflected in the ‘Overall finding’ summary statement, in particular,

• that the amount of high quality evidence available in the field is small and 
• that there are some positive studies, but these need to be repeated by independent teams for confirmation of their findings and/or repeated on a larger scale.
  This is using layman’s terms, but quoting the NHMRC (p.10-16).

Therefore, to make the ‘Overall finding’ easy to understand and beyond misinterpretation in future, we strongly recommend that this conclusion be amended as follows:

‘NHMRC concludes that there is insufficient evidence from research in humans to establish whether homeopathy is effective, or not, for treating most of the health conditions considered. Some studies showed homeopathic treatments to be effective for certain conditions, but these need to be repeated by independent teams for confirmation of their findings. In other instances positive studies involved a small number of patients, so they would need to be repeated on a larger scale to confirm the results before the treatments were recommended for widespread use.’

Overall finding (contd, p.10)

‘There were no health conditions for which there was reliable evidence that homeopathy was effective. No good-quality, well-designed studies with enough participants for a meaningful result reported either that homeopathy caused greater health improvements than a substance with no effect on the health condition (placebo), or that homeopathy caused health improvements equal to those of another treatment.’

This paragraph is not easy to understand because it is misleading. It does not give the reader an accurate understanding of how these conclusions were determined by the review process used by the NHMRC. This paragraph needs to capture the following factors, using wording which can be understood by the general public:

1. The approach used by the NHMRC involved analysis of mixed data sets (see below for
     further explanation)

2. The nature of the inclusion criteria applied and thus the specific limitations of this review

3. The NHMRCs definition of ‘enough participants’ was N=150

4. The NHMRC dismissed as ‘unreliable’ trials with N<150, even if they have statistically significant findings

5. The NHMRCs definition of ‘unreliable’ includes good quality studies repeated multiple times by one research team, but not yet been repeated by other independent teams

6. The NHMRCs definition of ‘unreliable’ includes good quality single studies that have not yet been repeated.

To accurately and clearly describe how the evidence was reviewed and interpreted, this paragraph would therefore need to be amended as follows:

‘For the 61 health conditions considered, if we consider only prospective, controlled trials published in English, and discount all trials with less than 150 participants (even if they had positive statistically significant results), and if we discount positive trials that have not yet been repeated by other teams of researchers, and if we then combine all trial results for each condition, we can say that there was no reliable evidence demonstrating that homeopathy was effective.’

Of the six issues identified above which need to be included in the amended version of this paragraph, the first is the most important i.e. analysis of mixed data sets.

The NHMRC reviewers considered the results of all trials for one condition together as a whole, despite the fact that the individual studies may have been assessing different types of homeopathic treatment. This fundamental error explains why the NHMRCs has failed to find any ‘reliable’ evidence that homeopathy is effective for any of the 61 conditions under consideration.

The NHMRC reviewers asked, Is homeopathy effective for condition Y? working from the premise that for a given condition, a positive trial showing one homeopathic treatment is effective is somehow negated by a negative trial which shows that a completely different homeopathic treatment is not effective. This is a bizarre and unprecedented way of assessing the effectiveness of interventions. In conventional research the question asked would of course be, Is treatment X effective for condition Y? not, Is conventional medicine effective for condition Y when you combine all trial results on all drugs tested?

Some treatments work, some don’t. The whole point of medical research is to establish which treatments are useful and which are of no value. This is no different in homeopathy.

Unfortunately this basic error by the NHMRC means that their findings tell us nothing about which homeopathic treatments do and don’t work for which conditions, making this whole exercise of questionable value. Either this needs to be conveyed in the Information Paper in an easy to understand way or the evidence needs to be re-analysed correctly i.e. by looking at each specific treatment separately.

The Homeopathy Research Institute (HRI) strongly recommends the latter, as it would enable the NHMRC review to provide extremely useful results, preventing the work done to date from being completely wasted.

The primary clinical research question (Overview Report, 3.1, p.13) should therefore be changed from,

‘For patients with a specific clinical condition, is homeopathy an effective treatment, compared with no homeopathy/other treatments?’

to,

‘For patients with a specific clinical condition, is any homeopathic treatment effective, compared with no homeopathy/other treatments?’

The larger and more heterogeneous the set of primary studies for a given medical condition, the more this approach will have distorted the conclusions drawn; the smaller and more homogeneous the evidence base, the less it will have effected the outcome. Two examples where the combined analysis approach has lead to highly inaccurate conclusions being drawn are childhood diarrhoea and allergic rhinitis.

In the case of childhood diarrhea, the NHMRC’s evidence statement is as follows (Overview report, Childhood diarrhea, p.38):

‘The one medium-sized, good-quality trial (292 participants) did not detect a difference between combined homeopathy and placebo in the treatment of children with diarrhoea.

The studies of individualised homeopathy are of insufficient quality and/or size to warrant further consideration of their findings. LOC: Low - moderate.

Based on the body of evidence evaluated in this review combined homeopathy is not more effective than placebo for the treatment of children with diarrhoea and there is no reliable evidence that individualised homeopathy is more effective than placebo for the treatment of children with diarrhoea.’

The NHMRC’s approach has lead to two conclusions:

1. The entire approach known as ‘combined homeopathy’ does not work for children with diarrhoea (i.e. giving the same medicine, containing multiple ingredients, to all patients)

2. There is no reliable evidence that individualised homeopathy works for children with diarrhoea (i.e. treatment by a homeopath, involving an individualised prescription tailored to the symptoms of each patient)

However, these conclusions are not supported by the evidence.

‘The one medium-sized, good-quality trial (292 participants) did not detect a difference between combined homeopathy and placebo in the treatment of children with diarrhoea.’

This refers to a trial testing a combination or ‘complex’ homeopathic medicine containing Arsenicum, Calcarea carbonica, Chamomilla, Podophyllum and Sulphur in 30c potency (Jacobs et al. 2006). The negative result tells us that this particular homeopathic medicine is no better than placebo. However it does not tell us whether other medicines with multiple ingredients (‘combined homeopathy’) work or not for this condition, so cannot justify the NHMRC’s conclusion that the entire approach is ineffective.

‘The studies of individualised homeopathy are of insufficient quality and/or size to warrant further consideration of their findings.’

This is only a valid statement because, rather than considering the existing meta-analysis of trials on individualised homeopathy for childhood diarrhoea (Jacobs et al, 2003), the reviewers have instead only assessed the three individual trials within that meta-analysis. By doing this, the reviewers can conclude that the trials are ‘unreliable’ as they have less than the N=150 participants required by the NHMRC to be classified as large enough to be reliable: N=126 (Jacobs et al, 2000), N=92 (Jacobs et al, 1994) and N=34 (Jacobs et al, 1993).

The Jacobs et al. 2003 meta-analysis was excluded on the following basis: ‘Unable to assign a level of evidence - non-systematic review. Wrong research type or publication type.’ However, Jacobs reviewed all available Level II studies on individualised homeopathy for childhood diarrhea – RCTs which she had previously carried out – and conducted a meta-analysis. This legitimate systematic review and meta-analysis meets the inclusion criteria and should have been considered by the authors.

Based on the pooled results within this meta-analysis (n = 242) the study concludes that individualised homeopathy is more effective than placebo (p = 0.008), providing a 0.66 day reduction in the duration of the condition.

The evidence therefore actually supports the following conclusions:

1. The combination medicine Arsenicum, Calcarea carbonica, Chamomilla, Podophyllum and Sulphur in 30c potency is ineffective for childhood diarrhea
2. Individualised homeopathic treatment is an effective treatment for childhood diarrhea (p = 0.008).       

In the case of allergic rhinitis, the NHMRC reviewers’ conclusion for this condition is as follows:

‘Based on the body of evidence evaluated in this review there is no reliable evidence that homeopathy is as effective as the other therapies for the treatment of people with allergic rhinitis.’

By contrast, if the evidence is analysed appropriately by specific treatments, we see that it supports the following very different conclusions:

1.  The non-individualised homeopathic medicine Galphimia glauca is effective for allergic rhinitis (Ernst 2011; Wiesenauer & Lüdtke1996)

2. The non-individualised isopathic medicine Betula (made from Birch pollen) is not effective for allergic rhinitis (Aabel 2000)

3.  The non-individualised isopathic medicine Pollen 30c is effective for allergic rhinitis (Reilly et al.1986).

These examples highlight the need for amendment of the ‘Overall findings’ of the Information Paper prior to it being finalised.

Overall findings (contd, p.10)

• ‘For some health conditions, homeopathy was found to be not more effective than placebo.'
   
• For other health conditions, some studies reported that homeopathy was more effective 
than placebo, or as effective as another treatment, but those studies were not reliable.
   
• For the remaining health conditions it was not possible to make any conclusion about whether homeopathy was effective or not, because there was not enough evidence.’


This section is not easy to understand because the reader will not appreciate that the term ‘homeopathy’ has been used to mean all different homeopathic treatments considered together. As stated above, the only valid way to clearly present the findings of this review is by treatment, then condition.

It is also unclear that these findings, according to the NHMRC review, are all based on unreliable evidence, not just the positive results (‘The quality of the evidence was generally low, so it was not possible to be confident that the evidence was reliable’ - Findings of the NHMRC Overview, p.11, para 6).

For clarity and balance we therefore recommend that these points be amended as follows:

• ‘Some homeopathic treatments for some health conditions were found to be not more effective than placebo, but the quality of evidence was generally low, so it was not possible to be confident that the evidence was reliable.'
   
• For other health conditions, some studies reported that some homeopathic treatments were more effective 
than placebo, or as effective as another treatment, but the quality of evidence was generally low, so it was not possible to be confident that the evidence was reliable.'
   
• For some health conditions it was not possible to make any conclusion about whether any homeopathic treatments were effective or not, because there was not enough evidence.’


Findings of the NHMRC Overview (p.12-13)

‘Homeopathy compared with placebo
 
For 13 health conditions, homeopathy was reported to be not more effective than placebo.

For 14 health conditions, some studies reported that homeopathy was more effective than placebo, but these studies were not reliable [....] they would need to be confirmed by other large, well-designed studies.(1)

For 29 health conditions, only one study that compared homeopathy with placebo was found, and each of these studies was unreliable [....] For these conditions, it was not possible to make any conclusion about whether homeopathy was effective or not.

Homeopathy compared with other treatments

For 8 health conditions, some studies reported that homeopathy was as effective as another treatment, or more effective than another treatment, but these studies were not reliable. [....] they would need to be confirmed by other large, well-designed studies.

For 7 health conditions, only one study that compared homeopathy with another treatment was found, and each of these studies was unreliable. [....] For these conditions, it was not possible to make any conclusion about whether homeopathy was effective or not.’

This extremely important section of the Information Paper is not easy to understand, because without going to the Overview Report and making significant effort, the reader cannot tell which conditions are in each of the five result categories described above, or understand the reasons why particular studies were unreliable. This is important as some factors used by the NHMRC to determine reliability (p. 5) are far more serious than others.

These findings are of great value to the general public, decision-makers and academics alike, but the lack of detail provided prevents the information from being easily accessed and used. Once the NHMRC has re-analysed the data by treatment and condition, and adjusted the figures to reflect other input from the public consultation process, we therefore strongly recommend that this section is amended as follows (with X marking point for insertion of the NHMRC’s new results):

‘Homeopathy compared with placebo

For X health conditions, X homeopathic treatments tested were reported to be not more effective than placebo. These were:

• [Insert list of conditions and treatments]

For X health conditions, some studies reported that certain homeopathic treatments were more effective than placebo. These studies would need to be confirmed by other large, well-designed studies before considering widespread use of the treatments tested(1) i.e.:

• Galphimia glauca for allergic rhinitis (Wiesenauer & Lüdtke 1996)
• Pollen 30c for allergic rhinitis (Reilly, 1986)
• Individualised homeopathic treatment for childhood diarrhoea (Jacobs et al. 2003)
• Individualised homeopathic treatment for otitis media (Jacobs 2001 – described by the NHMRC as a good quality double-blind RCT, n=75)
• [Insert list of other conditions and treatments identified by NHMRC]

For X health conditions, only one study that compared a homeopathic treatment with placebo was found, and each of these studies was unreliable [....] For these conditions, it was not possible to make any conclusion about whether homeopathy was effective or not. These conditions included:

• [Insert list of conditions, treatments and reason for being ‘unreliable’]

Homeopathy compared with other treatments

For X health conditions, some studies reported that certain homeopathic treatments were as effective as another treatment, or more effective than another treatment. These studies would need to be confirmed by other large, well-designed studies before considering widespread use of the treatments tested(1). These treatments included:

• Individualised homeopathic treatment for otitis media in children (Sinha et al. 2012, N=81 but trial showed no significant difference between conventional treatment and homeopathic treatment (p=0.247))
• Vertigoheel for vertigo (Schneider et al. 2005)
• [Insert list of other conditions and treatments identified by NHMRC]

For X health conditions, only one study that compared a homeopathic treatment with another treatment was found, and each of these studies was unreliable. [....] For these conditions, it was not possible to make any conclusion about whether homeopathy was effective or not.’ These conditions included:

• [Insert list of conditions, treatments and reason for being ‘unreliable’]

NHMRC’s approach to assessing health evidence (p. 5, para 3)

• there are enough participants to be reasonably confident that, if there is a bigger change in the health condition in one group, this is not just due to chance.

This is not easy to understand as it incorrectly implies that smaller trials can never exclude chance (false positive results). Researchers know that the larger the trial, the more likely one is to detect a clinical effect (and for the results to reach statistical significance), so in fact, if a clinical effect is detected in a small trial with statistically significant results, this is actually a more impressive result.

To explain this concept simply to the general public, we recommend that this bullet point be expanded as follows:

• there are enough participants to be reasonably confident that, if there is a bigger change in the health condition in one group, this is not just due to chance. However if statistical analysis shows that the result of a trial is ‘statistically significant’, this means that the effect of the treatment is likely to be real, not just due to chance, even with a small number of participants.

How did NHMRC find evidence about homeopathy? (p.6, para 2)

‘The NHMRC’s assessment was guided by a committee of experts appointed in 2012 (see The Homeopathy Working Committee).’

This sentence does not clearly outline how the evidence was reviewed, because the reader would assume from this sentence that the evidence was reviewed by experts in the topic under discussion i.e. homeopathy research.

Considering the crucial role played by The Homeopathy Working Committee in overseeing the review process conducted by Optum and giving external input on their findings, it is astonishing to discover that not one committee member has any expertise in homeopathy research.

The ramifications of this unusual decision are considerable. For example, the serious methodological error of analysing mixed data sets could have been prevented if anyone with experience in homeopathy research had been included in this process.

Although a link is provided to the committee membership information, the identity of who did this work is such an important part of the description of how evidence was reviewed, that it should be provided in the Information Paper itself. We therefore recommend that this sentence be amended as follows:

‘The NHMRC’s assessment was guided by a committee, appointed in 2012, comprising a general practitioner, a rheumatologist, a neuroscientist, a consumer representative, a pharmacist, a conventional medical research scientist and the Australian Government Chief Medical Officer (see The Homeopathy Working Committee). Usually when conducting a systematic review of this kind, the committee would include several experts in the field being studied, but in this instance the NHMRC chose not to include any experts in homeopathy or homeopathy research because [insert NHMRC justification for this decision].’

Overview of systematic reviews (p.6, para 3)

‘For each health condition, the research group collated the findings of the systematic reviews and assessed the quality and reliability of the evidence….’

This does not clearly outline how the evidence was interpreted because the average reader will have no understanding from the description of the methodology given here (or anywhere else in the Information Paper) about the inherent risk of bias involved in carrying out a systematic review of systematic reviews.

The fact that two layers of interpretation of the findings by the original authors is involved is something which needs to be made clear here, along with details to explain how the NHMRC established that their reviewers were truly independent and un-biased. This is a crucial issue as it determines the reliability of the NHMRC review.

Both for clarity, and to ensure that people can have confidence in the NHMRC’s findings, we therefore recommend that this paragraph be amended as follows:

‘For each health condition, the research group collated the findings of the systematic reviews and assessed the quality and reliability of the evidence. When a systematic review is carried out, it is important that the researchers involved interpret the findings of the trials they are reviewing totally objectively, without introducing strong opinions of their own which could influence their findings (known as bias).

In April 2011 the NHMRC’s position on Homeopathy – that we believed the practice of homeopathy to be unethical because our assessment of the available evidence had shown that

it didn’t work – became known to the public. However we have ensured that this opinion held by the NHMRC did not influence how the reviewers we hired assessed the evidence, by [insert explanation as to how reviewer lack of bias was ensured]. This means that you can trust the NHMRC to have conducted this review fairly, without prejudice against homeopathy.’

Literature provided by homeopathy interest groups and individuals (p. 6-7)

‘Only the types of evidence that were included in the overview (prospective, controlled studies) were assessed in detail…..’

This does not clearly outline how the evidence was reviewed as it omits a particularly important inclusion/exclusion criterion – that all non-English language studies were excluded. It also fails to convey to the average reader the potential of such criteria to influence the overall findings. For clarity we therefore recommend that this paragraph is amended as follows:

‘Only the types of evidence that were included in the overview
(prospective, controlled studies published in English) were assessed in detail. All studies published in other languages – whether positive or negative – were excluded…..’

Evidence included in the overview (p. 10-11)

‘…NHMRC took a range of factors into account when considering
the evidence in the systematic reviews: [….]
- whether studies included enough participants to provide meaningful results…’

This text does not clearly describe how the evidence was reviewed because it does not tell the reader what ‘enough participants’ means or why the NHMRC used this definition. Only by reading the Overview Report can one discover that the NHMRC defined the size needed for a reliable study as N=150, and there is no justification given for this definition, even in the Overview Report.

The public need to understand this definition as it has a major impact on the NHMRC’s findings; decision-makers will require an explanation demonstrating the scientific validity of using n=150, along with the rationale for dismissing statistically significant results in trials under n=150. We therefore recommend the following amendments:

‘…NHMRC took a range of factors into account when considering
the evidence in the systematic reviews: [….]
- whether studies included enough participants to provide meaningful results. The NHMRC decided that trials with 150 participants were large enough, whilst those with fewer participants were too small to be reliable. This number was chosen because [insert NHMRC’s reason for using N=150 and justification of the scientific validity of this decision]. Trials with less than 150 participants which had statistically significant positive results (meaning that the trial showed homeopathy worked and this was unlikely to be due to chance) were discounted because [insert NHMRC’s reason for doing this].’

Evidence included in the overview (p. 11)

‘For each health condition, all the available evidence was grouped together to form a body of evidence on that condition. A body of evidence was considered more reliable if it included studies that were high quality, well designed and with enough participants to make its results meaningful. A body of evidence was considered less reliable if there were very few studies, or if the studies were poor quality, badly designed, or included too few participants.’

Unless the NHMRC adjusts its method of analysis as recommended, this paragraph would need to be amended to give a clear description of how the evidence was reviewed i.e.:

‘For each health condition, all the available evidence was grouped together to form a body of evidence on that condition. Normally, trials on different treatments would then be looked at separately (so the evidence can show us which treatments do and don’t work), but instead of doing this, all trials on all different medicines and treatments were grouped together and analysed as a whole. This is not usually done in medical research because the results cannot be used to say which treatments work and which do not, but the NHMRC chose to do this because [insert justification].

A body of evidence was considered more reliable if it included studies that were high quality, well designed and with enough participants to make its results meaningful. A body of evidence was considered less reliable if there were very few studies, or if the studies were poor quality, badly designed, or included too few participants and had not reached statistically significant results.’

References

Aabel, S. (2000). "No beneficial effect of isopathic prophylactic treatment for birch pollen allergy during a low-pollen season: a double-blind, placebo-controlled clinical trial of homeopathic Betula 30c." British Homeopathic Journal 89(4): 169-173.

Ernst E. (2011). Homeopathic Galphimia glauca for hay fever: A systematic review of randomised clinical trials and a critique of a published meta-analysis.

Jacobs J. et al. (2006). "Homeopathic combination remedy in the treatment of acute childhood diarrhea in Honduras." Journal of alternative and complementary medicine 12(8): 723- 732.

Jacobs J. et al. (2003). Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials, Pediatr Infect Dis J, 22:229–34

Jacobs, J. et al. (2000). "Homeopathic treatment of acute childhood diarrhea: results from a clinical trial in Nepal." Journal of alternative and complementary medicine 6(2): 131-139.

Jacobs J. et al. (1993). "Homoeopathic treatment of acute childhood diarrhoea: a randomized clinical trial in Nicaragua." British Homeopathic Journal 82: 83-86.

Jacobs J. et al. (1994). "Treatment of acute childhood diarrhea with homeopathic medicine: a randomized clinical trial in Nicaragua." Pediatrics 93(5): 719-725.

Reilly, D.T. et al. (1986). "Is homoeopathy a placebo response? Controlled trial of homoeopathic potency, with pollen in hay fever as model." Lancet 2(8512): 881-886.

Schneider et al. (2005). Treatment of vertigo with a homeopathic complex remedy compared with usual treatments – a meta-analysis of clinical trials, Arzneim.-Forschung. 55(1) 23-29.

Wiesenauer & Lüdtke (2006). A meta analysis of the homeopathic treatment of pollinosis with Galphimia glauca. Forsch. Komplementärmed. 3: 230-234

The following new evidence should be considered before finalising the Information Paper:

1. Sinha et al. Randomised controlled pilot study to compare Homeopathy and conventional therapy in Acute Otitis Media. Homeopathy 2012, 101: 5-12.

It is assumed that this study was missed by the NHMRC literature search due to its recent publication date. Although it was brought to the reviewers’ attention by an external party

and was considered in the Review of Submitted Literature, it was excluded on the basis that,

‘[…] this is a self-selected study and other literature concerning the effectiveness of homeopathy for otitis media has not been systematically retrieved.’ 

This is not a legitimate reason for exclusion. Sinha et al. 2012 fits the inclusion criteria – it is good quality comparative RCT on a condition for which a systematic review exists. It is unreasonable to downgrade the relevance of any study by categorising it as a ‘self-selected study’ simply because the NHMRC did not identify it themselves. The secondary reason the reviewers gave as justification for exclusion, 

‘….and other literature concerning the effectiveness of homeopathy for otitis media has not been systematically retrieved.’

does not make sense. A full literature search on homeopathy for otitis media was carried out by Optum; if there was a concern that other more recent studies on this condition could have been missed, and that their results may qualify the findings of Sinha et al. in some way, surely Optum had the resources available to ‘systematically retrieve’ all relevant papers dated 2012 onwards? 

When Sinha et al. is included in the Overview Report following this public consultation, it is essential that the NHMRC also make it clear how their reviewers managed to reach such a vastly different conclusion from that of the original authors. 

Sinha and colleagues concluded that individualised homeopathic treatment was as effective as conventional care, 

“There were no significant differences between groups in the main outcome. Symptomatic improvement was quicker in the homeopathy group and there was a large difference in antibiotic requirements favouring homeopathy”

whilst the NHMRC reviewers concluded the direct opposite i.e. 

‘The addition of Sinha et al (2012) to the body of evidence for otitis media is consistent with the conclusion from the Overview Report that there is no reliable evidence that homeopathy is as effective as other therapies for the treatment of children with acute otitis media.’ 

The reviewers have dismissed the findings of this study based on the three following criticisms:

1. Small sample size (N=81)

This is a perfectly reasonable number of participants for a pilot study and does not negate the results of this comparative study as the lack of difference between arms was statistically significant (no significant difference between conventional treatment and homeopathic treatment (p=0.247)).

2. “Method of allocation concealment was not described, which may have been a source of selection bias.” 

This should be mentioned in a future paper, but is a very minor point to raise as there is nothing in table 3 and 4 to suggest that participants in both groups were sufficiently dissimilar to suggest that bias did occur.

3. “There was also a risk of bias in measuring the severity of disease, as the parents/guardians were asked to subjectively rate the severity of symptoms of their child.”  

This is an invalid criticism as the paper specifically states that, “The parents/guardian […] remained unaware of the patient’s group assigned throughout the study.” Furthermore the reviewer makes no mention of the fact that symptom response was also measured by examination of the tympanic membrane by an ENT specialist. Colour, transparency, mobility and bulging of the eardrum were rated by the specialist at the beginning and end of the trial, and the difference calculated. This outcome measure also showed that conventional and homeopathic treatment achieved similar results.

This study is therefore suitable for inclusion in the NHMRC review as a good quality pilot study whose findings warrant further exploration through repetition in a larger-scale trial.

2. Jacobs et al, Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials, Pediatr Infect Dis J, 2003;22:229–34

This study was excluded on the following basis, ‘Unable to assign a level of evidence – non-systematic review. Wrong research type or publication type.’ Jacobs reviewed all available Level II studies on individualised homeopathy for childhood diarrhea - RCTs which she had carried out – and conducted a meta-analysis. It is therefore a legitimate systematic review and also contains a meta-analysis. As this study fits the NHMRC’s inclusion criteria, it was incorrectly excluded and now needs to be included.

3. Schneider et al. Treatment of vertigo with a homeopathic complex remedy compared with usual treatments – a meta-analysis of clinical trials, Arzneim.-Forschung 2005, 55(1) 23-29.

According to the Overview Report this review was excluded on the basis of being a Level III-3 study (comparative study without concurrent control). This is incorrect because the review contains two Level II studies (in addition to the two Level III-3 studies). The study therefore fits the NHMRC’s inclusion criteria, it was incorrectly excluded and now needs to be included.

4. Wiesenauer and Lüdtke. A meta analysis of the homeopathic treatment of pollinosis with Galphimia glauca. Forsch. Komplementärmed. 1996; 3: 230-234

According to the document Review of Submitted Evidence, Wiesenauer & Lüdtke (1996) was excluded on the basis that the study was not published in the English language. This is incorrect as the article is in English. As this study fits the NHMRC’s inclusion criteria it was incorrectly excluded and now needs to be included.