Indian Journal of Pharmacology Home 

[Download PDF]
Year : 2014  |  Volume : 46  |  Issue : 1  |  Page : 117--120

A study of agreement between the Naranjo algorithm and WHO-UMC criteria for causality assessment of adverse drug reactions

Mahesh N Belhekar, Santosh R Taur, Renuka P Munshi 
 Department of Clinical Pharmacology, Topiwala National Medical College and Bai Yamunabai Laxmanrao Nair Charitable Hospital, Mumbai Central, Mumbai, Maharashtra, India

Correspondence Address:
Renuka P Munshi
Department of Clinical Pharmacology, Topiwala National Medical College and Bai Yamunabai Laxmanrao Nair Charitable Hospital, Mumbai Central, Mumbai, Maharashtra


Objectives: Reliability and usefulness of various adverse drug reaction (ADR) causality assessment scales have not been fully explored. There is no universally accepted method for causality grading of ADRs. In the present study we assessed agreement between the two widely used causality assessment scales, that is, the World Health Organization-Uppsala Monitoring Center (WHO-UMC) criteria and the Naranjo algorithm. Materials and Methods: The same observer assessed all ADRs (n = 913) collected between January 2010 and December 2012 using the WHO-UMC criteria and Naranjo algorithm at a tertiary care hospital in India. We found that the most frequently assigned causality category was «DQ»possible«DQ» with both the scales. Results: A disagreement in the causality assessment was found in 45 (4.9%) cases reflecting «SQ»«SQ»poor«SQ»«SQ» agreement between the two scales (Kappa statistic with 95% confidence interval = 0.143 [0.018, 0.268]). The mean time taken to assess causality of the ADR using the WHO-UMC criteria was shorter than that by the Naranjo algorithm. Conclusion: This study showed that there is a poor agreement between the WHO-UMC criteria and Naranjo algorithm with the former being less time-consuming.

How to cite this article:
Belhekar MN, Taur SR, Munshi RP. A study of agreement between the Naranjo algorithm and WHO-UMC criteria for causality assessment of adverse drug reactions.Indian J Pharmacol 2014;46:117-120

How to cite this URL:
Belhekar MN, Taur SR, Munshi RP. A study of agreement between the Naranjo algorithm and WHO-UMC criteria for causality assessment of adverse drug reactions. Indian J Pharmacol [serial online] 2014 [cited 2020 Sep 18 ];46:117-120
Available from:

Full Text


An adverse drug reaction (ADR) is any noxious and unintended response to a drug that occurs at doses normally used in man for prophylaxis, diagnosis, therapy, or modification of a physiological function. [1] ADRs are considered as one among the leading causes of morbidity and mortality. [2]

Causality assessment is the evaluation of the likelihood that a particular treatment is the cause of an observed adverse event. [3] It assesses the relationship between a drug treatment and the occurrence of an adverse event. It is an important component of pharmacovigilance, contributing to better evaluation of the risk-benefit profiles of medicines and is an essential part of evaluating ADR reports in early warning systems and for regulatory purposes. [4],[5] Causality assessment of ADRs may be undertaken by clinicians, academics, pharmaceutical industry, and regulators and in different settings, including clinical trials. [6]

At an individual level, health care providers assess causality informally when dealing with ADRs in patients to make decisions regarding therapy. Regulatory authorities assess spontaneous ADR reports, where causality assessment can help in signal detection and aid in risk-benefit decisions regarding medicines. [4] Algorithms, being structured systems specifically designed for the identification of an ADR, should theoretically make a more objective decision on causality. The objective causal assessments are based on four basic principles-temporal eligibility, dechallenge and outcome, rechallenge and outcome, and confounding factors. [7]

A number of algorithms or decision aids have been published including the Jones' algorithm, [8] the Naranjo algorithm, [9] the Yale algorithm, [8] the Karch algorithm, [8] the Begaud algorithm, [8] the Australian ADR advisory committee [8] and the World Health Organization-Uppsala Monitoring Center (WHO-UMC) criteria. [3] Each of these algorithms has similarities and differences. WHO-UMC system has been developed in consultation with the National Centers participating in the Program for International Drug Monitoring and is meant as a practical tool for the assessment of case reports. It is basically a combined assessment taking into account the clinical-pharmacological aspects of the case history and the quality of the documentation of the observation. [3] The Naranjo criteria [9] classify the probability that an adverse event is related to drug therapy based on a list of weighted questions, which examine factors such as the temporal association of drug administration and event occurrence, alternative causes for the event, drug levels, and previous patient experience with the medication. None of the causality assessment tools have been universally accepted as the gold standard. [8] Hence, the present study was conducted to assess the agreement between the WHO-UMC criterion and Naranjo algorithm-the two widely accepted tools in pharmacovigilance.

 Materials and Methods


The Institutional Ethics Committee for Academic Research Projects approved the study and also granted waiver of informed consent as the data assessed was anonymized.

Study Procedure

The Central Drug Standard Control Organization ADR reporting forms were used for the collection of ADRs. An intensive ADR reporting was done from January 2010 to December 2012 from the departments of medicine, pediatrics, intensive care units, therapeutic drug monitoring, dermatology, psychiatry, radiotherapy, antiretroviral therapy center and hematology.

The diagnosis of ADRs was primarily based on detailed histories and the correlation between drug intake and the onset of the ADR. One of the authors determined causality assessment using WHO causality assessment scale for each ADR report form. Causality assessment of ADRs obtained with WHO-UMC criteria were categorized into certain, probable, possible, unlikely, unclassified, and unclassifiable. The same author also used the Naranjo algorithm to categorize ADRs into definite, probable, possible, and doubtful.

ADRs were rated by the clinical pharmacologist, initially by the WHO-UMC criteria scale to assess the ADR causality and later by the Naranjo algorithm. The rating of the probability of an ADR depends on: the characteristics of the ADR; the characteristics of the rater (some raters are more reliable than others); the quality of the information (in some ADRs the information is incomplete or lacking, also it varies over time); and finally, it will also depend on the scale used to assess the ADR. Therefore, to make a proper comparison of the two scales, we maintained the three variables constant: the same raters assessed the same reactions and had identical information available.


To test the null hypothesis of H 0 : k 0 = 0.1 against the alternative hypothesis H A : k > 0.1 with significance level (α) of 0.05 and power (1-β) of 0.90 for k 1 = 0.21 (from previous studies), using the formula given by Cantor, [10] that is, [INSIDE:1] the minimum required sample size was 830 ADR cases.

Data are expressed as proportions or percentages of total observations. The agreement between two ADR causality scales was assessed using weighted kappa (κ) test. The κ value ranges from -1 (perfect disagreement) to +1 (perfect agreement). Statistical analysis was performed using GraphPadQuickCalcs software available online at


We found that out of 913 ADRs, 61.1% were seen in male patients and 38.9% were in female patients. The proportion of patients aged < 18, 18-65 and > 65 years were 15.5%, 80.1%, and 4.4%, respectively. As intensive ADR reporting was done by the medical doctors of our department, all collected ADR reports were complete. The ADRs were evaluated retrospectively for causality assessment using the Naranjo algorithm and WHO-UMC criteria. The most frequently assigned causality category with Naranjo algorithm and WHO-UMC criteria was "possible" (99.2% and 93.9%, respectively), followed by "doubtful" (Naranjo, 0.6%) or "unlikely" (WHO-UMC, 3.8%) [Table 1].{Table 1}

The agreement between the two scales was the highest for "possible" category (94.5%) and no agreement at all for "certain" category. Overall disagreement in causality assessment was found in 45 (4.9%) cases. However, there was ''poor'' agreement between Naranjo and WHO-UMC (Kappa statistic with 95% confidence interval = 0.143 [0.018, 0.268]) [Table 2].{Table 2}

The mean time taken to assess causality of an ADR using the WHO-UMC criteria and Naranjo probability scale was 4.1 ± 0.27 and 10.32 ± 1.05 min, respectively.


In the present study, there was ''poor'' agreement between Naranjo algorithm and WHO-UMC criteria.

The causality assessment has pivotal role in clinical practice as well as in drug development. Three approaches are mainly used to assess the causal relationship between drug treatment and the occurrence of adverse events: expert judgment (global introspection), probabilistic approaches, and algorithms. [11] Studies have shown that there is a lot of variation in between rater and within rater decisions on causality of ADRs; this applies both to pharmacologists and physicians. [12],[13]

The percentage disagreement (discordance) in causality assessment between the Naranjo algorithm and WHO-UMC criteria was lower in the present study (4.9%; κ = 0.145) compared with that by Rehan et al., [14] (31%; κ = 0.214), Son et al., [15] (45%), Macedo et al., [4] (51%; κ = 0.23) and Lei et al., [16] (84.9%). However, the observed differences between the present study and earlier studies could be because of subjective assessment inherent in many methods of ADR assessment. The Council for International Organizations of Medical Sciences (CIOMS) ⁄ Roussel Uclaf Causality Assessment Method (RUCAM) scale, used by many expert hepatologists, researchers, and regulatory authorities, is based on the international drug-induced liver injury consensus criteria. The causality score ranges from −8 to 14 and categorized as highly probable (>8), probable (6-8), possible (3-5), unlikely (1-2), and excluded (<0). [17] Garcia-Cortes et al., have shown that in cases of hepatotoxicity, the concordance between the Naranjo scale and the CIOMS-RUCAM scale was 24% (kw = 0.15), and that the former scale lacked validity and reproducibility in the attribution of causality. [17]

Pere et al., [18] also pointed out disagreements between the different algorithms in assessing causality for the same ADR reports. Disagreements was considerable for three major criteria: timing of event, dechallenge, and alternative etiologic candidates. [18] In the present study, few ADR cases were evaluated using Naranjo algorithm for parameters like objective evidence of ADR, ADR to previous exposures, rechallenge with suspected drug, responses to placebo, and the dose adjustment of drugs. Son et al., [15] and Arimone et al., [11] demonstrated similar findings.

The results of our study showed that the most common causality category using the WHO-UMC criteria as well as the Naranjo algorithm was "possible" which substantiates the findings by Macedo et al., [4] and Lei et al. [16] The present study corroborates the findings by Rehan et al., [14] that the time taken to assess causality using WHO-UMC criteria was shorter than that with the Naranjo algorithm.

The assumption that a drug is or was the cause of an illness may have far-reaching consequences for the current or future treatment of the patient. The decision to stop an effective drug, or the conclusion that a drug or group of drugs is contraindicated for future use, may disadvantage and even endanger patients. In individual patients, the outcome of an assessment algorithm may strengthen a conclusion but should not replace the clinical diagnosis. [19] Since several studies [20],[21] accept the categorization of possible or greater as an ADR, then perhaps clinicians simply need to understand how the scales perform at assessing ADRs using this widely agreed upon threshold of an ADR. [22]

However, our study is limited by the fact that only two ADR causality assessment scales were used for assessment of agreement.


Disagreement exists among the WHO-UMC criteria and the Naranjo probability scale, but the former method is simple and less time-consuming.

 Future Implications

None of the available ADR causality assessment methods have been validated to give reproducible results and shown agreement with other methods. Though use of more than one algorithm is advisable, this may not be feasible in clinical practice. However, in clinical research use of a specific ADR causality assessment method may impact the safety profile of an interventional drug.


Due to simplicity and ease of use, the WHO-UMC criteria may be preferred over Naranjo algorithm especially by clinicians in day-to-day practice.


1Medicines: Safety of medicines - adverse drug reactions definition. Fact sheet No. 273. Updated October 2008 WHO Available from: [Last accessed on 2013 Jan 28].
2Ditto AM. Drug allergy. In: Grammer LC, Greenberger PA, editors. Patterson's Allergic Diseases. 6 th ed. Philadelphia: Lippincott Williams & Wilkins; 2002. p. 295.
3The use of the WHO-UMC system for standardized case causality assessment. World Health Organization (WHO) - Uppsala Monitoring Centre. Available from: [Last accessed on 2013 Apr 10].
4Macedo AF, Marques FB, Ribeiro CF, Teixeira F. Causality assessment of adverse drug reactions: Comparison of the results obtained from published decisional algorithms and from the evaluations of an expert panel. Pharmacoepidemiol Drug Saf 2005;14:885-90.
5Arimone Y, Begaud B, Miremont-Salame G, Fourrier-Reglat A, Moore N, Molimard M, et al. Agreement of expert judgment in causality assessment of adverse drug reactions. Eur J Clin Pharmacol 2005;61:169-73.
6Agbabiaka TB, Savovic J, Ernst E. Methods for causality assessment of adverse drug reactions: A systematic review. Drug Saf 2008;31:21-37.
7Turner WM. The Food and Drug Administration algorithm. Special workshop-regulatory. Drug Inf J 1984;18:259-66.
8Srinivasan R, Ramya G. Adverse drug reaction - Causality assessment. Int J Res Pharm Chem 2011;1:606-12.
9Naranjo CA, Busto U, Sellars EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther 1981;30:239-45.
10Cantor AB. Sample-size calculations for Cohen's Kappa. Psychological Methods 1996;1:150-3.
11Arimone Y, Miremont-Salamé G, Haramburu F, Molimard M, Moore N, Fourrier-Réglat A, et al. Inter-expert agreement of seven criteria in causality assessment of adverse drug reactions. Br J Clin Pharmacol 2007;64:482-8.
12Blanc S, Leuenberger P, Berger JP, Brooke EM, Schelling JL. Judgements of trained observers on adverse drug reactions. Clin Pharmacol Ther 1979;25:493-8.
13Karsh FE, Smith CL, Kerzner B, Mazzullo JM, Weintraub M, Lasanga L. Adverse drug reactions-a matter of opinion. Clin Pharmacol Ther 1976;19:489-92.
14Rehan HS, Chopra D, Kakkar AK. Causality assessment of spontaneously reported adverse drug events: Comparison of WHO-UMC criteria and Naranjo probability scale. Int J Risk Saf Med 2007;19:223-7.
15Son MK, Lee YW, Jung HY, Yi SW, Lee KH, Kim SU, et al. Comparison of the naranjo and WHO-uppsala monitoring centre criteria for causality assessment of adverse drug reactions. Korean J Med 2008;74:181-7.
16Lei HS, Rahman AF, Haq AS. Adverse drug reaction reports in Malaysia: Comparison of causality assessments. Malays J Pharm Sci 2007;5:7-17.
17Garcia-Cortes M, Lucena MI, Pachkoria K, Borraz Y, Hidalgo R, Andrade RJ, et al., Spanish Group for the Study of Drug-induced Liver Disease (grupo de Estudio para las Hepatopatías Asociadas a Medicamentos, Geham). Evaluation of naranjo adverse drug reactions probability Scale in causality assessment of drug-induced liver injury. Aliment Pharmacol Ther 2008;27:780-9.
18Pere JC, Begaud B, Haramburu F, Albin H. Computerized comparison of six adverse drug reaction assessment procedures. Clin Pharmacol Ther 1986;40:451-61.
19Meyboom RH, Hekster YA, Egberts AC, Gribnau FW, Edwards IR. Causal or casual? The role of causality assessment in pharmacovigilance. Drug Safety 1997;17:374-89.
20Kane-Gill SL, Visweswaran S, Saul MI, Wong AK, Penrod LE, Handler SM Computerized detection of adverse drug reactions in the medical intensive care unit. Int J Med Inform 2011;80:570-8.
21Oderda GM, Said Q, Evans RS, Stoddard GJ, Lloyd J, Jackson K, et al. Opioid-related adverse drug events in surgical hospitalizations: impact on costs and length of stay. Ann Pharmacother 2007;41:400-7.
22Kane-Gill SL, Forsberg EA, Verrico MM, Handler SM. Comparison of three pharmacovigilance algorithms in the ICU Setting: A retrospective and prospective evaluation of ADRs. Drug Safety 2012;35:645-53.