Isabelle Barthélémy , Denis Lebel , Régis Vaillancourt , Chris Niro , Jonathan Mitchell , Jean-François BussièresABSTRACT
Background:
A recent symposium on change management highlighted the relatively slow pace of change in the drug-use process. This study was undertaken to determine the degree of concordance between different sources that document levels of conformity with optimal drug-use processes.
Objective:
The primary objective was to compare aggregate national results from the Managing Medications Standards (MMS) of Accreditation Canada and results from the biennial Hospital Pharmacy in Canada survey. The secondary objective was to discuss any significant discrepancies between the 2 sources.
Methods:
In this retrospective cross-sectional study, attempts were made to pair each Accreditation Canada MMS criterion with specific results from the Hospital Pharmacy in Canada 2009/2010 Report . Average conformity per criterion from the 2010 Accreditation Canada on-site surveys was compared with conformity as documented in the Hospital Pharmacy in Canada 2009/2010 Report . A discrepancy ratio was calculated for each criterion, with ratios less than 0.80 or greater than 1.20 being considered significant.
Results:
Overall, 82 (61%) of 134 MMS criteria could be paired with results from the 2009/2010 Hospital Pharmacy in Canada survey. The average calculated discrepancy ratio (± standard deviation) between the 2 sets of results was 0.62 ± 0.29 (range 0.05 to 1.19). The average discrepancy ratios by domain were as follows: 0.49 for safely administering medications, 0.58 for accurately preparing and dispensing medications, 0.61 for working together to promote medication safety, 0.62 for carefully selecting and procuring medications, 0.69 for monitoring quality and achieving positive results, 0.71 for appropriately ordering medications and transcribing medication orders, and 0.76 for properly labelling and storing medications. For 59 criteria, there was a significant discrepancy between the 2010 MMS on-site surveys and the 2009/2010 Hospital Pharmacy in Canada survey.
Conclusion:
Nearly two-thirds of the MMS criteria could be paired with results from the Hospital Pharmacy in Canada survey, but the average discrepancy ratio of 0.62 indicates substantial discrepancies in the data collected by these 2 methods. Further studies are required to explore the reasons for such discrepancies.
KEYWORDS: accreditation , pharmacy practice , comparisons
RÉSUMÉ
Contexte :
Un récent colloque sur la gestion du changement soulignait la lenteur du changement dans le processus de distribution des médicaments. Cette étude a été entreprise afin de déterminer le degré de concordance entre différentes sources qui rassemblent des informations sur les degrés de conformité aux processus optimaux de distribution des médicaments.
Objectif :
L’objectif principal était de comparer la somme des résultats nationaux des normes sur la gestion des médicaments (NGM) d’Agrément Canada aux résultats du sondage bisannuel sur les pharmacies hospitalières canadiennes. Le second objectif était d’étudier tout écart important entre les deux sources.
Méthodes :
Dans cette étude rétrospective transversale, on a tenté d’apparier chaque critère des NGM d’Agrément Canada à des résultats précis du Rapport 2009–2010 sur les pharmacies hospitalières canadiennes . Le degré moyen de conformité calculé pour chaque critère sur l’ensemble des visites d’Agrément Canada en 2010 a été comparé au degré de conformité dont fait état le Rapport 2009–2010 sur les pharmacies hospitalières canadiennes . Un indice d’écart a été calculé pour chaque critère. Les indices d’écart inférieurs à 0,8 et supérieurs à 1,2 étaient considérés comme importants.
Résultats :
En tout, 82 (61 %) des 134 critères des NGM ont pu être appariés à des résultats du sondage sur les pharmacies hospitalières canadiennes de 2009–2010. L’indice moyen d’écart (± écart-type) entre les deux ensembles de résultats était de 0,62 ± 0,29 (étendue de 0,05 à 1,19). Les indices moyens d’écart par domaine étaient : 0,49 pour l’administration sécuritaire des médicaments, 0,58 pour la préparation et la distribution des médicaments avec précision, 0,61 pour le travail d’équipe visant à promouvoir la sécurité des médicaments, 0,62 pour l’attention portée à la sélection et à l’approvisionnement en médicaments, 0,69 pour la gestion de la qualité et l’atteinte de résultats positifs, 0,71 pour l’émission et la transcription appropriées d’ordonnances de médicaments, et 0,76 pour l’étiquetage et l’entreposage convenables des médicaments. Il y avait un écart important, pour 59 critères des NGM, entre les résultats des visites d’Agrément Canada en 2010 et ceux du sondage sur les pharmacies hospitalières canadiennes de 2009–2010.
Conclusion :
Près des deux tiers des critères des NGM ont pu être appariés à des résultats du sondage sur les pharmacies hospitalières canadiennes, mais l’indice moyen d’écart de 0,62 montre qu’il y a d’importants écarts entre les données recueillies par ces deux méthodes. D’autres études sont nécessaires afin d’explorer les raisons de tels écarts.
MOTS CLÉS: agrément , pratique de la pharmacie , comparaisons
Numerous Canadian health care stakeholders recognize the importance of an optimal drug-use process within hospitals to ensure safe medication use.1–5 Before 2008, accreditation standards related to managing medications were integrated into various Accreditation Canada clinical standards. In 2008, Accreditation Canada released the Managing Medications Standards (MMS; renamed the Medication Management Standards in 2013) as part of its Qmentum program to “promote a collaborative approach to prevent and reduce medication errors and near misses by addressing all aspects of the medication management process, from prescription, selection, preparation and dispensing to administration of the medication and ongoing monitoring of clients.”4
Every 2 years in Canada, an independent editorial committee conducts a web survey of directors of hospital pharmacies with at least 50 acute care beds. Known as the Hospital Pharmacy in Canada (HPC) survey, this analysis has been conducted since 1985, with the results published online since 1997/1998.6 A recent symposium on change management organized by this editorial committee highlighted the relatively slow pace of change in the drug-use process, despite the availability of evidence.7
It was hypothesized that there exists a discrepancy in levels of conformity with drug-use processes as documented by different sources. Given the availability of a substantial dataset about the drug-use process and involvement in the Accreditation Canada Medication Management Standards Working Group to advise on updating the MMS standards, some of the authors proposed to Accreditation Canada that results of its national accreditation surveys be compared with data in the HPC report.
In this retrospective cross-sectional study, the main objective was to compare the Accreditation Canada MMS compliance ratings with results of the HPC survey. The secondary objective was to explain any significant discrepancies between these sources.
The compliance ratings (level of conformity) for all criteria of the MMS were obtained from Accreditation Canada for on-site visits conducted in 2010 by peer surveyors. Accreditation Canada surveyors are experienced senior health care professionals from Canadian health care organizations that are accredited by Accreditation Canada. During the on-site survey, the surveyors observe and evaluate the extent to which the standards are being met and offer advice and guidance on areas for improvement. The 2010 compliance ratings covered organizations from various sectors across Canada, including acute care, health systems (regional and district health authorities), long-term care, and mental health. The results were made available as average levels of conformity per criterion.
The Accreditation Canada on-site survey process includes all stakeholders involved in medication management within a health care organization (e.g., pharmacists, nurses, physicians, patients). Whenever possible, each MMS criterion was paired (by a pharmacy resident [I.B.]) with a specific result from the HPC 2009/2010 report.8 In some cases, if the theme or topic targeted by the MMS criterion had been covered only in the previous report (for 2007/2008),9 those results were also extracted. The response rates on the HPC surveys were 72% (160/222) for 2009/2010 and 74% (166/223) for 2007/2008.
The proposed pairing between the 2 sources was validated by a panel of 5 people (4 of whom are authors on this paper): a pharmacy resident (I.B.), a research assistant involved in data management for the HPC, a director of pharmacy involved on the HPC editorial board and as a respondent to the survey (J.F.B.), a pharmacist involved as a surveyor for Accreditation Canada (R.V.), and a pharmacist from the authors’ research team (D.L.).
Each panel member scored the similarity of the proposed pairings between the Accreditation Canada MMS criteria and the HPC results, using a scoring method chosen by consensus among the panel members. The following 3-point scale was considered sufficient to quickly evaluate the similarity of the proposed pairings: 1 = no similarity (HPC result cannot be paired with the proposed MMS criterion), 2 = some similarity (HPC result should not be paired with the proposed MMS criterion), 3 = sufficient similarity (HPC result can be paired with the proposed MMS criterion). We calculated the average similarity score of the panel members for each criterion. Criteria with an average similarity score higher than 2.6 (i.e., at least 3 of the 5 panel members assigned a score of 3) were paired for analysis of discrepancy.
Aggregate compliance ratings were provided by Accreditation Canada for each MMS criterion. In the case of the HPC report, when multiple results for a similar topic were paired with one MMS criterion, the average of those multiple results was calculated and used.
A discrepancy ratio between the results of the 2009/2010 HPC survey and the 2010 MMS evaluation was then calculated by division (e.g., 50% conformity on HPC survey ÷ 80% conformity on MMS evaluation = 0.63). A discrepancy ratio for a given criterion was considered significant if the value was either below 0.80 or above 1.20. For example, a ratio of 0.50 indicates that the activity was perceived as being implemented twice as often during the on-site Accreditation Canada visit as by self-report in the HPC survey. The discrepancy ratios were also averaged for each of the 7 domains of the MMS. Only descriptive statistics with proportions are reported.
A total of 82 (61%) of the 134 MMS criteria could be paired with results from the 2009/2010 HPC survey (Table 1). The average calculated discrepancy ratio (± standard deviation) was 0.62 ± 0.29 (range 0.05 to 1.19). Significant discrepancies (i.e., less than 0.80 or more than 1.20) were noted for 59 criteria. Table 2 shows 3 examples of how the MMS criteria were paired with results from the 2009/2010 HPC report.
Table 1.
Profile of Unpaired and Paired Criteria for the Accreditation Canada’s Managing Medications Standards (2010)* and Results of the 2009/2010 Hospital Pharmacy in Canada Survey† (part 1 of 3)
Table 2.
Three Examples of Pairing of Criteria from the Accreditation Canada’s Managing Medications Standards (2010) and Results of the 2009/2010 Hospital Pharmacy in Canada Survey
To the authors’ knowledge, this is the first descriptive study comparing the similarity between 2 different sources of information that aim to evaluate conformity with specific aspects of the drug-use process. Among the 59 paired criteria for which the discrepancy was significant, the degree of conformity with the specified drug-use process was consistently lower for self-reported (HPC) data than for surveyors’ observations (MMS data). These discrepancies appeared to be greatest for the domain of “safely administering medications to clients”.
In some cases, the gap was substantial, with values as low as 0.05 (see “Discrepancy Ratio” column in Table 1). Criteria with large differences in conformity between the 2 sources included medication histories, utilization of computerized checks for drug interactions, and existence of processes to update drug lists.
A number of factors may affect the level of conformity with criteria for drug-use processes, such as the clarity and assessability of the criteria, the observational methods used and any potential bias, and the period of observation. In a previous study, which used an earlier version of the MMS standards, 86% of the MMS criteria were deemed “clear” and 64% of the criteria were deemed “assessable”.10
The number of standards applicable to the drug-use process has been increasing throughout the country over the past decade. Such development of new criteria can create a “moving target” for both decision-makers and front-line pharmacists. The Accreditation Canada standards are intended to be complementary to other sources of information about medication management, and this complementarity was a significant focus of the Medication Management Standards Working Group. However, there was uncertainty about what other sources should be prioritized, including the Canadian Society of Hospital Pharmacists’ CSHP 2015 vision and its 36 associated goals, the Institute for Safe Medication Practices Canada (ISMP Canada) self-assessment tool, provincial pharmacy regulatory audits, or academic audits related to accreditation of undergraduate training at universities. Although we recognize the relevant efforts of all these stakeholders to mobilize hospitals and their staff to adopt an optimal drug-use process, thereby increasing patient safety, there should be limited overlap between these sources to help decision-makers and pharmacists target critical gaps in medication management.
In 2011, Accreditation Canada established a Standards Working Group to revise the MMS and relevant Required Organizational Practices (essential, evidence-based practices that mitigate risk and contribute to improving the quality and safety of health care services). The group included representation from client organizations, surveyors, and key associations such as CSHP and ISMP Canada to promote and support alignment with standards for medication management. The current study was undertaken in the context of this revision, and our raw results were provided to the group, as background evidence. The revised MMS standards were circulated for national consultation in 2012, to obtain broader feedback on the proposed revisions from stakeholders, client organizations, and surveyors. In particular, respondents were asked to comment on the importance, clarity, and assessability of each criterion. The revised standards were released in January 2013 and have been used during on-site surveys since January 2014.
This study had limitations. It was not possible to evaluate whether the same health care organizations were surveyed by the 2 methods. The HPC criteria address medication management topics at a more detailed level than do the MMS. The intent of the MMS is to provide a framework for client organizations to establish a medication management system that includes developing processes based on best practices and applicable regulations. The standards include references to key resources and guidelines such as those produced by CSHP and ISMP Canada. The 2 sources of information have different sets of themes, terms, definitions, and time periods. The validity of the pairing in the current study is therefore limited by the heterogeneity of the data. Data were collected differently for the 2 sources: the on-site visit for accreditation involves multiple direct care providers, staff members, and patients, whereas the HPC survey relies solely on a web-based self-assessment questionnaire completed by directors of pharmacy. A smaller-scale comparison involving a selected group of hospital pharmacies with data from both an on-site visit and the HPC survey could help to minimize confounding factors inherent to use of an “average score”; this approach was not used for the current study, to respect respondent confidentiality, but could be considered for future research. The pairing of MMS and HPC results relied on a limited panel of non-independent experts. A broader panel working independently might generate a different number of paired criteria, which could affect the average discrepancy ratio. An average score of 2.6 or greater (based on the panel’s ratings) was used to define the criteria to be evaluated using the discrepancy ratio. Results from the MMS and the HPC report were chosen for this study because of availability of these data; a comparison of conformity using other standards might yield different results.
A total of 61% of the 2010 Accreditation Canada MMS criteria could be paired with results from the 2009/2010 HPC survey. The average discrepancy ratio between the 2 sources was 0.62 ± 0.29. The average discrepancy ratio by domain was 0.49 for safely administering medications to clients, 0.58 for accurately preparing and dispensing medications, 0.61 for working together to promote medication safety, 0.62 for carefully selecting and procuring medications, 0.69 for monitoring quality and achieving positive results, 0.71 for appropriately ordering medications and transcribing medication orders, and 0.76 for properly labelling and storing medications. Further studies are required to explore the reasons for these discrepancies.
1. Advancing safe medication use [website]. Toronto (ON): Institute for Safe Medication Practices Canada; [cited 2013 Jul 25]. Available from: http://www.ismp-canada.org/index.htm
2. Official publications. Ottawa (ON): Canadian Society of Hospital Pharmacists; [cited 2013 Jul 25]. Available from: http://www.cshp.ca/productsservices/officialPublications/home_e.asp
3. Safer healthcare now! [website]. Edmonton (AB): Canadian Patient Safety Institute; [cited 2013 Jul 25]. Available from: http://www.saferhealthcarenow.ca/en/pages/default.aspx
4. Medication management standards. Ottawa (ON): Accreditation Canada; [cited 2014 Mar 18]. Available from: http://www.accreditation.ca/medication-management-standards
5. Shah A. Pharmacy intervention in the medication-use process: the role of pharmacists in improving patient safety. The Hague (Netherlands): International Pharmaceutical Federation; 2009 [cited 2013 Jul 25]. Available from: http://www.fip.org/files/fip/Patient%20Safety/PatientSafetyAdvidShah.pdf
6. Hospital pharmacy in Canada report [biennial]. Eli Lilly; 1993–2010 [cited 2013 Jul 25]. Available from: http://www.lillyhospitalsurvey.ca/hpc2/content/Reports3.asp
7. Are you driving change? What is your GPS? 2013 Canadian hospital pharmacy leadership conference; 2013 Jun 7–9; Ashton (ON) [proceedings]. Eli Lilly; 2013 [cited 2013 Jul 25]. Available from: http://www.lillyhospitalsurvey.ca/hpc2/content/Millcroft_13/Symposium%20Report%20-%20Millcroft%202013.pdf
8. Babich M, Bornstein C, Bussières JF, Hall K, Harding J, Lefebvre P, et al, editors. Hospital pharmacy in Canada 2009/2010 report. Eli Lilly; 2010 [cited 2013 July 25]. Available from: http://www.lillyhospitalsurvey.ca/hpc2/content/rep_2010_toc.asp
9. Babich M, Bussières JF, Hall KW, Harding J, Johnson N, Lefebvre P, et al., editors. Hospital pharmacy in Canada 2007/2008 report. Eli Lilly; 2008 [cited 2014 Mar 17]. Available from: http://www.lillyhospitalsurvey.ca/hpc2/content/rep_2008_toc.asp
10. Alemanni J, Brisseau L, Lebel D, Vaillancourt R, Rocheleau L, Bussières JF. A pilot comparative study of the clarity and assessability of the drug management standards of Accreditation Canada and the US Joint Commission. Can J Hosp Pharm. 2011;64(2):116–23.
Competing interests: Régis Vaillancourt is a surveyor for Accreditation Canada. Jean-François Bussières is a consultant for the editorial board of the Hospital Pharmacy in Canada survey. No other competing interests declared.
The authors thank Cynthia Tanguay, Pharmacy Research Assistant at the Centre hospitalier universitaire Sainte-Justine, for help with the pairing of data and revision of the manuscript, and Diana Sarakbi, Product Development Specialist with Accreditation Canada, for revising the manuscript.
Canadian Journal of Hospital Pharmacy , VOLUME 67 , NUMBER 2 , March-April 2014