Amélie Monnier, Charlotte Jacolin, Suzanne Atkinson, and Jean-François BussièresTo cite: Monnier A, Jacolin C, Atkinson S, Bussières JF. Assessment of effectiveness of 2 medication-use process quality audit tools using Clinical Performance Feedback Intervention Theory. Can J Hosp Pharm. 2024;77(3):e3487. doi: 10.4212/cjhp.3487
The hospital medication-use process encompasses more than 100 steps, all of which involve the risk of medication errors that can lead to patient harm.1 At the same time, all of these steps constitute opportunities for audit and evaluation of practice. Several professional organizations, regulatory authorities, and accreditation bodies have proposed standards to encourage best practices to reduce risks. The American Society of Health-System Pharmacists (ASHP)2 and the Canadian Society of Hospital Pharmacists3 have recognized the pharmacist’s importance in continuous quality improvement in hospital practice, including through the conduct of quality audits. The ASHP discussion guide2 presents a list of 17 reasons supporting pharmacists’ involvement in quality performance improvement activities (e.g., skill in analyzing complex systems, core knowledge of medication, ability to recognize an opportunity to standardize a process that might improve quality of care, good collaborative skills, understanding of the risks inherent in the medication-management process). Although pharmacists are perceived as the health care professionals of choice for carrying out audits, they are not necessarily aware of all the good practices surrounding the performance of audits. In hospitals, evaluation of the medication-use process can take the form of a structured research project or more simply a practice audit.
An audit generally involves a set of benchmarks, a data collection tool, and approval from the manager whose department or personnel will be evaluated with the audit. Therefore, an audit is defined as a “methodical and independent examination of a situation relating to a product, a process, or an organization in terms of quality, carried out in cooperation with the parties concerned, with the aim of verifying the conformity of the existing situation with pre-established criteria and the adequacy of these criteria to the desired objective”4 [authors’ translation].
In its standard on medication management,5 Accreditation Canada specifies requirements for acute care organizations to deliver high-quality and safe health services to patients and their families. The conduct of quality audits can be used to verify compliance with this standard. The practice standards of the Ordre des pharmaciens du Québec (Quebec Order of Pharmacists) and the associated application guide6,7 constitute additional benchmarks that can be used for the conduct of quality audits.
Pharmacists who complete a hospital pharmacy residency or a master’s degree in advanced pharmacotherapy usually participate in an evaluation or research project, but they are not necessarily exposed to tools allowing them to evaluate the quality of any audits that they might perform. This article describes the benefit of a tool developed to improve the conduct of audits.
In the literature, clinicians and health care managers have successfully used quality audits to improve their compliance with standards. In a Cochrane review, Ivers and others8 noted that audit and feedback generally led to small but potentially significant improvements in professional practice. The effectiveness of this methodology seemed to depend on baseline performance and how feedback was provided. They also noted that feedback appeared to be most effective when it was provided by a respected supervisor or colleague, when it was presented frequently, when it suggested both specific goals and action plans, when it was aimed at reducing a targeted behaviour, and when the recipients of feedback were not physicians.9 Brehaut and others10 identified 15 ways to improve the impact of feedback interventions. For instance, they suggested providing individualized, rather than general, data; addressing barriers to feedback use; and recommending specific actions. Tuti and others11 conducted a literature review focusing on the efficiency of electronic audit and feedback methods. They concluded that the effects of electronic audit and feedback were highly variable, reporting a weighted pooled odds ratio of compliance with desired practice of 1.93 (95% confidence interval 1.36–2.73) for electronic audits with feedback relative to no audit or feedback.11 Although the use of electronic tools can facilitate the real-time entry of observations and the analysis of data a posteriori, hospitals do not necessarily have access to data entry tools (e.g., tablets) and data entry. Free-text comments are often easier to capture with a paper tool than with a tablet, particularly when direct observations are made by shadowing a professional as they perform activities and provide care.
Colquhoun and others12 discussed methods for designing interventions to change the behaviour of health care professionals. Additionally, Colquhoun and others13 identified 313 theory-based, testable hypotheses that suggest favourable conditions for conducting audits and feedback. Brown and others14 proposed the Clinical Performance Feedback Intervention Theory (CP-FIT), a cyclical process of effective audit and feedback built on a comprehensive health care–specific feedback theory for the design, implementation, and evaluation of feedback in health care. The CP-FIT is based on 10 steps (i.e., goal, data collection and analysis method, feedback display, feedback delivery, health professional characteristics, behavioural response, organization or team characteristics, patient population, co-interventions, implementation process) and 42 high-confidence hypotheses that influence the effectiveness of the feedback cycle. For instance, feedback interventions are more effective when “they are supported by individuals in the organisation dedicated to making it a success”. To the authors’ knowledge, however, no previous studies have addressed the topic of improving the quality of pharmacy audits.
The aim of this short study was to assess, using the CP-FIT tool,14 2 medication-use process quality audits performed periodically in a Canadian mother–child university hospital centre. These 2 audits have been conducted annually since 2010. The first audit aims to assess compliance with practice standards of the medication-use process in patient care areas, and the second audit aims to assess the preparation and administration of medications by nurses in patient care areas. The grids referring to the 2 audits have been published previously.15,16 The 2 research assistants (A.M., C.J.) who conducted the audits in 2022 used the CP-FIT tool to independently evaluate the quality of both audits; they also recorded comments supporting their CP-FIT assessments. Performance of the CP-FIT evaluations by the same research assistants who conducted the 2022 audits was intended to reduce evaluation bias. For each CP-FIT hypothesis associated with greater efficiency following an audit with feedback (n = 42), each research assistant indicated whether, for the audits performed in 2022, practice was consistent with, was not consistent with, or was not applicable to the hypothesis proposed. Thereafter, 2 pharmacists (S.A., J.-F.B.) independently reviewed the research assistants’ evaluations to confirm the ratings; differences were resolved by consensus. These 2 pharmacists (both with baccalaureate and master’s degrees) had each been working in the hospital for more than 15 years, were involved in the pharmacy practice research unit, and were the designated pharmacists in charge of audits (Table 1).
TABLE 1 Individuals Involved in Each Step of Project
Overall consistency with the hypotheses proposed in the CP-FIT tool was 71% (27 of the 38 applicable criteria) for audit 1 and 71% (29 of the 41 applicable criteria) for audit 2. Four of the CP-FIT criteria were not applicable for audit 1, and one criterion was not applicable for audit 2. For 2 of the criteria (7, organization and team; 9, co-intervention), discrepancies in evaluations between research assistants and pharmacists were resolved by consensus. Detailed results are shown in Table 2.
TABLE 2 Consistency with Hypotheses Proposed in the CP-FIT Tool for 2 Medication-Use Process Audits Conducted in 2022
Using these results and information from the literature, the strengths, weaknesses, and areas for improvement were identified for each audit by consensus between the research assistants and the pharmacists. Each criterion with a rating of “not consistent” was used to find areas for improvement. We used the recommendations of Brehaut and others,10 Tuti and others,11 and Colquhoun and others12,13 to identify other potential improvements. The following improvements will be made to our audits: provide more personalized and faster feedback to the teams; computerize the data collection tool for audit 2 to improve efficiency; encourage other hospitals to use the same grid for their audits, to allow for and promote comparative analyses; and consider the use of agents of change to eliminate some discrepancies.
To our knowledge, this study is the first to use the CP-FIT as a tool to evaluate audit processes in the setting of hospital pharmacy. Consistency with the CP-FIT hypotheses was 71% for both audits, and these evaluations helped us to improve our medication-use process. Two other studies used the same tool, not to evaluate consistency with hypotheses, but to improve their audit process.17,18 Willis and others17 used the CP-FIT tool to assess the extent to which the design of their audit programs (the National Diabetes Audit and the Trauma Audit Research Network) and recent changes to the programs were consistent with best practices. They interviewed 19 individuals with an interest in audit and feedback and noted changes introduced in their 2 audits programs. Chima and others18 conducted 17 interviews and 3 focus groups, using the CP-FIT tool to evaluate the usefulness and feasibility of a new quality improvement tool to flag abnormal test results that might indicate undiagnosed cancer. That study helped to optimize cancer-related recommendations before the effectiveness of the recommendations was tested in a randomized controlled trial.
Use of the CP-FIT tool can help to reflect and improve feedback associated with audit practices and should be explored in hospital settings.
1 Bussières JF, Lebel D, Atkinson S, Tardif C, Meunier P. Le circuit du médicament en établissement de santé : une grille bonifiée pour mieux encadrer la formation des étudiants en pharmacie. Pharmactuel. 2021;54(2):74–6.
2 The ASHP discussion guide on the pharmacist’s role in quality improvement. American Society of Health-System Pharmacists; [cited 2023 Apr 11]. Available from: https://www.ashp.org/-/media/assets/pharmacy-practice/resource-centers/leadership/leadership-of-profession-pharmacists-role-quality-improvement-guide
3 Hospital pharmacists: information paper on enhancing quality and safety in medication use (2010). Canadian Society of Hospital Pharmacists; 2010 [cited 2023 Apr 11]. Available from: https://www.cshp.ca/docs/pdfs/Hospital%20Pharmacists_IP%20on%20Enhancing%20Quality%20and%20Safety%20(2010).pdf
4 Audit de la qualité. In: Grand dictionnaire terminologique. Office québécois de la langue française; 2007 [cited 2023 Mar 27]. Available from: https://vitrinelinguistique.oqlf.gouv.qc.ca/fiche-gdt/fiche/8377565/audit-de-la-qualite
5 Chapter 10. Gestion du circuit du médicament. In: Gouvernance, leadership et normes transversales. Agrément Canada/Accreditation Canada; updated annually. Subscription required. Publication also available in English.
6 Standards de pratique. Ordre des pharmaciens du Québec; 2016 [cited 2023 Mar 27]. Available from: https://www.opq.org/wp-content/uploads/wooccm_uploads/290_38_fr-ca_0_standards_pratique_vf-min.pdf
7 Guide d’application des standards de pratique. Ordre des pharmaciens du Québec; [cited 2023 Mar 27]. Available from: https://guide.standards.opq.org/
8 Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(6):CD000259.
9 Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med. 2014;29(11):1534–41.
Crossref PubMed PMC
10 Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164(6):435–41.
Crossref PubMed
11 Tuti T, Nzinga J, Njoroge M, Brown B, Peek N, English M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci. 2017;12(1):61.
Crossref PubMed PMC
12 Colquhoun HL, Squires JE, Kolehmainen N, Fraser C, Grimshaw JM. Methods for designing interventions to change healthcare professionals’ behaviour: a systematic review. Implement Sci. 2017;12(1):30.
Crossref PubMed PMC
13 Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, et al. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117.
Crossref PubMed PMC
14 Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers N, Francis JJ, et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14(1):40.
Crossref PubMed PMC
15 Monnier A, Jacolin C, Atkinson S, Bussières JF. Évolution de la conformité de l’étape de stockage du circuit du médicament dans les unités de soins et cliniques externes d’un établissement universitaire mère–enfant / Evolution of the conformity of the storage step of the medication-use process on patient care units and outpatient clinics of a mother–child university healthcare center. Pharm Clin. 2023 Jul 20. doi: https://doi.org/10.1016/j.phacli.2023.07.001
16 Jacolin C, Monnier A, Farcy É, Atkinson S, Pelchat V, Duval S, et al. Two-year audit of compliance in the preparation and administration of medications by nursing staff in a mother-and-child university hospital center. Arch Pediatr. 2024;31(2):100–5.
Crossref PubMed
17 Willis TA, Wood S, Brehaut J, Colquhoun H, Brown B, Lorencatto F, et al. Opportunities to improve the impact of two national clinical audit programmes: a theory-guided analysis. Implement Sci Commun. 2022;3(1):32.
Crossref PubMed PMC
18 Chima S, Martinez-Gutierrez J, Hunter B, Manski-Nankervis JA, Emery J. Optimization of a quality improvement tool for cancer diagnosis in primary care: qualitative study. JMIR Form Res. 2022;6(8):e39277.
Crossref PubMed PMC
Address correspondence to: Jean-François Bussières, CHU Sainte-Justine, 3175, chemin de la Côte Sainte-Catherine, Montréal QC H3T 1C5, email: jean-francois.bussieres.hsj@ssss.gouv.qc.ca
Competing interests: None declared.
Funding: None received.
Submitted: April 12, 2023
Accepted: February 9, 2024
Published: July 10, 2024
© 2024 Canadian Society of Hospital Pharmacists | Société canadienne des pharmaciens d’hôpitaux
Canadian Journal of Hospital Pharmacy, VOLUME 77, NUMBER 3, 2024