Alexandra Hinse, Sophie Gauthier, Tyler Morissette, Caroline Phuong, Rusaila Shakhtur-Alqawasma, Nancy L Sheehan, and Catherine SicardDOI: https://doi.org/10.4212/cjhp.3331
ABSTRACT
Background
Peer review to assess the quality of documentation is essential, as it provides a framework for constructive feedback, using evaluators with similar qualifications to increase acceptability.
Objective
To determine the feasibility of implementing a peer review continuous quality improvement program for pharmacists’ documentation at the Montreal Children’s Hospital.
Methods
A prospective, single-centre mixed-methods feasibility study was conducted (from January to June 2021) to evaluate the practicality and acceptability of a peer review program (PRP) for assessing the quality of pharmacists’ documentation. A peer review committee of 5 pharmacists evaluated their peers’ clinical notes using a standardized assessment tool. Practicality was determined through the time required for administrative and evaluative tasks and the resources needed for each evaluation cycle. Acceptability was determined through pooled quantitative data related to pharmacists’ perceived relevance of the PRP, confidence in their peers, and satisfaction with the evaluation process. Qualitative data collected through surveys, a focus group, and semistructured individual interviews helped to further explain the results.
Results
A total of 37.4 hours was required to complete both administrative and evaluative tasks in one peer review cycle, which respected the budgeted cut-off for practicality. Acceptability was also achieved, given that more than 80% of survey respondents found the PRP relevant to their practice, were confident in their peers, and were satisfied with the PRP. Qualitative results showed that participants found the PRP to be instructive and that qualitative feedback was preferred over a grade issued as a percentage.
Conclusion
This study showed that it is feasible to implement a PRP to assess the quality of pharmacists’ documentation. To ensure success, it is key that documentation objectives and department resources be predefined.
KEYWORDS: peer review, pharmacist, documentation, quality improvement
RÉSUMÉ
Contexte
L’évaluation par les pairs afin d’évaluer la qualité de la documentation est essentielle, car elle fournit un cadre pour une rétroaction constructive émise par des évaluateurs ayant des qualifications similaires afin d’augmenter l’acceptabilité.
Objectif
Déterminer la faisabilité d’implanter un programme d’évaluation par les pairs en continu de la qualité de la documentation des pharmaciens à l’Hôpital de Montréal pour Enfants.
Méthodes
Une étude de faisabilité prospective, monocentrique et de méthodes mixtes a été menée (de janvier à juin 2021) pour évaluer la praticité et l’acceptabilité d’un programme d’évaluation par les pairs (PEP) ayant pour but d’évaluer la qualité de la documentation des pharmaciens. Un comité d’évaluation par les pairs composé de 5 pharmaciens a évalué les notes cliniques de leurs pairs à l’aide d’un outil d’évaluation standardisé. La praticité a été déterminée par le temps requis pour les tâches administratives et d’évaluation et les ressources nécessaires pour chaque cycle d’évaluation. L’acceptabilité a été déterminée grâce à des données quantitatives regroupées liées à la pertinence perçue du PEP par les pharmaciens, à la confiance envers leurs pairs et à la satisfaction à l’égard du processus d’évaluation. Les données qualitatives recueillies par le biais de sondages, d’un groupe de discussion et d’entretiens individuels semi-structurés ont permis d’expliquer davantage les résultats.
Résultats
Un total de 37,4 heures a été nécessaire pour accomplir les tâches administratives et d’évaluation dans un cycle d’évaluation par les pairs, ce qui respectait le seuil budgété pour des raisons pratiques. L’acceptabilité a également été atteinte puisque plus de 80 % des répondants au sondage trouvaient le PEP pertinent pour leur pratique, avaient confiance en leurs pairs et étaient satisfaits du programme. Les résultats qualitatifs ont montré que les participants trouvaient le PEP instructif et que la rétroaction sous forme de commentaires état préférée à une note émise en pourcentage.
Conclusion
Cette étude a démontré qu’il est possible de mettre en place un PEP pour évaluer la qualité de la documentation des pharmaciens. Pour garantir sa réussite, il est essentiel de prédéfinir les objectifs de documentation et les ressources départementales à disposition.
Mots-clés: évaluations par les pairs, pharmacien, documentation, amélioration de la qualité
Pharmacists’ clinical documentation in patients’ health records is a practice standard recommended by the Canadian Society of Hospital Pharmacists (CSHP).1 High-quality documentation ensures good communication among health care providers and better continuity of care.2 Conversely, failure to document may lead to undesirable consequences for both the patient and the health care team.1 A recent study in a large Montréal university hospital showed that pharmacists’ documentation was “sufficient or extensive” in only 66% of patient medical records, and the rate of conformity with pre-established criteria was 57%.3 The CSHP recommends the “implementation of an educational program ... and processes to assess each pharmacist’s documentation skills as a means to promote and support high-quality pharmacy practice”.1 For quality control, peer review engages individuals with similar qualifications to provide constructive feedback, continuous learning, and reflection on current practices.4,5
As pharmacists’ roles become more differentiated and specialized, managers and supervisors may not have sufficient expertise to judge the quality of care provided. The main barrier appears to be lack of anonymity during assessment, because those being assessed fear repercussions in the workplace.6 Therefore, evaluation by the manager or supervisor may have poor acceptability. Few papers could be found specifically describing peer review programs (PRPs) related to pharmacists’ documentation, and these studies often lacked details on the feasibility of implementation.6–8 Milchak and others8 developed a peer review audit tool to assess pharmacists’ documentation in primary care clinics. Although the audit tool was not assessed for interrater reliability, results showed that the peer review process generated significant and sustainable improvement in pharmacists’ clinical documentation in electronic medical records.8 In the study by Haines and others,6 pharmacists expressed, in response to surveys and during focus groups, that peer review was important for quality control but was also time-consuming. However, these authors did not mention the time or resources required to undertake a cycle of peer review.
The primary objective of the current study was to determine the feasibility of implementing a peer review continuous quality improvement program for pharmacists’ documentation according to predefined criteria for practicality and acceptability. The secondary objectives were to identify aspects of the assessment tool requiring improvement and to explore pharmacists’ opinions about facilitators of and obstacles to the PRP.
This prospective, single-centre feasibility study used a convergent mixed-methods approach. The study was conducted from January to June 2021 at the Montreal Children’s Hospital (MCH), a pediatric hospital that is part of the McGill University Health Center (MUHC). During the study period, 21 pharmacists provided pediatric pharmaceutical care at the MCH, and 10 pharmacy residents completed their residency at the MUHC. Documentation standards at our institution are based on the standards of the Ordre des pharmaciens du Québec.2 No specific formal standards have been developed or adopted within the pediatric department. In our centre, all pharmacists are assigned dispensary hours, and the majority also practise in a clinical area (neonatal intensive care unit, pediatric intensive care unit, hematology–oncology, and/or general pediatrics). An invitation to participate on the peer review committee (PRC) was sent by email to all pharmacists, and 5 individuals were selected to represent, to the extent possible, each of the above clinical areas. The investigators were not allowed to be members of the PRC, to participate in any surveys, or to attend any focus groups or interviews. However, the investigators’ clinical notes could be included, if selected, for evaluation by the PRC.
Given that this was a quality improvement study, the Research and Ethics Board of the MUHC waived the requirement to obtain ethics approval. Participants signed consent forms before taking part in the focus group and interviews and provided implicit consent when they completed the surveys.
During the study, 2 cycles of peer review were performed. Detailed steps and associated timing are presented in Table 1. During each cycle, every instance of documentation by all pharmacists was eligible for review by the PRC.
TABLE 1 Steps of the Study and Associated Timing
For both cycles, the investigators extracted an electronic listing of all pharmacists’ notes from the institution’s electronic medical record system, OACIS (Telus Health). For each pharmacist, the notes were numbered chronologically, Research Randomizer (https://www.randomizer.org/) was used to generate 5 random numbers, and the corresponding 5 notes were selected for evaluation, such that a total of 105 clinical notes were selected per cycle. If a pharmacist had fewer than 5 eligible notes for a given cycle, then additional notes from the other pharmacists were randomly selected on a pro rata basis, to ensure a total of 105 notes per cycle. The notes were then anonymized and randomly distributed among the 5 PRC members. If, after random assignment of notes, the investigators detected that any member of the PRC would be reviewing one of their own notes, any such note was switched with a note previously assigned to another PRC member. The same pharmacists served as PRC members for both cycles.
During each of the 2 cycles, each PRC member was allowed a paid 8-hour day to complete their assessment of 21 clinical notes using the Standardized Tool for the Evaluation of Pharmacists’ Documentation (STEP-D). This tool was previously developed using a modified Delphi method based on a survey of pharmacists from 2 Canadian pediatric centres, the MCH and the Children’s Hospital of Eastern Ontario. It consists of 43 items, divided into 5 sections (Appendix 1, available from https://www.cjhp-online.ca/index.php/cjhp/issue/view/215). The investigators created, on the basis of anticipated use, a user guide (Appendix 2, available from https://www.cjhp-online.ca/index.php/cjhp/issue/view/215), which was provided to each PRC member to assist in standardization of their evaluations. At the end of each cycle, all 21 pediatric pharmacists received a feedback report by email, which included the number of notes evaluated, the average grade obtained, copies of the selected notes, and the completed STEP-D form associated with each note. The pharmacists were blinded to the PRC members who assessed their notes.
The time required to complete each administrative task (i.e., extraction, preparation, randomization, and distribution of notes; preparation and distribution of feedback) was measured by the investigators. The time required to complete evaluation of each note with the STEP-D was self-reported by PRC members. Because it was possible to adjust the evaluation process on the basis of results of the surveys and focus groups conducted after cycle 1, data for these time measures were collected only in cycle 2. Also, using the measures of time from cycle 2 was more representative of reality, given that the PRC members became familiar with the flow of each step during cycle 1.
To assess pharmacists’ acceptance of the PRP, an anonymous survey was sent, at the end of each cycle, to all pharmacists whose documentation was assessed. Similarly, an anonymous survey was sent to all members of the PRC to obtain their assessment of PRP relevance and their confidence in their ability to complete the steps appropriately.
After completion of cycle 1, a focus group, led by the pharmacy residents (S.G., T.M., C.P., R.S.-A.) and involving PRC members, was held to address the fluidity of the PRP and any logistic problems. After completion of cycle 2, semistructured individual interviews were conducted by the same pharmacy residents with 5 of the pharmacists whose documentation was evaluated and the 5 PRC members.
To determine the feasibility of the PRP, we assessed its practicality and acceptability. The PRP was considered practical if the time required to complete administrative tasks was 16 hours or less per PRP cycle, the average time required to assess a note with the STEP-D was 24 minutes or less for at least 4 of the 5 PRC members, and the total human resources needed to complete 1 PRP cycle was 56 hours or less. The PRP was considered acceptable if, when responses to survey questions related to each of the primary and secondary objectives were pooled (i.e., considered as a group), at least 80% of survey respondents answered 4 (partially agree) or 5 (totally agree) on the 5-point Likert scale for all questions, weighted by the number of questions per objective. This threshold was deemed reasonable by the investigators, given that the PRP could be improved after completion of the study. Table 2 presents the pre-established evaluation criteria and their associated thresholds for feasibility.
TABLE 2 Pre-established Evaluation Items and Associated Thresholds for Feasibility
The various thresholds were set to fit within typical 8-hour days and were approved by the management team of the MUHC pharmacy department. The investigators and pharmacy administrators met during the drafting of the protocol to assess the time available to devote to peer review for the following year based on actual resources and anticipated constraints. With respect to prioritizing core departmental activities, including medication dispensing and patient care, it was deemed possible to free up each of the 5 reviewers for 1 day per cycle (for the evaluations) and the investigators for 2 days per cycle (for administrative tasks related to the process), with an expectation of 4 cycles per year. To complete the necessary workload of 21 notes per PRC member, 24 minutes per note was considered a reasonable cut-off to fit within the schedule and was tested by the investigators before initiation of the current study. To address the secondary objectives, facilitators and obstacles related to the PRP were explored qualitatively through open-ended questions in the surveys, interviews, and focus group. The interviews and the focus group were held on the Microsoft Teams platform. Questions used to guide the interviews and focus group are available in Appendix 3 and Appendix 4, respectively (available from https://www.cjhp-online.ca/index.php/cjhp/issue/view/215).
Inductive coding was used. A coding frame was developed, and if modifications were made to the coding frame, the qualitative data were updated.
The total time to perform administrative tasks for cycle 2 of the PRP was 14.9 hours (details shown in Table 3). For the same PRP cycle, the total time required by all 5 members of the PRC to complete their review of all 105 notes was 22.6 hours, for an average of 12.9 (standard deviation 8.7) minutes for each note assessment using the STEP-D. The proportion of PRC members who reported that the allotted 8-hour day was sufficient to evaluate the 21 notes assigned was 80% for cycle 1 and 100% for cycle 2. The administrative and evaluative tasks required a total of 37.4 hours for the second PRP cycle, which was within the predetermined threshold of 56 hours.
TABLE 3 Total Time to Perform Administrative Tasks for Cycle 2 of Documentation Peer Review Program
The rate of survey participation among pharmacists whose documentation was evaluated was 71% (n = 15) for cycle 1 and 43% (n = 9) for cycle 2. All 5 PRC members (100%) participated in the PRC survey for both cycles.
When survey results pertaining to the same primary or secondary objective were pooled, the threshold of at least 80% of respondents answering favourably (i.e., “partially agree” or “totally agree”) was reached in cycle 2 for each surveyed group. Scores for pertinence and confidence in the PRP were below 80% in the cycle 1 surveys, especially the survey of PRC members, driven by lower rates of satisfaction with the STEP-D and limited self-confidence in evaluating their peers during cycle 1. Table 4 contains detailed results from the acceptability survey.
TABLE 4 Results of Acceptability Surveys in Terms of Perceived Relevance, Confidence in Peers, and Satisfaction with PRP
The PRP was appreciated by PRC members mostly for its instructive potential, as it allowed them to review notes written by colleagues working in different clinical units. These new perspectives offered ways for them to improve their own documentation and to reflect on their own practice and the importance of documenting activities and interactions.
All pharmacists viewed the idea of a continuous PRP as an opportunity to keep the quality of their documentation on track and to standardize documentation within the MCH. When discussing peer review, some pharmacists limited their concept of “peers” to health care professionals within the same clinical unit, while others viewed any MCH pharmacists as peers. Those who showed a preference for evaluators to be drawn from their own clinical unit felt that only pharmacists with a similar practice could adapt the evaluation to the clinical unit’s reality. Conversely, other pharmacists found that having an evaluator from a different unit encouraged improvement of documentation by bringing a different perspective and disrupting the status quo. However, given that the main objective of the PRP was to assess the quality of documentation, not the quality of the clinical act itself, all pharmacists found it acceptable to have their notes evaluated by a pediatric pharmacist from any clinical unit. Although the pharmacists who were interviewed informed us that anonymization was pertinent to avoid bias or judgment by peers, complete anonymization was not fully achievable. Indeed, the MCH pharmacy team is small, and pharmacists were able to guess who had written a particular note or who had performed an evaluation. Interestingly, 2 pharmacists shared that they would prefer to lift anonymity to allow exchange with their evaluators on how to improve their documentation.
According to survey results, PRC members’ satisfaction with the PRP increased after cycle 2, especially their satisfaction with the STEP-D, which evolved from 60% in cycle 1 to 100% in cycle 2. Notably, no changes were made to either the PRP or the STEP-D between cycles 1 and 2, because no major issues were raised during the focus group. The main cause of dissatisfaction with the STEP-D was concern for high interrater variability, reported by both the PRC members (evaluators) and the pharmacists whose documentation was evaluated. The PRC members commented that some elements of the STEP-D were interpreted differently by their colleagues. In addition, some PRC members found the STEP-D items too rigid. For example, a note that they considered appropriate for certain contexts might have resulted in a low grade because of the restrictive nature of the grading scale. This rigidity diminished their acceptance of the tool. PRC members also noticed a lot of repetition in the STEP-D items, and many felt that the evaluation grid could be made more concise. Ultimately, all study participants agreed on the necessity of using a standardized assessment tool to ensure consistency among evaluators. They expressed that the STEP-D facilitated documentation assessment through its structure and its focus of evaluating the quality of the note, rather than the clinical aspect described in the documentation. Moreover, pharmacists greatly appreciated the comments sections in the STEP-D, explaining that these were more valuable than the grade itself, especially when recommendations on how to improve documentation were provided. The user guide was especially useful during cycle 1, when all PRC members reported using it, although the proportion using the guide dropped to 40% during the second cycle, when PRC members had become more familiar with the process.
The grades generated by the STEP-D offer objectivity and precision, but many participants argued that they were perceived as degrading and were useless in terms of suggesting ways to improve; these limitations could reduce the acceptability of the PRP. Some feared that the negative perception of grades would discourage pharmacists from writing notes, because of a fear of workplace repercussions. Overall, it was suggested that the PRP should be used only to provide qualitative feedback, with summarized and personalized recommendations for improvement.
The findings of this study suggest that it is feasible to implement a peer review process for evaluating pharmacists’ documentation according to the practicality and feasibility criteria established within our method. The average time required to complete a STEP-D was shorter than originally allocated and was similar across evaluators. However, the time required was highly variable depending on a note’s length and complexity. Pharmacists shared that the PRP helped them to reflect on their documentation practice and to develop their skills in this area, and PRC members considered their exposure to a variety of notes highly constructive. Milchak and others8 also reported that their peer review process was highly appreciated and considered it a unique learning experience because pharmacists were involved in performing the evaluations.
Despite meeting our threshold of 80% for acceptability, some factors affecting pharmacists’ approval of the PRP should be addressed. Although the STEP-D had the lowest proportion of acceptability in survey results, its use was crucial to allow evaluators to focus on the quality of documentation rather than the quality of the clinical act. PRC members’ satisfaction with the STEP-D improved by 40 percentage points after cycle 2, which suggests that exposure to the tool promoted its appreciation. Some participants suggested adding more options in the evaluation scale to make the tool less rigid and grouping certain items together to avoid redundancy. Our primary objective concerned the feasibility of the PRP as a whole, and thus we did not consider its success to be defined by the assessment tool alone.
To our knowledge, no validated tool to assess pharmacists’ documentation has been previously described in the literature, and it is therefore difficult to compare the STEP-D with other methods. In our opinion, the method used to develop the STEP-D was a reasonable attempt to create a tool specifically adapted to our project, since the evaluation criteria were based on what has been published in the literature, and the modified Delphi approach allowed Canadian pediatric hospital pharmacists to select the criteria most pertinent to their practice for inclusion in the tool. Moreover, to improve the acceptability of this method, we suggest that grades be omitted from the feedback report sent to pharmacists whose notes are evaluated. In the version of the PRP to be implemented in our department (the operational version), grades will be presented as a performance indicator for the entire pharmacist team, to allow us to track yearly progress, and we will emphasize that the PRP is not meant to be either competitive or punitive. Likewise, double anonymization should be preserved for subsequent cycles, as most pharmacists felt that such an approach would reduce their fear of judgment. The importance of anonymity to avoid tension among peers was also noted by Haines and others.6 Although the use of grades was criticized, pharmacists highly valued constructive feedback through comments, which were included at the evaluators’ discretion. Future PRC members will be encouraged to add comments more consistently.
The main limitation of this study was the potential for selection bias during evaluation of the PRP’s acceptability. Indeed, most PRC members showed interest in participating in the project, which suggests that they may have been inclined to have a more favourable opinion of the PRP. To limit confirmation bias, the investigators were excluded from surveys, focus groups, and interviews. Moreover, the low cycle 2 response rate for our survey of pharmacists whose documentation was evaluated might not accurately reflect the opinion of the entire study population. It is possible that some pharmacists felt it was not pertinent, and was perhaps redundant, to answer the same questions for both cycles, which likely introduced participation bias. It would also have been interesting to stratify satisfaction according to the number of notes evaluated, to verify whether pharmacists for whom no or only a few notes were evaluated necessarily had a negative view of the PRP. However, because of the small size of our pharmacist team, such stratification would have jeopardized the anonymity of survey responses. We also note the absence of prior validation of the survey questionnaires and the STEP-D as limitations. Our predetermined and detailed feasibility criteria certainly represented a strength of this study, as they allowed us to limit confirmation bias.9 Although the quantitative data offered us clear and objective measures to answer our study question, qualitative data and the use of a mixed methodology offered us a deeper insight into pharmacists’ opinions.
The results from this study are of particular interest not only for the MUHC pharmacy department, but also for other health care centres. Our results may inspire others to reflect on the quality of their documentation and to implement a similar PRP. Other centres can easily adapt our PRP model to their department, with or without using an evaluation tool. Now that it has been shown that implementation of such a program is feasible, future research should focus on validating the tool and evaluating the impact of the PRP on the quality of documentation over time.
1 Documentation of pharmacists’ activities in the health record: guidelines. Canadian Society of Hospital Pharmacists; 2013 [cited 2023 May 4]. Available from: https://www.cshp.ca/Site/Resources/Official-Publications/Guidelines/Site/Content/Resources/guidelines.aspx?hkey=324de15b-f6c8-4ee2-b100-ac648c36d80f
2 Guide d’application des standards de pratique. Ordre des pharmaciens du Québec, Direction des communications; 2016 [cited 2020 Nov 30]. Available from: https://guide.standards.opq.org
3 Adam JP, Trudeau C, Pelchat-White C, Deschamps ML, Labrosse P, Langevin MC, et al. Documentation in the patient’s medical record by clinical pharmacists in a Canadian university teaching hospital. Can J Hosp Pharm. 2019;72(3):194–201. Erratum in: Can J Hosp Pharm. 2019; 72(5):411–2.
PubMed PMC
4 Moore IN, Pichert JW, Hickson GB, Federspiel C, Blackford JU. Rethinking peer review: detecting and addressing medical malpractice claims risk. Vanderbilt Law Rev. 2006 [cited 2020 Nov 30];59(4):1175. Available from: https://scholarship.law.vanderbilt.edu/vlr/vol59/iss4/6/
5 Crowl A, Sharma A, Sorge L, Sorensen T. Accelerating quality improvement within your organization: applying the Model for Improvement. J Am Pharm Assoc (2003). 2015;55(4):2364–74.
Crossref
6 Haines ST, Ammann RR, Beehrle-Hobbs D, Groppi JA. Protected professional practice evaluation: a continuous quality-improvement process. Am J Health Syst Pharm. 2010;67(22):1933–40.
Crossref PubMed
7 Cram DL Jr, Stebbins M, Eom HS, Ratto N, Sugiyama D. Peer review as a quality assurance mechanism in three pharmacist-run medication-refill clinics. Am J Hosp Pharm. 1992;49(11):2727–30.
PubMed
8 Milchak JL, Shanahan RL, Kerzee JA. Implementation of a peer review process to improve documentation consistency of care process indicators in the EMR in a primary care setting. J Manag Care Pharm. 2012; 18(1):46–53.
PubMed
9 Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.
Crossref PubMed PMC
Sophie Gauthier, Tyler Morissette, Caroline Phuong, and Rusaila Shakhtur-Alqawasma contributed equally to the writing of this manuscript.
Competing interests: Sophie Gauthier has received payment from Ensemble IQ to support the publication of an article in another journal. No other competing interests were declared.
Funding: The Faculty of Pharmacy, Université de Montréal, provided funding to support statistical assistance for this article.
Canadian Journal of Hospital Pharmacy, VOLUME 76, NUMBER 3, Summer 2023