Robert Taylor , Joanne Jung , Peter Loewen , Carrie Spencer , Anar Dossa , Jane de LemosABSTRACT
Background
The prevalence of online modules for continuing education in the health professions has been increasing in recent years. However, the effectiveness of online modules for pharmacist learning has not been thoroughly studied.
Objectives:
The primary aim of this study was to determine if providing education to pharmacists through a self-paced enhanced online module was non-inferior to a face-to-face learning module with respect to knowledge application on the topic of postoperative insulin dosing. Secondary aims were to determine pharmacists’ knowledge gain and retention, as well as their satisfaction with the modules.
Methods:
The participants in this prospective, randomized, parallel-group non-inferiority trial were pharmacists in a large multicentre health region. Outcomes were measured by comparing scores obtained on pre- and post-module knowledge-assessment questionnaires. A between-group difference in change on knowledge application scores of less than 25 percentage points was the predetermined non-inferiority margin.
Results:
A total of 74 pharmacists consented to participate, 38 randomly assigned to use the enhanced online module and 36 to attend the face-to-face learning session. For questions examining knowledge application, the mean improvement achieved by the online learning group was 26 percentage points greater than that achieved by the face-to-face learning group (95% confidence interval [CI] 25 to 27; p < 0.001). For questions testing knowledge gain, the improvement achieved by the online learning group was 7 percentage points less than that achieved by the face-to-face learning group (95% CI 2 to 12; p = 0.008). Therefore, the enhanced online module was deemed to be non-inferior to the face-to-face learning session in terms of knowledge application and knowledge gain. Insufficient data were available to analyze the secondary outcome of knowledge retention over time. Participant satisfaction was similar for the 2 groups ( p = 0.62).
Conclusion:
The self-paced enhanced online module was non-inferior to facilitated face-to-face learning in terms of improving application and knowledge of insulin dosing. Pharmacists had similar levels of satisfaction with the 2 modes of learning.
KEYWORDS: online learning module , web learning , applied skill teaching , pharmacist education
RÉSUMÉ
Contexte :
La prévalence des modules d’apprentissage en ligne pour la formation continue des professionnels de la santé a augmenté ces dernières années. Cependant, l’efficacité de ces modules pour l’apprentissage des pharmaciens n’a pas fait l’objet d’études rigoureuses.
Objectifs :
Le principal objectif de cette étude était de déterminer si la formation offerte aux pharmaciens au moyen d’un module d’autoapprentissage en ligne amélioré était non inférieure à un module d’apprentissage en personne pour ce qui est de l’application des connaissances sur le sujet de la détermination des doses d’insuline postopératoires. Les objectifs secondaires étaient de mesurer l’acquisition et la rétention des connaissances par les pharmaciens et leur satisfaction par rapport à l’apprentissage.
Méthodes :
Les participants à cette étude prospective de non-infériorité à répartition aléatoire et groupes parallèles étaient des pharmaciens d’une importante régie régionale de la santé multicentre. Les résultats ont été mesurés en comparant les scores obtenus aux questionnaires d’évaluation des connaissances pré- et post-formation. Une différence intergroupe du changement des scores de l’application des connaissances de moins de 25 points de pourcentage constituait la marge de non-infériorité prédéterminée.
Résultats :
En tout, 74 pharmaciens ont consenti à participer à l’étude, dont 38 ont été répartis de façon aléatoire dans le groupe module d’apprentissage en ligne amélioré et les 36 autres, dans le groupe séance d’apprentissage en personne. Pour les questions examinant l’application des connaissances, l’amélioration moyenne obtenue dans le groupe apprentissage en ligne était supérieure de 26 points de pourcentage à celle obtenue dans le groupe apprentissage en personne (intervalle de confiance [IC] à 95 %, 25 à 27; p < 0,001). Pour les questions testant l’acquisition de connaissances, l’amélioration obtenue dans le groupe apprentissage en ligne était inférieure de 7 points de pourcentage à celle obtenue dans le groupe apprentissage en personne (IC à 95 %, 2 à 12; p = 0,008). Par conséquent, le module d’apprentissage en ligne amélioré a été jugé non inférieur à la séance d’apprentissage en personne pour ce qui est de l’application des connaissances et de l’acquisition des connaissances. L’insuffisance des données n’a pas permis d’analyser le résultat secondaire relatif à la rétention des connaissances au fil du temps. La satisfaction des participants était similaire entre les deux groupes ( p = 0,62).
Conclusion :
Le module d’autoapprentissage en ligne amélioré était non inférieur à une séance d’apprentissage en personne au chapitre des connaissances et de leur application pour déterminer les doses d’insuline postopératoires. Les niveaux de satisfaction des pharmaciens étaient similaires entre les deux méthodes d’apprentissage.
MOTS CLÉS: module d’apprentissage en ligne , apprentissage sur Internet , enseignement du savoir-faire , formation des pharmaciens
[Traduction par l’éditeur]
Health care is a continuously evolving discipline requiring professionals to perpetually maintain best practices through ongoing learning and continuing education. Traditionally, such learning has been accomplished through lectures, presentations, and continuing education courses. Since the rise of online learning, many health care disciplines have used online courses to provide continuing education to practitioners. Online continuing medical education courses have increased by over 700% in recent years.1 Research evaluating online learning has also increased exponentially.2
In medical programs, online learning has compared favourably to traditional didactic lecture formats. A meta-analysis conducted by Wutoh and others3 demonstrated that examination performance was essentially equivalent for the 2 modes of learning, and all of the studies included in the meta-analysis reported subjective comments favouring online modules. However, other authors have stated that there is too much variation in course configuration and instructional methods to allow online learning to be generalized as better or worse than other forms of learning.4
Online learning among pharmacists, despite its widespread availability, is less well studied. A meta-analysis comparing online learning with traditional classroom sessions among students in health care professions involved mostly medicine and nursing (26% and 20% of studies, respectively), with only a small proportion of the 76 studies representing pharmacy (7%).2 The situation was similar in a systematic review in which only a small proportion of the studies identified involved online pharmacy education (9% of studies reviewed), with very few conducted in Canada (6%).4
Online courses for pharmacists remain scarce. Within the authors’ large, urban multisite health authority, only a few online courses are offered specifically for pharmacists, whereas a vast number are offered for professionals in other disciplines. However, online learning would seem to be a feasible alternative to traditional classroom sessions for this profession. In one study,5 most pharmacists reported that they regularly used computers at work and at home, and the majority of pharmacists also rated themselves as being computer literate and comfortable accessing Internet sites.
The growing demand for health care staffing combined with resource limitations will require administrators and educators to more closely examine alternatives for delivering consistent policy or educational messages over wider geographic areas than has traditionally been possible. In this context, online learning may represent a practical option for delivery of educational modules to pharmacists. Both enhanced online modules (EOMs) and traditional face-to-face learning (F2FL) sessions have advantages and disadvantages (Table 1).1,6 For example, the creation of an EOM requires a considerable investment of resources, particularly up front. As such, evidence for the effectiveness of EOMs in pharmacy education may influence resource allocation decisions.
Table 1.
Advantages and Disadvantages of Online and Live Modes of Education1,6
The overall aim of this study was to evaluate the effectiveness of an EOM as an alternative to existing F2FL sessions for pharmacists. More specifically, we attempted to determine if EOM was non-inferior to F2FL in terms of pharmacists’ gain, application, and retention of knowledge. Participants’ satisfaction was also compared between the 2 modes of learning.
A randomized, parallel-group, open-label non-inferiority trial was conducted to compare the effectiveness of online and live teaching on the topic of postoperative glucose management, according to guidelines current at the time.7–9 This topic was chosen because diabetes mellitus is increasingly common, has high potential for intervention by pharmacists, and involves therapeutic concepts (i.e., insulin dosing based on basal, nutritional, and correctional [BNC] strategies) that were relatively new and relatively unfamiliar to the pharmacist population in the study region.10 Insulin is also a high-alert medication.11 A new preprinted order set incorporating BNC insulin dosing principles had been developed by the Pharmacy and Endocrinology departments of a tertiary care hospital within the local health authority. The intention was to use this new order set across the entire health authority. As a result, many pharmacists across the region required education on the concepts of BNC insulin dosing. Approval was granted by the University of British Columbia Behavioural Research Ethics Board and the boards at all hospital sites where the research was conducted.
Pharmacists who met the following inclusion criteria were eligible to participate: licensed by the provincial college of pharmacists, employed in the health authority, currently practising, able to attend a lunchtime lecture, having personal or work-based access to a computer, and available to complete both the pre- and post-module knowledge assessment questionnaires. Pharmacists who did not provide consent and those with previous training or extensive experience with BNC insulin dosing were excluded. Eligible pharmacists who provided consent were randomly assigned to one of the learning modules by a computer-generated list of the integers 1 and 2, balanced per block of 4. Participants were informed of the randomization and group assignment less than 12 hours before beginning the EOM or attending the F2FL lecture. The sample size was calculated to ensure 80% power with an α of 0.1 (one-sided), absolute standard deviation of 15% in test scores, and test score improvement in the EOM group to be no more than 25 percentage points less than the test score improvement in the F2FL group (where test score improvements were based on a comparison of pre- and post-module questionnaire results). Given the dearth of active-controlled non-inferiority studies in pharmacy education, effect size differences and standard deviations were based on previous studies in pediatric and nursing education.12,13 Assuming a follow-up rate of 90%, it was determined that 73 pharmacists were required to participate.
The primary objective was to determine if the application of skills learned from a self-paced EOM was non-inferior to the application of skills learned by a traditional facilitated F2FL method. The secondary objectives were to determine if EOM was as effective as live teaching for knowledge gain of BNC insulin dosing, to examine retention of knowledge, and to assess whether the satisfaction of pharmacists participating in the EOM was similar to that of pharmacists participating in F2FL.
The insulin teaching module was delivered by EOM or F2FL over a 6-week period. The F2FL sessions were conducted at 5 hospital sites, with each participant attending a single session. The EOM participants had 7 days to complete the module once it was made available at each site. Participants completed a pre-module questionnaire to measure baseline knowledge. Immediately after finishing the module, each participant completed a post-module knowledge assessment questionnaire, with the same questionnaire being used for both groups. To evaluate retention of information, a follow-up knowledge assessment questionnaire was administered 2 weeks later. The 2-week period was chosen on the basis of previous studies, which showed that knowledge retention was approximately 50% after 1 week and insignificant after 55 days.14
The course materials were the same for the EOM and F2FL sessions. The F2FL module was designed and presented by the region’s diabetes clinical pharmacist expert (A.D.), who used a PowerPoint presentation of the length, style, and format that would typically be used for lunchtime education sessions in the health authority. Each F2FL session was approximately 50 minutes long, and the instructor-to-learner ratio ranged from 1:3 to 1:22, depending on the site. The EOM was based on the F2FL presentation and was designed collaboratively by the diabetes pharmacist expert and an educational design expert (C.S.), with input from other members of the investigative team (J.J., R.T.). The EOM consisted of pages designed with Moodle software (an open-source course management system) and enhanced with charts, photos, and questions to engage the learner. To emulate a teacher’s questions and engagement with the learner, repetition and practice exercises were included, as these features have been shown to improve learning in web-based education.15 The EOM was designed to take a total of about 50 minutes to complete, and could be completed over multiple sessions.
The pre- and post-module knowledge assessment questionnaires were designed by the diabetes pharmacist expert and were directly related to material taught in the BNC insulin modules. The content of the knowledge assessment questionnaires was checked by 2 other diabetes experts for clarity and correctness. Multiple-choice, fill-in-the-blank, and short answer questions were used to test knowledge gain and application of skills. Knowledge gain refers to learning facts about a subject, whereas application of skills refers to the ability to apply the learned facts to hypothetical clinical scenarios. An example of the questions is shown in Table 2. Demographic data, such as years of experience, as well as previous knowledge of insulin dosing were also collected in the pre-module questionnaire. The post-module questionnaire included questions to assess participants’ satisfaction with aspects such as convenience, expectations, information, and interest in further sessions (graded on a Likert scale), as recommended by other studies.1,12,16 The follow-up knowledge assessment used the same questionnaire as the post-module knowledge assessment.
Table 2.
Example of BNC Insulin Dosing Questions Used in a Questionnaire to Test Knowledge and Application of Information*
The pre- and post-module knowledge assessment questionnaires were completed electronically by the EOM participants and on paper by the F2FL module participants. All questionnaires were marked by the same marker (R.T.) using a standardized answer key. The numbers of points available for the pre- and post-module knowledge assessments were slightly different, so the scores were normalized to a 10-point scale. For each participant, the pre-module score for questions related to each objective was subtracted from the post-module score to generate a difference value, termed the improvement score. The mean improvement scores for the 2 arms were compared using the Student t test for the primary non-inferiority and secondary objectives. Participant satisfaction data were collected and compared using the Mann–Whitney U test. All statistical analyses were conducted on a per-protocol basis. Subjective comments were also collected as part of the post-module assessment.
Of the 123 eligible participants, 74 gave consent. Thirty-eight were assigned to the EOM group and 36 to the F2FL group. In total, 65 pharmacists completed the pre- and post-module knowledge assessment questionnaires, 30 from the EOM and 35 from the F2FL module. Fifty-four participants went on to complete the follow-up questionnaire, 26 in the EOM group and 28 in the F2FL group (see Figure 1).
|
|
||
|
Figure 1. Enrolment of participants and study profile in a randomized comparison of learning through an enhanced online module (EOM) and face-to-face learning (F2FL). |
||
The baseline characteristics of the 2 groups were similar (data not shown). In particular, participants had similar levels of clinical experience and ranked themselves as having low clinical knowledge of insulin dosing but a high level of interest. The dropout rate was higher in the EOM arm than in the F2FL arm (Figure 1).
For questions related to knowledge application (the primary objective), the improvement score achieved by the EOM learning group was 26 percentage points greater than that achieved by the F2FL group (95% confidence interval [CI] 25 to 27), whereas for questions testing knowledge gain (secondary objective), the improvement score achieved by the EOM group was 7 percentage points less than that achieved by the F2Fl group (95% CI 2 to 12) (see Figure 2). For the non-inferiority analysis, the upper bound of the 95% CI for the difference in improvement score between the 2 groups did not exceed 25 percentage points favouring F2FL (Figure 3); therefore, we concluded that the EOM mode of learning was non-inferior to F2FL. Because of large variation in the timing of completion of follow-up questionnaires (especially among EOM participants) and low rates of questionnaire return, there were insufficient data for analysis of knowledge retention. Better coordination and communication with participants might have improved data collection at this time point.
|
|
||
|
Figure 2. Mean improvement in questionnaire scores after completion of the learning module, according to study objectives. For each participant who completed the post-module questionnaire, pre- and post-module test scores were calculated for each objective (based on results for the questions pertinent to each objective). The pre-module test score was then subtracted from the post-module test score to generate the improvement score for each objective. Improvement scores for each objective were then averaged. |
||
|
|
||
|
Figure 3. Difference in mean score improvement between enhanced online module and face-to-face learning module for clinical application and knowledge gain, expressed in relation to the defined non-inferiority margin. For the primary objective, the difference in improvement score between the 2 groups was 26 percentage points (95% confidence interval [CI] 25 to 27, p < 0.001), favouring the online learning module. For the secondary objective, the difference in improvement score between the 2 groups was 7 percentage points (95% CI 2 to 12, p = 0.008), favouring the face-to-face learning module. |
||
There was no significant difference in participant satisfaction between the EOM and F2FL groups (overall p = 0.62) (Figure 4).
|
|
||
|
Figure 4. Mean satisfaction scores for various aspects of the learning experience, as graded on a 5-point Likert scale, where 1 = not at all satisfied and 5 = very satisfied. NSS = not statistically significant (Mann–Whitney U test). |
||
The results of this study suggest that the self-paced EOM was non-inferior to live (face-to-face) delivery of similar material for teaching pharmacists about contemporary insulin dosing concepts. In addition, the pharmacists were at least as satisfied with the EOM as a method of delivery of education as they were with F2FL. This finding is important because EOM offers advantages over F2FL for large health regions. In particular, EOM does not require expert facilitators to travel large distances to teach. Rather, use of an EOM allows unlimited numbers of people over any geographic area to access the same educational experience. The results reported here are similar to those from previous studies that have compared online and classroom education in other disciplines.2,3,17
Although satisfaction with the 2 modes of delivery was high and similar, participants made more negative anecdotal comments about the EOM than about the F2FL sessions. Explanations for this difference may include the high quality of the live presentation, participants’ familiarity with the F2FL design, technical challenges with the EOM software, or greater comfort in critiquing a computer than a colleague. The dropout rate (see Figure 1) was higher in the EOM group than in the F2FL group. This may have been related to the self-paced nature of the EOM and may be a potential limitation of this mode of teaching.
The efficacy of the EOM and F2FL modules depended on the type of learning that was being assessed. For obvious reasons, the primary objective of this study was to evaluate application of acquired knowledge, not just its recall. The finding that EOM outperformed F2FL in terms of knowledge application (see Figure 2) may encourage acceptance of EOMs by health administrators or educators.
The scenario in which a difference in improvement score falls wholly between the non-inferiority margin and the line of no effect, as in the knowledge gain analysis reported here (see Figure 3), is unusual and emphasizes that questions testing knowledge application must be well crafted to avoid overlap with those testing knowledge gain.18 This type of result usually occurs with very large populations or with non-inferiority margins that are too wide.19 The study population here was not particularly large, but the non-inferiority margin may have been too lenient. However, even if we had reduced the non-inferiority margin by half, to 12.5%, the data would still have fallen wholly within the margin. Even with this reduced margin, we would have concluded non-inferiority of EOM relative to F2FL. More research may clarify the situation further.
This study had several limitations. Cross-communication between groups was possible, since the F2FL sessions at each site were run before most EOM participants had had a chance to complete the module. Also, differences in study start date between sites had the potential to influence test scores, as some pharmacists worked at multiple sites. Because we wanted to be able to easily and directly compare data obtained before and after the module, as well as between trial arms, we used identical questions. To mitigate the potential for memorization, the post-module knowledge assessment questionnaires in both groups had several extra questions that were not part of the pre-module assessments. For future studies, researchers should consider creating tests with different questions of similar difficulty or altering the order of answers in multiple-choice problems. The time required to complete the knowledge assessment questionnaires was longer than anticipated. Because the F2FL sessions were conducted over a noon-hour lunch break, some participants may have rushed to complete the questionnaire and return to work. Likewise, EOM participants may have rushed to complete the questionnaires during non-work hours to attend to private matters. Either way, unaccountable bias may have influenced the data. The learning environment in which the participants completed the modules was different between the EOM and the F2FL groups. This difference was intentional, as we wanted to be able to measure differences in convenience between the 2 modules, if present. However this also meant that the environment for the EOM participants was not controlled, which created another potential source of bias that may have affected the results. EOM participants completed the pre- and post-module assessments online, whereas F2FL participants completed paper versions. Again, this difference may have introduced bias into the data. Lastly, there is some disagreement among researchers on the value of randomized controlled trials in education. Some consider this type of study to be fraught with confounding variables,20 whereas others view them as the acknowledged “gold standard of evaluative research”.21
Future studies should examine other topics with application and knowledge-based information to determine whether these results can be generalized beyond insulin dosing.
For pharmacists working in a large multicentre health region, self-paced learning by EOM was non-inferior to facilitated F2FL in terms of improving application of skills and knowledge gain related to insulin dosing. The EOM format demonstrated superiority with respect to scores on clinical application questions, and poorer improvement in scores for questions assessing knowledge gain. Differences in pharmacist satisfaction between the 2 groups were not statistically significant.
1. Fordis M, King JE, Ballantyne CM, Jones PH, Schneider KH, Spann SJ, et al. Comparison of the instructional efficacy of Internet-based CME with live interactive CME workshops: a randomized controlled trial. JAMA. 2005;294(9):1043–51.
2. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300(10):1181–96.
3. Wutoh R, Boren SA, Balas EA. eLearning: a review of Internet-based continuing medical education. J Contin Educ Health Prof. 2004;24(1):20–30.
4. Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? A systematic review of the variability of interventions. Med Educ. 2010;44(8):765–74.
5. Balen RM, Jewesson PJ. Pharmacist computer skills and needs assessment survey. J Med Internet Res. 2004;6(1):e11. 
6. Cook DA. Where are we with web-based learning in medical education? Med Teach. 2006;28(7):594–8.
7. Inzucchi SE. Clinical practice: management of hyperglycemia in the hospital setting. N Engl J Med. 2006;355(18):1903–11.
8. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee. In-hospital management of diabetes. In: Canadian Diabetes Association 2008 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes. 2008;32 Suppl 1:S71–6.
9. American Diabetes Association. Standards of medical care in diabetes—2012. Diabetes Care. 2012;35 Suppl 1:S11–63.
10. Wild S, Roglic G, Green A, Sicree R, King H. Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care. 2004;27(5):1047–53.
11. ISMP’s list of high-alert medications. Horsham (PA): Institute for Safe Medication Practices; 2012 [cited 2013 Jan 20]. Available from: www.ismp.org/Tools/institutionalhighAlert.asp
12. Horiuchi S, Yaju Y, Koyo M, Sakyo Y, Nakayama K. Evaluation of a web-based graduate continuing nursing education program in Japan: a randomized controlled trial. Nurse Educ Today. 2008;29(2):140–9.
13. Weeks DL, Molsberry DM. Pediatric advanced life support re-training by videoconferencing compared to face-to-face instruction: a planned non-inferiority trial. Resuscitation. 2008;79(1):109–17.
14. Bell DS, Harless CE, Higa JK, Bjork EL, Bjork RA, Bazarquan M, et al. Knowledge retention after an online tutorial: a randomized educational experiment among resident physicians. J Gen Intern Med. 2008;23(8): 1164–71.
15. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909–22.
16. Lam-Annotates M, Ratnapalan S, Tait G. Electronic continuing education in the health professions: an update on evidence from RCTs. J Contin Educ Health Prof. 2009;29(1):44–51.
17. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Acad Med. 2002;77(10 Suppl):S86–93.
18. Fenwick T, Parsons J. The art of evaluation: a handbook for educators and trainers. Toronto (ON): Thomson Education Publishing; 2000.
19. Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ; CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006;295(10):1152–60.
20. Torgerson CJ. Educational research and randomised trials. Med Educ. 2002;36(11):1002–3.
21. Norman G. RCT = results confounded and trivial: the perils of grand educational experiments. Med Educ. 2003;37(7):582–4.
Competing interests: None declared.
Canadian Journal of Hospital Pharmacy , VOLUME 66 , NUMBER 4 , July-August 2013
