Mona Jabbour , Sarah Reid , Danica Irwin , Andrea Losier , Eleanor Holmgren , Dennise Albrecht , Kristina Rohde , Katherine Moreau
Pediatric patients represent about 25% of emergency department (ED) visits in Ontario.1 Although pediatric EDs work to ensure best-evidence care in their settings, most pediatric visits occur outside these centres. In fact, more than 80% of Ontario pediatric emergency visits occur in general hospital EDs,1 representing over 800 000 visits to sites that are less likely to have pediatric expertise, medications, equipment, and policies in place. Health care professionals in these settings may be less comfortable delivering pediatric care because of limited training2–5 or because they lack familiarity with pediatric medications and dosing. Even though children might receive better care from pediatric subspecialists, their care will continue to be provided in general EDs.6 All children should receive quality emergency care, regardless of where they present.
The Children’s Hospital of Eastern Ontario (CHEO), a large academic general hospital, and 14 community hospitals lie within the geographic region of the Champlain Local Health Integrated Network (LHIN). In 2012/2013, CHEO had 66 051 ED visits and 6245 admissions. Beyond CHEO, pediatric expertise is limited within this LHIN, as the other centres have less or no access to in-house pediatric consultants. However, pediatric visits account for 15%–40% of ED visits at these community hospitals, and 2 of the sites admit pediatric patients.
In 2007, funding was received for a 1-year project to develop and implement an outreach program with the goal of helping nonpediatric health care professionals to deliver standardized, evidence-based pediatric emergency care across the Champlain LHIN. The outreach program focused on using clinical pathways for a set of common pediatric conditions seen in the ED. A clinical pathway is a tool to operationalize best-evidence guidelines into accessible bedside formats for health care provider teams.7 To support the use of clinical pathways, health care professionals were offered education workshops, delivered by an interprofessional ED team, and auxiliary resources through an outreach website.
With judicious budgeting and ongoing interest, the outreach program has continued for 6 additional years. To formally assess the program’s impact, grant funding was secured in 2010 to conduct a mixed-methods program evaluation. This report describes the ED Outreach Program and preliminary evaluation results, specifically participant satisfaction with the educational workshops and results of knowledge tests conducted before and after the workshop (pre/post tests). Subsequent evaluation findings, including qualitative interviews, pre/post chart audits, and the benefits, harms, failures, and unanticipated outcomes of the program, will be reported in a subsequent paper.
The outreach program was targeted toward ED staff at the academic teaching hospital and the 14 community hospitals located within a 3-h drive from CHEO.
The shared system goals of the ED Outreach Program (see Box 1) were established with site partners in 2007. For pragmatic reasons, the program was launched with 8 pilot sites in 2007, before rollout and full participation of all 15 sites by 2009. The 8 pilot sites were selected according to first come, first served responses to an e-mail invitation to participate. Subsequent rollout was based on the logistics of coordinating sessions at each site outside CHEO and availability of site staff. Components of the outreach program are described below.
Box 1. Shared and System Goals for the Champlain LHIN ED Outreach Project
After hospital approvals were secured, a CHEO ED pharmacist and nurse team conducted site visits to identify needs and resources, to promote project awareness, and to demonstrate the interest of the outreach team in working within each site’s capacity. Specific information gathered during this phase included data on pediatric visits and common diagnoses, staffing levels, experience with clinical pathways, availability of pediatric medications in appropriate dosages, availability of pediatric resuscitation equipment, computer access, and existing educational opportunities.
The ED outreach team consisted of 2 physicians (S.R., A.L.), 1 nurse, and 1 pharmacist (D.I.) from the CHEO ED. Led by the CHEO Partnership and Advocacy Director (D.A.), a steering committee (consisting of the ED outreach team and 1 administrator and 1 clinical director from each site) was formed to guide program development, implementation, and evaluation.
Clinical pathways were integral to this initiative because of their capacity to generate standardized evidence-based practices, patient safety, and efficiency. Specific management recommendations and drug dose guidelines enhance medication safety and allow staff to focus on more complex tasks.8–15 A full clinical pathway includes a preprinted order set, which can significantly reduce prescription errors,16 a medical directive, and health education resources. From among the clinical pathways developed by clinical and research experts at CHEO, a set of 4 clinical pathways addressing common pediatric conditions seen in the ED was selected: bronchiolitis, croup, gastroenteritis, and asthma. A CHEO diabetic ketoacidosis (DKA) treatment algorithm was also included because of recognized concerns resulting from differences between adult and pediatric DKA management. The CHEO documents were adapted to ensure relevance and feasibility for use in community ED settings.
Each site approved the final versions of the clinical pathways. Tripartite branding, which referred to the community hospital, CHEO, and the Champlain LHIN, indicated to users that the clinical pathways represented quality standardized pediatric care.
During the site visits, the outreach team identified the need for an intranet website to host the program’s resources. The website was launched in 2009, with input from information technology departments at the various locations. The clinical pathways, preprinted order sets, parent education handouts, videotaped teaching sessions, parenteral and neonatal drug manuals, drug calculation programs, a pediatric dosage manual, and narcotic dosing pocket cards were shared online.
Drug selection and pediatric dosing are challenging in general hospital EDs, especially during infrequent pediatric resuscitations, which have been associated with high rates of medication errors.17–19 The outreach pharmacist addressed this challenge by providing teaching for the calculation and preparation of pediatric medications.20 CHEO’s drug calculation programs to facilitate safe dosing and preparation of high-alert medications were shared online for medications used in the following situations: pediatric advanced life support, anaphylaxis, rapid-sequence intubation, seizures, analgesia and sedation, sepsis and meningitis, respiratory conditions, and pediatric drug infusions.
Initial site-based workshops, which used case-based, interactive, pathway-specific instruction, were provided for ED health care professionals. The educator at each site was involved through a train-the-trainer model to ensure that all staff would ultimately receive the education. Workshops were integrated into each site’s existing education days. First-year funding allowed reimbursement for nurses’ educational time to attend the workshop. The ED Outreach Program nurse and pharmacist linked (via personal communication) with site educators and pharmacists to provide additional support. Community ED physicians initially participated in separate sessions administered by outreach physicians. However, in response to feedback from participants, combined interprofessional sessions were provided in later years of the program, to reinforce the true team focus of the clinical pathways. Refresher workshops were provided for new staff and those unable to attend previously, and recorded videos were posted online. To encourage online participation, information links were provided for collecting education credits.
Initial program funding was provided through a 1-year grant from the Ontario Ministry of Health and Long-Term Care. Judicious budgeting, along with a smaller additional grant, allowed the program to be offered for an additional 6 years. The funds were allocated primarily to salaries for the outreach pharmacist and nurse, development of health education materials, and nursing relief costs of the community hospitals. In 2010, grant funding was obtained from the Academic Health Sciences Centre Academic Funding Plan Innovation Fund to evaluate the program.
With funding to assess the program’s impact and effectiveness, a formal program evaluation was launched in 2010 in partnership with the 8 pilot sites. The evaluation assessed implementation of the clinical pathways for bronchiolitis, croup, and asthma, as well as the preprinted orders. The 2 other clinical topics (gastroenteritis and DKA) introduced as the ED Outreach Program evolved were included in the pre/post knowledge tests, but they were not part of the formal evaluation. The 7 sites that did not participate in the evaluation were, at the time of writing (late 2013), at various stages of implementing the clinical pathways, having all received outreach training and resources.
A meeting of the Evaluation Team (which included an administrative and/or medical lead and site champion from each site) in April 2010 revealed that despite identified need and interest, the clinical pathway tools were not fully implemented at any of the 8 pilot sites. Although outreach education and access to resources had been provided at all 8 sites, hospital approval processes were incomplete at 2 sites, and the clinical pathways were inconsistently available at the other 6 sites. For example, clinicians often referenced the website and transcribed orders onto blank order sheets, instead of using the preprinted order sets, which were intended for efficiency and prescribing accuracy. To account for this incomplete implementation, several barriers were identified, such as wait time pressures, system-wide hospital initiatives, and other competing demands. It was agreed that further work to complete implementation would be required before an evaluation could be conducted. With support from the administrative lead and site champion at each site, the Evaluation Team committed to ensuring the availability of all clinical pathway documents at all sites, repeating pathway-specific refresher workshops at each site, and performing the evaluation. Approval for the evaluation component of the study was obtained from the research ethics board at CHEO and at each site.
Informed by the Kirkpatrick model for evaluating programs,21 a total of 7 evaluation questions, clustered into 3 domains (reactions, learning, practice behaviour), were developed (see evaluation framework, Table 1). Focusing on the first 2 domains, the preliminary evaluation described here covers the results of participant satisfaction surveys and the pre/post knowledge tests.
The 8 sites participating in this evaluation were nonacademic community hospitals. Two were located in urban centres and 6 in small to medium-sized rural communities. Because of logistic and staffing challenges, one site withdrew and was replaced with another committed site.
Three types of data collection instruments were used: a demographic questionnaire, a participant reaction questionnaire, and profession-specific knowledge tests.
Each participant completed an anonymous demographic questionnaire to report their clinical background, clinical experience, and prior training through the CHEO ED Outreach Program.
After the workshop, participants completed a validated satisfaction questionnaire, with a 4-point Likert-type scale for each question.22 The questionnaire consisted of 8 closed-ended questions focusing on amount of training received, whether it helped with understanding the topics, and whether participants would return for future sessions. Space was provided for additional open-ended comments.
The education workshops were designed to guide use of the clinical pathways. Therefore, pre/post knowledge tests were administered to ensure the sessions resulted in knowledge acquisition. The tests were developed by content experts on the study team and were pretested for clarity with local physicians and nurses. The test for physicians consisted of 15 multiple-choice questions related to bronchiolitis, croup, gastroenteritis, asthma, and DKA. The 12-question test for nurses involved similar content but focused on their relevant responsibilities. For each group, the same multiple-choice questions were used for testing before and after the workshop, but the order of questions was shuffled to minimize recall.
Evaluation packages were distributed at the beginning of each refresher workshop. Consenting participants completed the profession-specific demographic questionnaire and multiple-choice “pre” knowledge test before the workshop began. They completed the reaction questionnaire and multiple-choice “post” knowledge test at the end of the workshop.
Questionnaire and knowledge test data were analyzed using SPSS, version 19 (IBM, Armonk, New York). All non-parametric data, including percentages for the demographic and reaction questionnaires and mean scores for the pre/post knowledge tests, were analyzed using descriptive statistics, with frequencies and overall percentages reported for each response. Average total scores were also calculated for the participant reaction questionnaire. For scoring of the knowledge tests, an answer key was developed and converted into SPSS syntax to assess and compare the number of participants who answered each question correctly in the pre/post knowledge tests. The difference was examined through a paired analysis using the 2-sided t test, with p values less than 0.05 considered significant.
A total of 98 health care professionals participated in out-reach workshops at the 8 pilot sites. Of these 98 participants, 85 (42 physicians and 43 nursing professionals) completed a demographic questionnaire, 84 (41 physicians and 43 nursing professionals) completed the reaction questionnaire, and 79 (38 physicians and 41 nursing professionals) completed both the “pre” and “post” knowledge tests. Although some pharmacists and respiratory therapists attended parts of the outreach workshop, staffing logistics prevented professionals in these roles from attending the complete workshop, and they did not complete any of the questionnaires.
The largest proportion of participants (40%) had been working in their respective emergency departments for less than 5 years, and the majority (64%) had not attended a previous CHEO ED Outreach session.
On the basis of both numeric ratings (Table 2) and written comments, participants’ reactions to the workshops were positive: 86% (71/83) rated the quality of training received as excellent, 86% (70/81) would recommend the ED Outreach training session to a colleague, and 89% (75/84) would return to the program for future assistance.
Table 2.
Participant Reactions
In terms of knowledge acquisition, the average score for physicians increased from 6.5/15 before to 10.4/15 after the workshop, indicating a significant knowledge increase ( p < 0.001). Similarly, the average score for nurses increased from 7.2/12 to 9.1/12, which was also a significant increase ( p < 0.001).
The ED Outreach Program is an innovative leadership initiative designed to train health care providers to deliver best-evidence pediatric emergency care within community ED settings. Site visits have occurred at all Champlain LHIN sites, the website is fully operational, and all documents for the 5 critical pathway tools have been adapted for and are available at the participating sites. In addition, refresher workshops have been delivered at all pilot sites.
Staff and physicians at the partner sites reported that they valued the pediatric emergency clinical pathways, although no site had achieved full implementation of these pathways at the time the evaluation was launched. Implementation barriers identified included wait-time pressures, system-wide hospital initiatives, and other competing demands in the ED. Without this evaluation, or other monitoring, further implementation would have waned. The program evaluation and partnership team provided impetus for renewal of commitment and full implementation of the clinical pathways.
Refresher workshops delivered to support the use of clinical pathways were successful, as indicated by participant satisfaction questionnaires and tests of knowledge acquisition. Findings from preliminary evaluations such as these must be established before higher-order impacts, such as effects on practice behaviour and patient outcomes, can be evaluated.
Studies of educational programs for health care professionals have yielded variable results, especially in terms of changing practice behaviours. A Cochrane review provided evidence that educational meetings, either alone or combined with other interventions, led to small improvements in practice and patient outcomes, but educational sessions alone were insufficient to effectively change complex behaviours.23 The ED Outreach Program described here is innovative in its focus on clinical pathway tools to reinforce best-evidence care at the bedside.
In terms of study limitations, the regional conditions may be unique. As such, the results of this evaluation may not be generalizable to other locations. Limited access to pediatric consultants in the community hospitals may have enhanced interest in the program. However, discomfort with managing pediatric patients likely occurs in most general EDs.3,5
The numbers of participants in the refresher workshops were small. At most sites, staff numbers were limited, meaning that relatively few staff members were available to attend. To minimize travel and promote team-based learning, sessions were held at each site. Among those who did attend, motivated staff may have been overrepresented, and those more in need of training may have been absent. To optimize access to training, an additional open session was held at CHEO and videos of the workshops were posted online. Knowledge retention beyond the refresher workshops was not assessed, as it is unlikely to persist without reinforcement. This was the rationale for providing ongoing knowledge support through the ED Outreach Program website and through the clinical pathways directly.
This ED outreach initiative leveraged the use of clinical pathways to operationalize best evidence for pediatric emergency care. Delivered to guide the use of these tools, the refresher workshops were highly valued and associated with significant knowledge gains. Ongoing support is being provided through access to auxiliary resources. To promote further uptake, future activity will focus on continued engagement, site-specific feedback, and discussion on use of the tools. A more complex analysis of this knowledge translation and program uptake will be reported in a subsequent qualitative analysis paper.
1. Emergency departments and children in Ontario. Ottawa (ON): Canadian Institute for Health Information; 2008 Apr [cited 2013 Feb 6]. Available from: www.cihi.ca/cihi-ext-portal/internet/en/document/health+system+performance/access+and+wait+times/release_24apr08
2. Casey MM, Wholey D, Moscovice IS. Rural emergency department staffing and participation in emergency certification and training programs. J Rural Health. 2008;24(3):253–62.
3. Tolhurst H, McMillan J, McInerney P, Bernasconi J. The emergency medicine training needs of rural general practitioners. Aust J Rural Health. 1999;7(2):90–6.
4. Examination of special competence in emergency medicine. Toronto (ON): College of Family Physicians of Canada; [cited 2013 Feb 6]. Available from: www.cfpc.ca/ExamofSpecialCompetenceinEmergencyMedicine/
5. Bhimani M, Dickie G, McLeod S, Kim D. Emergency medicine training demographics of physicians working in rural and regional southwestern Ontario emergency departments. CJEM. 2007;9(6):449–52.
6. Prentiss KA, Vinci R. Children in emergency departments: who should provide their care? Arch Dis Child. 2009;94(8):573–6.
7. De Bleser L, Depreitere R, De Waele K, Vanhaecht K, Vlayen J, Sermeus W. Defining pathways. J Nurs Manag. 2006;14(7):553–63.
8. Kurtin P, Stucky E. Standardize to excellence: improving the quality and safety of care with clinical pathways. Pediatr Clin North Am. 2009; 56(4):893–904.
9. Vanhaecht K, De Witte K, Panella M, Sermeus W. Do pathways lead to better organized care processes? J Eval Clin Pract. 2009;15(5):782–8.
10. Browne GJ, Giles H, McCaskill ME, Fasher BJ, Lam LT. The benefits of using clinical pathways for managing acute paediatric illness in an emergency department. J Qual Clin Pract. 2001;21(3):50–5.
11. Thomson P, Angus NJ, Scott J. Building a framework for getting evidence into critical care education and practice. Intensive Crit Care Nurs. 2000;16(3):164–74.
12. McCue JD, Beck A, Smothers K. Quality toolbox: clinical pathways can improve core measure scores. J Healthc Qual. 2009;31(1):43–50.
13. Kent P, Chalmers Y. A decade on: has the use of integrated care pathways made a difference in Lanarkshire? J Nurs Manag. 2006;14(7):508–20.
14. American Academy of Pediatrics; Committee on Pediatric Emergency Medicine; American College of Emergency Physicians; Pediatric Committee; Emergency Nurses Association Pediatric Committee. Joint policy statement—guidelines for care of children in the emergency department. Pediatrics. 2009;124(4):1233–43.
15. Gaddis GM, Greenwald P, Huckson S. Toward improved implementation of evidence-based clinical algorithms: clinical practice guidelines, clinical decision rules, and clinical pathways. Acad Emerg Med. 2007;14(11): 1015–22.
16. Kozer E, Scolnik D, MacPherson A, Rauchwerger D, Koren G. Using a preprinted order sheet to reduce prescription errors in a pediatric emergency department: a randomized, controlled trial. Pediatrics. 2005; 116(6):1299–302.
17. Kozer E, Seto W, Verjee Z, Parshuram C, Khattak S, Koren G, Jarvis DA. Prospective observational study on the incidence of medication errors during simulated resuscitation in a paediatric emergency department. BMJ. 2004;329(7478):1321. 
18. Lesar TS, Lomaestro BM, Pohl H. Medication-prescribing errors in a teaching hospital. A 9-year experience. Arch Intern Med. 1997;157(14): 1569–76.
19. Luten R, Wears RL, Broselow J, Croskerry P, Joseph MM, Frush K. Managing the unique size-related issues of pediatric resuscitation: reducing cognitive load with resuscitation aids. Acad Emerg Med. 2002;9(8):840–7.
20. Irwin D, Vaillancourt R, Dalgleish D, Thomas M, Grenier S, Wong E et al. Standard concentrations of high-alert infusions across paediatric acute care. Paediatr Child Health 2008;13(5):371–6.
21. Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. 3rd ed. San Francisco (CA): Berrett-Koehler Publishers; 2005.
22. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Eval Progr Plann. 1979;2(3):197–207.
23. Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009;(2):CD003030.
Competing interests: None declared.
The outreach program was supported by the Ontario Ministry of Health and Long-Term Care (MOHLTC). The program evaluation was supported by the Academic Health Sciences Centre Academic Funding Plan Innovation Fund, Ontario MOHLTC (a peer-reviewed grant competition).
Canadian Journal of Hospital Pharmacy , VOLUME 67 , NUMBER 3 , May-June 2014