Title:Inter-rater Reliability of Preceptors on Clinical Pharmacy Competency Evaluation
Volume: 16
Issue: 2
Author(s): Dixon Thomas*, Sherief Khalifa, Jayadevan Sreedharan and Rucha Bond
Affiliation:
- Clinical, College of Pharmacy, Gulf Medical University, Ajman,United Arab Emirates
Keywords:
Inter-rater reliability, preceptorship, pharmacy, clinical competence, correlation, coefficient.
Abstract:
Background: Clinical competence of pharmacy students is better evaluated at their practice
sites compared to the classroom. A clinical pharmacy competency evaluation rubric like that of
the American College of Clinical Pharmacy (ACCP) is an effective assessment tool for clinical
skills and can be used to show item reliability. The preceptors should be trained on how to use the
rubrics as many inherent factors could influence inter-rater reliability.
Objective: To evaluate inter-rater reliability among preceptors on evaluating clinical competence
of pharmacy students, before and after a group discussion intervention.
Materials and Methods: In this quasi-experimental study in a United Arab Emirates teaching hospital,
Seven clinical pharmacy preceptors rated the clinical pharmacy competencies of ten recent
PharmD graduates referring to their portfolios and preceptorship. Clinical pharmacy competencies
were adopted from ACCP and mildly modified to be relevant for the local settings.
Results: Inter-rater reliability (Cronbach's Alpha) among preceptors was reasonable being practitioners
at a single site for 2-4 years. At domain level, inter-rater reliability ranged from 0.79 - 0.93
before intervention and 0.94 - 0.99 after intervention. No inter-rater reliability was observed in relation
to certain competency elements ranging from 0.31 - 0.61 before the intervention, but improved
to 0.79 - 0.97 after the intervention. Intra-class correlation coefficient improved among all individual
preceptors being reliable with each other after group discussion though some had no reliability
with each other before group discussion.
Conclusion: Group discussion among preceptors at the training site was found to be effective in improving
inter-rater reliability on all elements of the clinical pharmacy competency evaluation. Removing
a preceptor from the analysis did not affect inter-rater reliability after group discussion.