OBJECTIVES: POCUS is a core emergency medicine skill and mainstay of early pregnancy assessment. The ultrasound competency assessment tool was developed as an entrustment-based assessment tool for use by content experts evaluating trainees performing multiple POCUS study types. The objective of this study was to evaluate the scoring and extrapolation inferences of the tool within Kane's validity framework when used to assess trainees performing an early pregnancy POCUS. METHODS: This was a multicentered study of emergency medicine residents participating in a POCUS assessment. After a background questionnaire, participants were read a case stem requesting a POCUS evaluation of an early pregnancy patient. Trainees were independently assessed by two fellowship-trained faculty. Descriptive statistics and two-way random, intraclass correlation coefficients, Cronbach's alpha were calculated on the merged data and used to assess all domains. Domain scores and an entrustment score for each participant were used to create a composite score. A one-way analysis of variance was performed. RESULTS: 36 trainees and 5 assessors completed the study. When used to assess trainee POCUS performance in early pregnancy, the tool demonstrated good to excellent interrater reliability for image acquisition, image generation, clinical integration, and entrustment (intraclass correlation coefficients 80-91 p <
.001). The preparation domain had poor, but statistically significant interrater reliability (intraclass correlation coefficient 0.46 p = .04). An analysis of variance suggested the POCUS performance scores differed based on prior experience [F(2,32) = 3.74, p = .021). CONCLUSION: This study adds further validity evidence relating to scoring and extrapolation of the ultrasound competency assessment tool when used to assess trainees performing a POCUS study in early pregnancy.