Introduction
Feedback is fundamental to surgical training and has been found to be positively associated with performance improvement1,2 and ultimately patient outcomes after surgery.3 Residents have reported there are limited opportunities for feedback during training due to clinical pressures, duty hours, and decreased autonomy.4,5 As a result, there is an increasing need for a universal objective feedback system. Many training programs have started to implement universal rating tools to objectively assess skills.6,7 However, despite the development of good validated tools for objective surgical assessment, previous studies have shown that residents consider most of the feedback that they receive to be subjective, untimely, random, and nonactionable.8,9 This may be due to a lack of implementation of the validated tools.
Researchers in technical skill assessment in many fields have used video-recorded performances and expert reviewers to generate more data points for a single performance, lending more educational relevance to each individual performance.10–12 Expert review is very time intensive and quite expensive, making routine expert review of video-recorded performance untenable for regular educational use. Crowd-Sourced Assessment of Technical Skills (CSATS) is a commercial program that uses the power of crowd-sourced internet reviewers to provide low-cost, rapid, data-rich assessment of surgical videos.10
If objective and comparable, CSATS may be a tool that will enable more impartial, actionable, quantitative assessment of trainees. We sought to understand how crowd-sourced assessment of residents’ laparoscopic skills compares with current faculty assessment of residents’ laparoscopic skills in a single obstetrics/gynecology (Ob/Gyn) residency program.
Materials and Methods
After obtaining approval from the MedStar Health institutional review board (No. CR00001361), senior residents (postgraduate years 3 and 4) from an academic Ob/Gyn program were recruited to participate in the study from September 2018 to January 2020. All residents rotating on the minimally invasive gynecological surgery rotation were invited to participate. Their performance of major laparoscopic gynecologic cases, including myomectomies, hysterectomies, and endometriosis excisions, were video recorded. To qualify for video submission, residents were required to complete more than 50% of the case. All cases were done with the programs’ 2 minimally invasive gynecological surgery attendings. They could submit as many cases as they wanted. Individual performance videos were recorded and uploaded to the CSATS database, where they were analyzed, and written feedback reports were sent to the residents. During their cases, the residents received their standard intraoperative feedback from the attending physician at the time of surgery, as well as the standard electronic feedback forms on the program’s existing evaluation platform. The existing platform requires residents to send questionnaires to faculty after surgeries, asking for assessment. This standard feedback form is made up of 7 broad questions, only 1 of which titled “Surgical Technique,” addresses surgical skill. The in-person feedback was not standardized. The attendings gave feedback as they deemed appropriate, both during the case and/or in a debrief after.
CSATS uses anonymous crowd-sourced internet reviewers to evaluate surgical performance. It assesses techniques using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) that includes a 5-point Likert scale with verbal descriptors associated with 1, 3, and 5. Subjective written comments are also provided. Four of the GOALS domains (depth perception, bimanual dexterity, efficiency, and tissue handling) are separately assessed with the combination of all 4 averaged scores making up the final score (20 being the highest possible score). The fifth GOALS domain, autonomy, was not able to be assessed via video evaluation.
After completion of the video submission/feedback portion of the study, all residents were asked to complete a survey addressing satisfaction, timeliness, and utility of the CSATS feedback method overall and in comparison with the traditional attending feedback and the existing formal evaluation platform (see appendix for full questionnaire).
Results were analyzed by hospital system–employed statisticians. The mean and SD of the residents’ review scores were computed overall. They were then broken into low usage of CSATS (1-4 videos) or high usage (>5 videos). The mean and SD for each group was computed and compared using a t test. The P values were calculated using the Wilcoxon signed-rank test. We considered a P value less than .05 to be significant.
Results
From September 2018 to August 2020, residents submitted videos to CSATS for evaluation. A total of 16 residents submitted 65 videos of operations. Of the 16 residents, 6 (37.5%) submitted 5 or more videos and 10 (62.5%) submitted 1 to 4 videos. All residents completed the feedback satisfaction survey once at the end of the study. Residents reported low completion of the existing electronic feedback form, with residents estimating less than 5% of their surgical cases received a written procedure evaluation (mean [SD], 4.3% [3.77%]).
All CSATS feedback was received by all residents within 1 week of submission. Sixty percent of feedback was received in less than 72 hours. The remaining 40% was received within 1 week (Figure 1).
The lowest scores on the survey were the questions comparing CSATS vs immediate, in-person faculty evaluation. Overall comparison of CSATS vs faculty evaluation scored a 3.25 (3 meaning CSATS and faculty evaluations were equivalent). Residents found that CSATS response time was slower than receiving in-person feedback from faculty (Table). We were unable to assess the response time in comparison with the standard evaluation form because of the low completion rate of the form.
When comparing CSATS with the standard feedback form, residents felt strongly that CSATS was superior to the standard electronic form for surgical skill feedback and procedure evaluation, with both categories receiving a mean top score. Residents agreed that CSATS was thorough, precise, and could help with laparoscopic skills (Table). However, the majority reported that they are not practicing laparoscopy more or less based on CSATS feedback.
When comparing responses from residents with low-volume (1-4 submitted videos) CSATS vs high-volume (≥5 videos) CSATS feedback, there were no statistical differences for any of the questions (Table).
Overall, residents reported satisfaction with CSATS evaluation, with a mean score of 4.13 out of 5. Residents reported that there may be an adjunctive role for CSATS as a standard tool for laparoscopic evaluation, with no statistical difference in low-volume compared with high-volume feedback (P = .84).
Discussion
As changes in residency training have evolved, we have to continue to create new education strategies to help maximize trainees’ learning. Decreasing work hours and fewer opportunities4,5 in the operating department mean we must maximize our opportunities for effective feedback. Feedback has been shown to improve efficiency, safety, and skill.1 However, at our training hospital, laparoscopic education and evaluation have limitations. Feedback is largely subjective and varies from intraoperative feedback to general feedback at the end of a 5-week rotation. Residents have to request formal evaluations after individual cases. This process is not a residency requirement, and it is therefore infrequently completed. This likely contributed to the residents’ increased satisfaction with CSATS over the standard written feedback. The goal of the study was to investigate how crowd-sourced assessment of residents’ laparoscopic skills compares with faculty assessment of residents’ laparoscopic skills in an Ob/Gyn residency program.
Previous studies have shown that CSATS is an accurate, acceptable method for objective technical skills assessment.11 Residents were generally satisfied with CSATS as a form of feedback, both in timeliness and as an overall feedback tool. In terms of timeliness, all results were back within 7 days. This is important because previous studies have found that feedback given more than 7 days out is perceived as less effective.13 While residents agreed that CSATS should be implemented as a standard tool for evaluation, most found it was comparable with, but not better than, traditional real-time and in-person feedback. They strongly felt that it was superior to the standard feedback form. This is consistent with previous studies that have shown residents find written feedback to be a good adjunct to face-to-face feedback, but find face-to-face feedback more important to their education.13
Residents felt strongly that CSATS as a form of feedback was far better than the existing written system. The top scores that CSATS received in comparison with the existing form demonstrates the value that CSATS can add to resident surgical education in terms of formal feedback. Formal, reviewable feedback has been found to be fundamental for resident assessment and is an important component of resident education.14 A more longitudinal study of objective resident improvement after the use of CSATS could help solidify its value as a feedback tool for residents.
Despite their satisfaction with CSATS, residents did not feel they would implement new practices or exercises based on their CSATS feedback. This suggests that the practical implications of CSATS may be limited. However, in this study, residents only used CSATS for a short period, which may be why they did not consistently implement the feedback. A more longitudinal study in which residents use it consistently may show better utilization of the feedback.
Additional studies in the future may include development of new validated simulation exercises that predictably drive improvement in operative skill deficits as assessed by CSATS as well as following skill improvement by tracking CSATS scores. The long-term goal of studying CSATS in the educational arena is to drive improvement in surgical assessment, skills acquisition, patient safety, and quality surgical outcome measures. This research also demonstrated the importance to resident education of timely in-person feedback from clinical faculty.
A significant disadvantage of CSATS is the associated cost. The cost of implementing CSATS for 1 year is approximately $30 000. This can be a burden to the graduate medical education budget. Educational materials make up approximately one-third of the hidden costs of graduate medical education that are not reimbursed by the federal government.15 This could be a significant hurdle that many programs would have difficulty overcoming, especially given our findings of equivalent satisfaction with standard face-to-face feedback. However, if future studies explored resident surgical skill acquisition after using CSATS and saw an improvement in those who utilized it, the cost could be justified.
There were limitations to this study. This was a small study with only 16 participants, which increases the chance of type 2 error. Implementation of CSATS to the program was difficult. Almost two-thirds of the participants submitted fewer than 5 videos, indicating poor participation. However, there was not a significantly higher score in any category for those residents who submitted 5 or more videos, implying that there may not be increased satisfaction or value for those who use it more frequently.
While the study provided insight into residents’ opinion regarding CSATS compared with the program’s standard feedback form, the 2 feedback methods used different criteria for evaluation. CSATS uses the GOALS criteria, while the existing feedback form merely had 1 question referencing surgical technique. It is possible that the preference towards CSATS may actually be a preference towards the GOALS criteria. We did not change the existing feedback form to use the GOALS criteria for this study because we wanted a comparison of CSATS with what was already in use. A future study comparing GOALS criteria in a standard feedback form vs CSATS could potentially provide a more direct comparison of the 2 different evaluations. This study was additionally limited by the low response rate of the existing feedback form. Therefore, there may have been a bias response when comparing against CSATS, where trainees will always have a 100% response rate. However, this limitation could also be considered a strength for a tool like CSATS that allows for consistent and reliable feedback.
To our knowledge, there are no previous studies that have evaluated CSATS utility in an Ob/Gyn residency program or that assessed resident satisfaction with CSATS as a tool.
Conclusions
While the standard method of face-to-face feedback is invaluable, CSATS may be of value as an adjunct tool for trainee feedback, and it may improve on some existing written and electronic surgical evaluation systems.
Acknowledgments
We thank the statisticians at MedStar Health Research Institute for their statistical support.
Disclosures
The authors report no conflicts of interest.
Sources of Funding
This study was funded by MedStar Washington Hospital Center Graduate Medical Education.
Meeting Presentation
This study was presented at the 49th Annual Scientific Meeting of Society of Gynecologic Surgeons; March 20th, 2023; Tucson, Arizona.