Human error during surgery can cause unexpected accidents. Elucidating the causes of human error by monitoring the behavior of the surgical team can improve safety. Conventional methods include a questionnaire that is presented to the surgical team, but this is subjective, and its accuracy depends on memory. However, pose estimation is an emergent technology that is able to capture a person’s skeletal information. Objective monitoring of group behaviors using pose estimation may assist in identifying the causes of human error. In this study, we utilized the surveillance video images of the entire surgical team to verify the effectiveness of the proposed method in extracting and quantifying group behaviors in order to identify the intraoperative situation. Specifically, we attempted to extract the behaviors focusing on the actions of handing over surgical instruments and the actions that multiple members pay attention to simultaneously. In the proposed method, pose estimation is applied to surveillance camera images to model individual behavior based on the individual’s position, facial orientation, body orientation, arm bending, and wrist position. The individual’s behavior and ID are then combined and input into an estimation model to extract group behavior. The proposed method was applied to six actual surgical videos to estimate group behaviors. Accordingly, we were able to accurately extract behaviors involving cooperation of multiple members in addition to behaviors involving receiving and delivering, as well as behaviors warranting the attention of multiple members. Further, we successfully identified important points of occurrence of characteristic group behaviors from redundant surgical videos.