To prevent aspiration pneumonia, a system for non-invasive and quantitative evaluation of the swallowing function is required. Therefore, we have previously proposed a method of using ultrasound videos to establish evaluation indicators of the swallowing function. The proposed method aims to automatically estimate the velocities of the esophageal wall region and the bolus region in the ultrasound video. In this method, estimation of the bolus region comprises two steps: estimating the esophageal region through which the bolus flows and extracting only the frame in which the bolus passes through the esophageal region. However, the step of extracting the frame in which the bolus passes is still performed manually. Therefore, to automate this step, the purpose of this study was to automatically determine the frame in which the bolus flowed into the screen. This method was tested five times on five healthy adult male subjects by recording a cervical ultrasound video while swallowing a bolus of water. We identified the different elements of the esophageal region in the video by first identifying the esophageal wall region with the maximally stable extremal regions (MSER). Then, we used the luminance histogram of each frame to establish the graph of the histogram similarity. This, in turn, was used to detect a change in the observed region, thus indicating the inflow of the bolus. Moreover, we could distinguish the change caused by the inflow from the change caused by the elevation of the esophageal wall using the velocity results obtained by optical flow estimation in the anterior esophageal wall region. Our results showed that in most cases, the proposed method was successful in recognizing the inflow of the bolus and distinguishing it from the elevation of the esophageal wall. Furthermore, an accuracy sufficient for estimation of the velocity of the bolus was achieved.