Atonia during rapid eye movement (REM) sleep is absent in patients with REM sleep behavior disorder (RBD), a phenomenon called REM sleep without atonia (RWA). RBD patients have symptoms in common with neurodegenerative diseases, and data from follow-up studies on idiopathic RBD patient indicate that RBD predicts development of neurodegenerative diseases, particularly Parkinson’s disease (PD). Therefore, early diagnosis of RWA can help identify and possibly prevent neurodegenerative diseases. Currently, RWA assessment by visual analysis of polysomnogram (PSG) is only moderately reliable and extremely time-consuming, making it difficult to obtain objective, quantifiable results. We developed an algorithm to automatically quantify tonic and phasic electromyographic (EMG) activities of the musculus mentalis during REM sleep using the scoring manual proposed by the American Academy of Sleep Medicine. Hilbert transform and average rectification were used to calculate the amplitudes of phasic and tonic muscular activities, respectively. Parameter values in the algorithm were optimized by cross-referencing the classification result obtained from the algorithm with the result from epoch-by-epoch visual inspection by a neurologist. A total of 2315 REM epochs from 24 PD patients were analyzed. We calculated the optimal parameter set, at which the sum of sensitivity and specificity was the highest, as well as the area under the receiver operating characteristic (ROC) curve (AUC). Verification tests showed good detection accuracy (phasic: sensitivity = 88%, specificity = 82%, AUC = 0.92; tonic: sensitivity = 88%, specificity = 85%, AUC = 0.93). Thus, this automated RWA detection algorithm is potentially useful for rapid and accurate diagnosis of RBD.