The aim of this study was to identify factors affecting the relationship between heart rate variability (HRV) and physical acceleration (PA) in daily lives of free-moving adults. In 65 subjects comprising 18 young (20 to 39 years), 26 middle-aged (40 to 59 years), and 21 elderly (≥ 60 years) subjects, ECG R-R intervals and PA data were simultaneously obtained every minute for 24 hours as the subjects went about their normal daily lives. The R-R intervals were subjected to linear frequency domain analysis, and the low frequency (LF)/high frequency (HF) and HF/(LF+HF) ratios were used as HRV indices. Lag was defined as the time difference that gave maximum cross-correlation between the HRV parameters and PA. The proportion of subjects who exhibited no lag was significantly higher in young subjects, and was significantly lower in elderly subjects. After waking, the proportion of subjects who exhibited no lag tended to decrease in all groups. The lag between HRV and PA correlated significantly with psychological conditions and/or mental stress.