We propose data augmentation using fractal images to train deep learning models for medical image diagnosis. Deep learning models for image classification typically demand large datasets, which can be challenging in the context of medical image diagnosis. Current approaches often involve pre-training of model parameters using natural image databases such as ImageNet and fine-tuning of the parameters with specific medical image data. However, natural and medical images have distinct characteristics, which questions the suitability of pre-training using natural image data. Moreover, the scalability of natural image databases is limited; thus, acquiring sufficient data for large-scale deep learning models is difficult. In contrast, Kataoka et al. introduced a mathematical model for generating image data and demonstrated its effectiveness when used in pre-training for natural image classification. In this study, we employed a pre-trained model utilizing fractals among mathematical models and experimentally classified CT images of COVID-19 pneumonia. The experimental results demonstrated that this fractal-based pre-training model achieved accuracy comparable to conventional natural image-based approach. Fractal images are easily generated compared to natural images. Furthermore, generating appropriate data for specific applications may be possible by adjusting the parameters. This flexibility in generating data allows customization and optimization of the model for different scenarios or specific requirements. We believe that this approach holds promise in medical image diagnosis, where the number of samples is often limited.