Electroencephalogram (EEG) signals are critical for detecting abnormal brain activity, but their high dimensionality and complexity pose significant challenges for effective analysis. In this paper, we propose CwE-T, a novel framework that combines a channel-wise convolutional neural network (CNN)-based encoder with a single-head transformer classifier for efficient EEG abnormality detection. The channel-wise encoder compresses raw EEG signals while preserving channel independence, reducing computational costs and retaining biologically meaningful features. CwE-T was evaluated using two public datasets. For the TUH Abnormal EEG Corpus, the proposed model achieved 85.0% accuracy, 76.2% sensitivity, and 91.2% specificity at per-case level, outperforming baseline models such as EEGNet, Deep4Conv, and FusionCNN. For the CHB-MIT dataset, the proposed model achieved 85.4% sensitivity and 90.0% specificity for per-signal evaluation. Furthermore, CwE-T requires only 202M FLOPs and 2.9M parameters, making it significantly more efficient than transformer-based alternatives. The framework incorporates a channelwise design that provides potential for interpretability, offering promising directions for future research in neuroscience and clinical applications. The source code is available at https://github.com/YossiZhao/CwE-T.