Text-dependent speaker verification (TD-SV), which verifies speaker identity using predefined phrases, has gained attention as a reliable contactless biometric authentication method for smart devices, internet of things (IoT), and real-time applications. However, in real-world environments, limited training data, background noise, and microphone channel variability significantly degrade TD-SV performance, particularly on resource-constrained devices that require real-time inference. To address these challenges, we propose a lightweight end-to-end TD-SV model based on a convolution-augmented transformer (Conformer) architecture enhanced with a channel-temporal attention (CTA) module as an input enhancement that specifically targets speaker-discriminative patterns in short, fixed utterances. Unlike existing attention mechanisms (Squeeze-and-Excitation Networks (SENet), Convolutional Block Attention Module (CBAM)) designed for computer vision tasks, our CTA module employs frequency-wise statistical pooling to capture acoustic variability patterns crucial for speaker discrimination within identical phonetic content. Experiments conducted on the challenging far-field and noisy SLR 85 HI-MIA dataset demonstrate that the proposed CTA-Conformer achieves an equal error rate (EER) of 2.04% and a minimum detection cost function (minDCF) of 0.20, achieving competitive performance compared to recent TD-SV approaches. Additionally, INT8 quantization reduces the model size by 75.8%, significantly improves inference speed, and enabling real-time deployment on edge devices. Our approach thus offers a practical solution for robust and efficient TD-SV in embedded internet of things (IoT) environments.