Pancreas segmentation in computed tomography (CT) images remains a challenging task due to the organ’s variable shape, size, and low contrast against surrounding tissues. In this study, we propose a two-stage pancreas segmentation framework that combines region localization using Faster R-CNN and pixel-level segmentation using a hybrid TransUNet architecture. To address issues related to class imbalance, we utilize the publicly available NIH pancreas CT dataset and apply downsampling techniques to construct a balanced training set. The segmentation stage incorporates a Transformer encoder into the U-Net framework and employs a novel DHD loss function, which combines the Dice Similarity Coefficient (DSC) and the Hausdorff Distance (HD) to enhance both region-level and boundary-level accuracy. Through the experiments on the open dataset, our proposed model achieved a mean DSC of 88.98%, precision of 91.0%, and recall of 94.4%, the highest among compared approaches on the NIH dataset, outperforming the baselines such as U-Net and standard-loss TransUNet; however, differences in preprocessing protocols should be considered when making direct comparisons with prior work. To assess generalizability beyond the training domain, we further evaluated the NIH-trained model on the BTCV Multi-Organ Segmentation dataset, a completely different institution’s CT dataset, using inference-time adaptation strategies without any fine-tuning, achieving a mean DSC of 62.96% in a zero-shot cross-dataset setting. A fully automatic end-to-end pipeline where a Faster R-CNN detector fine-tuned on BTCV training cases predicts bounding boxes used for cropping was evaluated using 5-fold cross-validation on all 30 BTCV cases and achieved a mean DSC of 66.50% ± 8.55%, with no manual annotation used at any stage.