Promptable medical segmentation models can be sensitive to small targets, weak boundaries, heterogeneous texture, and inaccurate user prompts. We propose HBF-BCER, a prompt-preserving adaptation of the SAM ViT-B architecture for medical image segmentation. Historical Branch Fusion (HBF) reinjects aligned bottleneck summaries from previous enhanced encoder blocks, while Balanced Convolutional Expert Routing (BCER) applies load-balanced, pixel-wise soft weighting to heterogeneous convolutional experts. The evaluation uses fixed held-out evaluation sets, three training seeds, matched original SAM ViT-B, image-encoder and mask-decoder full fine-tuning, decoder-adapter, LoRA, and DoRA baselines, repeated prompt perturbations, paired statistics, and internal module ablations. HBF-BCER achieved Dice scores of $96.69\pm 0.12\%$ and $92.24\pm 0.18\%$ for REFUGE2 disc and cup, $96.18\pm 0.31\%$ for ISIC2016, and $92.08\pm 0.19\%$ for TNMIX. Image-encoder and mask-decoder full fine-tuning produced slightly higher Dice for the REFUGE2 disc and TNMIX, whereas HBF-BCER produced higher Dice for the REFUGE2 cup and ISIC2016 and the lowest HD95 on all four targets. Among the four paired comparisons with available per-image outputs, HBF-BCER showed significant gains for REFUGE2 cup versus SAM + Adapter and LoRA and for TNMIX versus SAM + Adapter; the ISIC2016 difference versus full fine-tuning was not significant. The method reduced trainable parameters by 81.5% relative to image-encoder and mask-decoder full fine-tuning but increased total parameters, FLOPs, and inference latency. These results support HBF-BCER as a parameter-efficient training strategy rather than a generally lightweight or uniformly superior model.

Accurate medical image segmentation is a core task in biomedical engineering and computer-assisted image analysis. Prompt-based segmentation models, such as Med-SAM, allow anatomical structures or lesions to be delineated from sparse user inputs, but two-dimensional clinical images often contain small targets, weak boundaries, heterogeneous lesion textures, and modality-specific noise. These factors can reduce the reliability of prompt-conditioned masks when early structural cues are weakened during deep encoder propagation. We propose HBF-BCER MedSAM, a prompt-preserving enhancement framework that primarily strengthens MedSAM image-encoder representations while retaining the original point- and box-prompt interface. Historical Branch Fusion (HBF) reinjects cross-layer structural summaries from previous enhanced blocks, whereas Balanced Convolutional Expert Routing (BCER) performs soft pixel-wise weighting of heterogeneous convolutional experts in bottleneck space under a load-balancing constraint. Experiments on REFUGE2, ISIC2016, and TNMIX under matched simulated-prompt protocols show consistent improvements over the matched MedSAM baseline. Dice/IoU increased from 82.1%/73.8% to 91.0%/83.7% for REFUGE2 optic cup segmentation, from 87.5%/78.6% to 95.9%/92.4% on ISIC2016, and from 81.6%/75.1% to 91.4%/85.1% on TNMIX. Qualitative comparisons, convergence analysis, and ablation studies further support the effectiveness of preserving historical structural evidence and applying adaptive local expert weighting for prompt-conditioned medical image segmentation.