Promptable medical segmentation models can be sensitive to small targets, weak boundaries, heterogeneous texture, and inaccurate user prompts. We propose HBF-BCER, a prompt-preserving adaptation of the SAM ViT-B architecture for medical image segmentation. Historical Branch Fusion (HBF) reinjects aligned bottleneck summaries from previous enhanced encoder blocks, while Balanced Convolutional Expert Routing (BCER) applies load-balanced, pixel-wise soft weighting to heterogeneous convolutional experts. The evaluation uses fixed held-out evaluation sets, three training seeds, matched original SAM ViT-B, image-encoder and mask-decoder full fine-tuning, decoder-adapter, LoRA, and DoRA baselines, repeated prompt perturbations, paired statistics, and internal module ablations. HBF-BCER achieved Dice scores of $96.69\pm 0.12\%$ and $92.24\pm 0.18\%$ for REFUGE2 disc and cup, $96.18\pm 0.31\%$ for ISIC2016, and $92.08\pm 0.19\%$ for TNMIX. Image-encoder and mask-decoder full fine-tuning produced slightly higher Dice for the REFUGE2 disc and TNMIX, whereas HBF-BCER produced higher Dice for the REFUGE2 cup and ISIC2016 and the lowest HD95 on all four targets. Among the four paired comparisons with available per-image outputs, HBF-BCER showed significant gains for REFUGE2 cup versus SAM + Adapter and LoRA and for TNMIX versus SAM + Adapter; the ISIC2016 difference versus full fine-tuning was not significant. The method reduced trainable parameters by 81.5% relative to image-encoder and mask-decoder full fine-tuning but increased total parameters, FLOPs, and inference latency. These results support HBF-BCER as a parameter-efficient training strategy rather than a generally lightweight or uniformly superior model.