Membrane permeability is central to drug absorption and intracellular target engagement, but the contribution of specific noncovalent interactions to passive permeation remains difficult to quantify. While hydrogen bonds (HBs) are known to strongly influence membrane crossing, the role of halogen bonds (XBs) is still poorly understood, despite the widespread use of halogenation in medicinal chemistry. Here, we systematically investigate how explicit modeling of halogen anisotropy through an extra point (EP), used to emulate the σ-hole, affects the permeability of two activated XB probes and six halogenated drugs. Molecular dynamics and umbrella sampling simulations were combined with the inhomogeneous solubility--diffusion model (ISDM) and a PMF-based permeability ranking. Inclusion of the EP had little effect on chlorinated compounds, a modest effect on bromazepam, and a pronounced effect on amiodarone, indicating that the impact of halogen anisotropy is strongly dependent on the strength of the XB donor and on the molecular environment. ISDM reproduced the experimental Caco-2 permeability trends with high accuracy ( r = 0.99 without EP and r = 0.93 with EP), although amiodarone was underestimated when the EP was included, consistent with an excessive stabilization of iodinated molecules in polar environments when using current force field parameters. Beyond permeability coefficients, we analyzed XB/HB formation probabilities and introduced a spatially resolved decomposition of ligand--membrane interaction energies into electrostatic and van der Waals contributions along the membrane insertion coordinate. This atom/group-resolved analysis shows that XBs can stabilize halogenated compounds at the water--lipid interface, but that excessive electrostatic stabilization may also favor interfacial trapping and reduce effective permeability. Overall, our results show that σ-holes do affect membrane permeability, but mainly for strong XB donors, and that their contribution reflects a delicate balance between membrane recognition, desolvation, and hydrophobic partitioning.