Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over $1.5\times {10}^8$ forward passes on ImageNet validation images. PatchMap reveals systematic “hot-spots” where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation-guided placement heuristic that leverages off-the-shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet-50-our method boosts attack success rates by 8–13 percentage points compared to random or fixed placements.