Breast ultrasound image segmentation is the foundation of the diagnosis and treatment of breast cancer. The level set method is widely used for medical image segmentation. However, it remained a challenge for traditional level set methods because they cannot fully understand the tumor regions with complex characteristics by only low-level features. Considering that contextual features can provide complementary discriminative information to low-level features, this paper proposed a contextual level set method for breast tumor segmentation. Firstly, an encoder-decoder architecture network such as UNet is developed to learn high-level contextual features with semantic information. After that, the contextual level set method has been proposed to incorporate the novel contextual energy term. The proposed term has the ability to embed the high-level contextual knowledge into the level set framework. The learned contextual features with semantic information can provide more discriminative information, which has been directly associated with category labels, instead of the original intensity. Therefore, it is robust to serious intensity inhomogeneity, which is helpful to improve segmentation performance. The experiments had taken place with the help of three databases, which indicates that the proposed method outperformed traditional methods.