Deep learning in ultrasound(US) imaging aims to construct foundational models that accurately reflect the modality's unique characteristics. Nevertheless, the limited datasets and narrow task types have restricted this field in recent years. To address these challenges, we introduce US-MTD120K, a multi-task ultrasound dataset with 120,354 real-world two-dimensional images. This dataset covers three standard plane recognition and two diagnostic tasks in ultrasound imaging, providing a rich basis for model training and evaluation. We detail the data collection, distribution, and labelling processes, ensuring a thorough understanding of the dataset's structure. Furthermore, we conduct extensive benchmark tests on 27 state-of-the-art methods from both supervised and self-supervised learning(SSL) perspectives. In the realm of supervised learning, we analyze the sensitivity of two main feature computation methods to ultrasound images at the representational level, highlighting that models which judiciously constrain global feature computation could potentially serve as a viable analytical approach for US image analysis. In the context of self-supervised learning, we delved into the modelling process of self-supervised learning models for medical images and proposed an improvement strategy, named MoCo-US, a solution that addresses the excessive reliance on pretext task design from the input side. It achieves competitive performance with minimal pretext task design and enhances other SSL methods simply. The dataset and the code will be available at https://github.com/JsongZhang/CDOA-for-UMTD .