Real-time tracking and segmentation of ultrasound video sequence are prerequisite for identifying and analyzing lesions. While significant progress has been made in natural video object segmentation, developing a model for ultrasound video that can maintain both a certain speed and a high level of accuracy is still challenging. This paper develops an end-to-end network for segmentation of ultrasound video, UVOSNet. The model adopts an encoder-attention-decoder architecture, where the encoder utilizes resnet-50 to extract features from ultrasound video frames, the attention module uses LSTT to enhances the extracted features with historical information, and finally, the decoder employs the FPN network to reconstruct the enhanced features into a segmentation mask. To improve the speed of the model, the model is analyzed, and we found optimizing attention is the key to improving the inference speed of the model. So, we replaced the classical attention mechanism in the LSTT module with flow-attention to obtain UVOSNet-flow We conducted experiments on real lymphoma ultrasound video dataset, which demonstrates that UVOSNet and UVOSNet-flow achieve the best performance with respect to J&F measure, Param and FPS measures when compared to the four competing baselines.