Recent years have ushered in a transformative era in in vitro modeling with the advent of organoids, three-dimensional structures derived from stem cells or patient tumor cells. Still, fully harnessing the potential of organoids requires advanced imaging technologies and analytical tools to quantitatively monitor organoid growth. Optical coherence tomography (OCT) is a promising imaging modality for organoid analysis due to its high-resolution, label-free, non-destructive, and real-time 3D imaging capabilities, but accurately identifying and quantifying organoids in OCT images remain challenging due to various factors. Here, we propose an automatic deep learning-based pipeline with convolutional neural networks that synergistically includes optimized preprocessing steps, the implementation of a state-of-the-art deep learning model, and ad-hoc postprocessing methods, showcasing good generalizability and tracking capabilities over an extended period of 13 days. The proposed tracking algorithm thoroughly documents organoid evolution, utilizing reference volumes, a dual branch analysis, key attribute evaluation, and probability scoring for match identification. The proposed comprehensive approach enables the accurate tracking of organoid growth and morphological changes over time, advancing organoid analysis and serving as a solid foundation for future studies for drug screening and tumor drug sensitivity detection based on organoids.