Real objects-based full-color holographic display systems usually collect data with a depth camera and then modulate the input light source to reconstruct the color three-dimensional scene of the real object. However, at present, the main problems of the real-time high quality full-color 3D display are slow speed, low reconstruction quality, and high consumption of hardware resources caused by excessive computing. Based on the hybrid Taylor Rayleigh–Sommerfeld diffraction algorithm and previous studies on full-color holographic systems, our paper proposes Taylor Rayleigh–Sommerfeld diffraction point cloud grid algorithm (TR-PCG), which is to perform Taylor expansion on the radial value of Rayleigh–Sommerfeld diffraction in the hologram generation stage and modify the data type to effectively accelerate the calculation speed and ensure the reconstruction quality. Compared with the wave-front recording plane, traditional point cloud gridding (PCG), C-PCG, and Rayleigh–Sommerfeld PCG without Taylor expansion, the computational complexity is significantly reduced. We demonstrate the feasibility of the proposed method through experiments.