Existing forest height estimation methods based on polarimetric interferometric synthetic aperture radar (PolInSAR) typically process each pixel independently, potentially introducing inconsistent estimates and additional decorrelation in the covariance matrix estimation. To address these limitations and effectively exploit the spatial context information, this paper proposes the first patch-based inversion method named joint pixel optimization inversion (JPO). By leveraging the smoothness and regularity of homogeneous pixels, a joint-pixel optimization problem is constructed, incorporating a first-order regularization on the ground phase. To solve the non-parallelizable problem of the alternating direction method of multipliers (ADMM), we devise a new parallelizable ADMM algorithm and prove its sublinear convergence. With the contextual information of neighboring pixels, JPO can provide more reliable forest height estimation and reduce the overestimation caused by additional decorrelation. The effectiveness of the proposed method is verified using spaceborne L-band repeat-pass SAOCOM acquisitions and LiDAR heights obtained from ICESat-2. Quantitative evaluations in forest height estimation show that the proposed method achieves a lower mean error (1.23 m) and RMSE (3.67 m) than the existing method (mean error: 3.09 m; RMSE: 4.70 m), demonstrating its improved reliability.