The high standard of communication quality in optical access networks makes forward error correction (FEC) schemes, such as LDPC, an integral part of the system. However, pipeline conflict arising from data dependencies is a common issue encountered in the hardware implementation of layered QC-LDPC decoders. This paper proposes an efficient layered decoding architecture to reduce pipeline conflicts without introducing stall cycles. It can solve some of the pipeline conflicts by flexibly reordering the processing order of inter-layer and intra-layer submatrices offline. In addition, the patch method, based on variable-to-check messages, allows for the delayed use of gains between layer iterations, which can further minimize the performance loss caused by the remaining pipeline conflicts. The experimental results on the LDPC code of the IEEE802.16 standard in the OFDM-PON system demonstrate that the proposed architecture has sensitivity improvements of 0.125 dBm and 0.375 dBm, respectively, compared with our previous work and the method described in the other work. The optimized architecture improves the reliability of the decoder and can also make a contribution to efficient PON systems.