The precise identification of damp, sticky coal gangue; efficient jet nozzle separation; and process layout in a narrow, restricted space are essential technologies for gangue source reduction based on underground gangue photoelectric separation, which is critical for the long-term growth of coal mines. In this paper, the X-ray absorption fine structure (XAFS) method was used to identify the X-ray absorption law of different atoms in coal-based minerals and explore the differences in the microscopic crystal properties of coal gangue; the numerical simulation calculation of four commonly used nozzles—namely, flat, convergent, flat–convergent, and streamline—was carried out using Fluent software for coal gangue jet separation to optimize the nozzle morphology and parameters. The technical characteristics of the underground layout of the photoelectric separation system for coal gangue were expounded, and the technological layout of the separation system was explored. The results showed that the absorption coefficients μ(E) of Al and Si atoms in minerals to X-rays are significantly different, and the XAFS method has the ability to identify coal, gangue, and other minerals. The streamlined nozzle has a long jet core area, slow decay of jet velocity, low gas consumption per unit time, and better performance than the other three types of nozzles. Based on the development and mining system of the Renjiazhuang Coal Mine, three kinds of photoelectric separation system layout schemes of coal gangue were designed, namely centralized layout, distributed layout, and mobile layout. The advantages and disadvantages of each scheme were compared, which enriched the technical means of gangue source reduction.