The heat transfer and flow characteristics of TiO₂/R123 nano-organic working fluid are investigated and compared with that of R123. A three-dimensional numerical model of the smooth circular tube with a diameter of 10 mm and a length of 1 m is established, and the thermodynamic properties of the nano-organic working fluids are rectified with the volume of fluid model. The grid independence validation is conducted, and the simulation results from three models (the k-ε model, the realizable k-ε model, and the Reynolds Stress Model) are evaluated against experimental data. When using the TiO₂/R123 nano-organic working fluid, the error between the simulation and experimental results is 6.1%. The flow field distribution is examined, and the effect of mass flux on heat transfer coefficient and pressure drop is discussed. Results demonstrated that the inclusion of TiO₂ nanoparticles significantly enhances heat transfer performance. At a 0.1 wt% nanoparticle concentration, the heat transfer coefficient increases by 23.2%, reaching a range of 1430.11 to 2647.25 W/(m²·K), compared to pure R123. However, this improvement in heat transfer performance is accompanied by an increase in flow resistance, with the flow resistance coefficient rising from 0.0353 to 0.0571. Additionally, pressure drops increase by up to 18.7%.