The growing demand for sustainable water management solutions has prompted the development of membrane brine concentration (MBC) technologies, particularly in the context of desalination and minimum liquid discharge (MLD) applications. This study presents a simple model of high-pressure nanofiltration (HPNF) for MBC. The model integrates reverse osmosis (RO) transport equations with mass balance equations, thereby enabling acceptable predictions of water flux and total dissolved solids (TDS) concentration. Considering the limitations of the pilot plant data, the model showed reasonable accuracy in predicting flux and TDS, with R² values above 0.99. The simulation results demonstrated that an increase in feed flow rate improves flux but raises specific energy consumption (SEC) and reduces recovery. In contrast, an increase in feed pressure results in an increased recovery and brine concentration. Increasing feed TDS decreases flux, recovery, and final brine TDS and increases SEC. Response surface methodology (RSM) was employed to optimize process performance across multiple criteria, optimizing flux, SEC, recovery, and final brine concentration. The optimal feed flow rate and pressure vary depending on the criteria in the improvement scenarios, underscoring the importance of systematic process improvement.