Model Predictive Control (MPC) performance depends on prediction and control horizons, particularly for experimentally identified inverse-response processes with noise, actuator constraints, and plant–model mismatch. This study evaluates an offline reinforcement learning (RL)-assisted supervisory framework for horizon selection in constrained MPC of a membrane-filtration process. Pump-speed-command and pump-line-flow data identified a second-order state-space model using Numerical Algorithms for Subspace State-Space System Identification (N4SID), achieving 92.16% estimation fit and 90.90% independent one-step-ahead validation fit, although long-horizon fidelity remained limited. The model was embedded in target-tracking MPC with input and input-move constraints, while RL selected only the horizons. Nominally, the representative RL-selected pair reduced the prespecified composite score from 2.5700 to 2.4562 (4.43%) with zero simulated quadratic-programming failures. Exhaustive evaluation of 146 feasible pairs identified 1.2214, confirming that RL was not globally optimal. Under a 120-iteration budget, Bayesian optimisation reached the global optimum in 20/30 runs versus 3/30 for epsilon-greedy. Extending the domain to 208 pairs reduced the score to 1.1838, indicating sensitivity to the lower prediction-horizon bound. Across 11 model-based robustness scenarios, RL-tuned MPC achieved an average 3.13% score reduction versus baseline, with individual-metric trade-offs. Overall, horizon selection improved the MPC trade-off, but results remained dependent on the search domain and identified-model environment.