The secondary cooling water system is difficult to control because of loop coupling, thermal inertia, and strict actuator constraints. In addition, when conventional proximal policy optimization (PPO) uses Gaussian action sampling with clipping, the mismatch between sampled and executed actions may degrade learning and control smoothness near actuator limits. To address these issues, this paper develops a Beta-policy and PID-inspired augmented-state proximal policy optimization framework, termed BPAS-PPO, for the secondary cooling water system. The framework augments the state with proportional, integral, and derivative error features, adopts a Beta-distribution policy for bounded continuous-action generation, and uses a piecewise dense reward for the dual-loop tracking task. Simulation studies on an identified linear two-input two-output (TITO) model within the selected operating region show that the complete PID-augmented state yields the most effective training representation among the tested alternatives. Compared with PID, Fuzzy-PID, and Gauss-PPO, BPAS-PPO shows lower overshoot, shorter settling time, better setpoint tracking and disturbance rejection, and smoother control actions near actuator limits. The proposed framework is effective for the studied system within the selected operating region, while its performance beyond this region requires further validation.