Addressing the challenge of radar detection of small targets under sea clutter, target detection methods based on a three-dimensional feature space have shown effectiveness. However, their application has revealed several problems, including high dependency on linear relationships between features for dimensionality reduction, unclear reduction objectives, and spatial divergence of target samples, which limit detection performance. To mitigate these challenges, we constructed a feature density distance metric employing copula functions to quantitatively describe the classification capability of multidimensional features to distinguish targets from sea clutter. On the basis of this, a lightweight nonlinear dimensionality reduction network utilizing a self-attention mechanism was developed, optimally re-expressing multidimensional features into a three-dimensional feature space. Additionally, a concave hull classifier using feature sample distance was proposed to mitigate the negative impact of target sample divergence in the feature space. Furthermore, multivariate autoregressive prediction was used to optimize features, reducing erroneous decisions caused by anomalous feature samples. Experimental results using the measured data from the SDRDSP public dataset demonstrated that the proposed detection method achieved a detection probability more than 4% higher than comparative methods under Sea State 5, was less affected by false alarm rates, and exhibited superior detection performance under different false alarm probabilities from 10⁻³ to 10⁻¹.