Unmanned aerial vehicles (UAVs) and birds are typical low-altitude small targets in optical remote sensing, often occupying only a few pixels and exhibiting highly similar appearances, which limits the effectiveness of appearance-based discrimination at long distances and low resolutions. To overcome this, we propose a non-appearance-based classification framework using kinematic and time–frequency features. At the trajectory level, kinematic features—including the coefficient of variation of velocity and acceleration, the Spatiotemporal Box-counting Fractal Dimension (SBFD), and the Local Higuchi Fractal Dimension (LHFD)—quantify multi-scale trajectory complexity. At the scale-variation level, time–frequency features, specifically the Time-Frequency Aware Singular Value Entropy (TF-SVE) derived from bounding-box area sequences, capture non-stationary oscillations from bird wing flapping, reflecting behavioral differences from rigid UAV motion. Experiments on a complex real-world dataset show that stacking these features achieves 99.47% classification accuracy, demonstrating a robust, resolution-invariant, and practically effective approach for non-appearance-based recognition of low-altitude targets.