This paper describes a vision-based sense-and-avoid framework to detect approaching aircraft especially observed with cluttered background. The proposed framework consists of a vision system with a camera that processes the incoming images using a series of algorithms in real time to isolate moving aerial objects on the image plane and classify them using a particle filter. Once an approaching aerial object has been detected on a potential collision course, the aircraft performs an evasive maneuver. The performance of the proposed sense-and-avoid algorithm is validated in a series of test flights using two unmanned aerial vehicles.