Urban air mobility (UAM) is a revolutionary approach to transportation in densely populated cities. UAM involves using small, highly automated aircraft to transport passengers and goods at lower altitudes within urban and suburban areas, aiming to transform how people and parcels move within these environments. On average, UAM can reduce travel times by 30% to 40% for point-to-point journeys, with even greater reductions of 40% to 50% in major cities in the United States and China, compared to land transport. UAM includes advanced airborne transportation options like electric vertical takeoff and landing (eVTOL) aircraft and unmanned aerial vehicles (UAVs or drones). These technologies offer the potential to ease traffic congestion, decrease greenhouse gas emissions, and substantially cut travel times in urban areas. Studying the applications of eVTOLs and UAVs in parcel delivery and passenger transportation poses intricate challenges when examined through the lens of operations research (OR). By OR approaches, we mean mathematical programming, models, and solution methods addressing eVTOL- and UAV-aided parcel/people transportation problems. Despite the academic and practical importance, there is no review paper on eVTOL- and UAV-based optimization problems in the UAM sector. The present paper, applying a systematic literature review, develops a classification scheme for these problems, dividing them into routing and scheduling of eVTOLs and UAVs, infrastructure planning, safety and security, and the trade-off between efficiency and sustainability. The OR methodologies and the characteristics of the solution methods proposed for each problem are discussed. Finally, the study gaps and future research directions are presented alongside the concluding remarks.