Urban last-mile logistics increasingly faces small-batch, high-frequency, time-sensitive, and dispersed delivery. Multi-truck–multi-drone cooperation couples routing, drone sorties, battery operations, and synchronization, forming a complex system with multiple trade-offs. This study develops a unified framework to reduce composite operating costs under payload, endurance, synchronization, and soft time-window constraints. We propose a six-component cost model and a Q-learning-assisted hybrid genetic adaptive large neighborhood search algorithm (Q-HGALNS) that integrates delayed-assignment encoding, two-stage unified decoding, and state-dependent selection among 12 destroy–repair actions. Across 12 Solomon instance groups, Q-HGALNS achieved the lowest average cost and significantly outperformed GA, SA, and VNS in all cases, with overall relative cost reductions of 10.876%, 4.483%, and 16.869%. Ablation showed reductions of 0.972%, 1.064%, and 0.678% over NoQ, NoLS, and ALNS-AW. In a 100-point Shenzhen simulation, the lowest-cost feasible solution reduced total cost by 5.566% and the sum of vehicle completion times by 59.2 min versus the truck-only scenario. This study establishes a system-level coordination mechanism linking delayed service-mode decisions, unified decoding, and state-feedback search, enabling routing, sortie, battery, and temporal decisions to be jointly optimized. Q-HGALNS consistently achieves lower costs and supports executable planning and operational strategy design for urban last-mile delivery.