Weather extremes put smallholder farmers’ livelihoods and income at risk. In this context, insurance solutions provide compensation when weather-induced losses occur. However, particularly in a smallholder context, in-field loss assessments can be costly and drive insurance premiums to unaffordable levels. Here, we developed a computer vision-based crop loss assessment that uses smartphone pictures of insured plots taken at multiple crop growth stages. Based on >20,000 pictures assessed by agricultural experts from 3,626 farmer-managed fields between April 2020 and September 2021, we trained five Convolutional Neural Networks to predict three loss categories, “Minor” [0%, 33%) damage, “Moderate” [33%, 66%), and “Large” [66%, 100%]. We obtained high prediction performance for minor and large loss categories, and modest performance for moderate losses, correctly predicting 86%, 75%, and 40% of actual losses, respectively. This result is robust across all convolutional neural models. To enhance transparency, we used a class activation mapping algorithm, which confirms, for each picture, the areas that contribute most to the loss prediction likelihood, providing a nuanced understanding of each prediction. We conclude that computer vision-based crop loss assessment can substantially simplify indemnity-based loss assessment, potentially unlocking affordable, accurate and comprehensible crop insurance for smallholder farmers.