The cultivation of potatoes is essential for rural food security, and the use of Unmanned Aerial Vehicle Red-Green-Blue (UAV-RGB) imagery allows for precise and cost-effective estimation of yield and identification of varieties, overcoming the limitations of manual assessment. We evaluated four INIA varieties (Bicentenario, Canchán, Shulay and Tahuaqueña) by integrating agronomic measurements (height, number and weight of tubers, leaf health) with color and textural indices derived from RGB orthomosaics. Yield prediction was modeled using Random Forest (RF) and Gradient Boosting (GB); varietal identification was approached with (i) a Convolutional Neural Network (CNN) that classifies RGB images and (ii) classical models such as Random Forest, Support Vector Machines (SVMs), K-Nearest Neighbors (KNNs), Decision Trees and Logistic Regression trained on EfficientNetB0 embeddings. The results showed significant genotypic differences in yield (p < 0.001): Tahuaqueña 13.86 ± 0.27 t ha⁻¹ and Bicentenario 6.65 ± 0.27 t ha⁻¹. The number of tubers (r = 0.52) and plant height (r = 0.23) correlated with yield; RGB indices showed low correlations (r < 0.3) and high redundancy (r > 0.9). RF achieved a better fit (Coefficient of determination, R² = 0.54; Root Mean Square Error, RMSE = 2.72 t ha⁻¹), excelling in stolon development (R² = 0.66) and losing precision in maturation due to foliar senescence. In classification, the CNN and RF on embeddings achieved F1-macro ≈ 0.69 and 0.66 (Receiver Operating Characteristic—Area Under the Curve, ROC AUC RF = 0.89), with better identification of Bicentenario and Shulay. We conclude that UAV-RGB is a cost-effective alternative for phenotypic monitoring and varietal selection in high Andean contexts. These findings support the integration of UAV-RGB imagery into breeding and monitoring pipelines in resource-limited Andean systems.