Medicinal plants have always been studied and considered due to their high importance for preserving human health. However, identifying medicinal plants is very time-consuming, tedious and requires an experienced specialist. Hence, a vision-based system can support researchers and ordinary people in recognising herb plants quickly and accurately. Thus, this study proposes an intelligent vision-based system to identify herb plants by developing an automatic Convolutional Neural Network (CNN). The proposed Deep Learning (DL) model consists of a CNN block for feature extraction and a classifier block for classifying the extracted features. The classifier block includes a Global Average Pooling (GAP) layer, a dense layer, a dropout layer, and a softmax layer. The solution has been tested on 3 levels of definitions (64 × 64, 128 × 128 and 256 × 256 pixel) of images for leaf recognition of five different medicinal plants. As a result, the vision-based system achieved more than 99.3% accuracy for all the image definitions. Hence, the proposed method effectively identifies medicinal plants in real-time and is capable of replacing traditional methods.