Herb-target interactions (HTIs) are pivotal for unveiling the underlying pharmacological mechanisms between herbs and biological targets (e.g., proteins and nucleic acids). Unlike typical drugs, each of which is made of one pharmacological compound, the herb is composed of different pharmacological ingredients, while contemporary computational approaches mainly focus on HTIs, neglecting the more refined and informative ingredient-target interactions (ITIs). Furthermore, those methods also disregard the complex associations between herbs and ingredients, which have multi-granular interactions with targets. We propose a multi-instance learning based solution (HTI-MIL) to identify HTIs and the finer ITIs. Particularly, HTI-MIL employs Autoencoder, Convolutional Neural Networks, and Graph Convolution Networks to learn representations of herbs, targets, and ingredients, respectively. Next, it takes herbs as bags and ingredients as instances to model the interplay between herbs and ingredients by multi-instance learning, and aggregates ingredient features toward their hosting herb. After that, it leverages the representation of ingredients and targets to predict ITIs, and fuses the representations of herbs, ingredients and targets to predict HTIs. Experimental results show that HTIMIL outperforms competitive methods by at least 5.2%, 4.8%, and 6.4% in AUROC, AUPRC, and F1-score, respectively. HTIMIL validates on typical herbs and confirms ≥90% of the top 20 candidate targets, and also finds out novel interactions. In addition, the targets identified by HTI-MIL help to dissect the mechanism of herb-induced liver injury, which further validates its effectiveness.