License plate recognition system (LPR) plays a vital role in intelligent transport systems to build up smart environments. Numerous country specific methods have been proposed successfully for an LPR system, but there is a need to find a generalized solution that is independent of license plate layout. The proposed architecture is comprised of two important LPR stages: (i) License plate character segmentation (LPCS) and (ii) License plate character recognition (LPCR). A foreground polarity detection model is proposed by using a Red-Green-Blue (RGB) channel-based color map in order to segment and recognize the LP characters effectively at both LPCS and LPCR stages respectively. Further, a multi-channel CNN framework with layer aggregation module is proposed to extract deep features, and support vector machine is used to produce target labels. Multi-channel processing with merged features from different-level convolutional layers makes output feature map more expressive. Experimental results show that the proposed method is capable of achieving high recognition rate for multinational vehicles license plates under various illumination conditions.