Advanced deep learning models have shown excellent performance in the task of fault bearing diagnosis over the traditional machine learning and signal processing techniques. Few-shot learning approach has also been attracting a lot of attention in this task to address the problem of limited training data. Nevertheless, cutting-edge models for fault bearing diagnosis are often based on convolutional neural networks that emphasize on local features of input data. Besides, accurate classification of fault bearing signals is still nontrivial due to the variations of data, fault types, acquisition conditions, and extremely limited data, leaving a space for research on this topic. In this study, we propose a novel end-to-end approach for fault bearing diagnosis even in the case of limited data with artificial and real faults. In particular, we propose a module for automatic feature extraction from input data namely Multiscale Large Kernel Feature Extraction. The extracted features are then fed into a two-branch model including a global and a local branch. The global one includes a transformer architecture with cross-attention to handle global context and obtain the correlation between the query and support sets. The local branch is a metric-based model consisting of Mahalanobis distance for separating local features from the support set. The outputs from the two branches are then ensembled for classification purpose. Intensive experiments and ablation studies have been made on the two public datasets including CWRU and PU. Qualitative and quantitative results with different degrees of training samples by the proposed model in comparison with other state-of-the-arts have shown the superior performance of the proposed approach. Our code will be published at: https://github.com/HungVu307/Few-shot-via-ensembling-Transformer-with-Mahalanobis-distance.