Early detection of abnormalities in chest X-rays is essential for COVID-19 diagnosis and analysis. It can be effective for controlling pandemic spread by contact tracing, as well as for effective treatment of COVID-19 infection. In the proposed work, we presented a deep hybrid learning-based framework for the detection of COVID-19 using chest X-ray images. We developed a novel computationally light and optimized deep Convolutional Neural Networks (CNNs) based framework for chest X-ray analysis. We proposed a new COV-Net to learn COVID-specific patterns from chest X-rays and employed several machine learning classifiers to enhance the discrimination power of the presented framework. Systematic exploitation of max-pooling operations facilitates the proposed COV-Net in learning the boundaries of infected patterns in chest X-rays and helps for multi-class classification of two diverse infection types along with normal images. The proposed framework has been evaluated on a publicly available benchmark dataset containing X-ray images of coronavirus-infected, pneumonia-infected, and normal patients. The empirical performance of the proposed method with developed COV-Net and support vector machine is compared with the state-of-the-art deep models which show that the proposed deep hybrid learning-based method achieves 96.69% recall, 96.72% precision, 96.73% accuracy, and 96.71% F-score. For multi-class classification and binary classification of COVID-19 and pneumonia, the proposed model achieved 99.21% recall, 99.22% precision, 99.21% F-score, and 99.23% accuracy.