Panic phenomenon is one of the main challenges in the current pandemic time. In this work, we aim to explore the approaches to detect the panic-related COVID-19 tweets. Aligned to this, we propose an unsupervised clustering approach considering negation cues as an extracted feature input to the pre-trained model. This task cannot be done by simply applying state-of-the-art transformer models, since we observed that they occasionally fail in handling negations. Hence, we propose to utilize features based on Contextual Valence Shifters (CVS) along with the pre-trained BERT embeddings. We evaluate and compare the approaches in an unsupervised setup, using standard clustering metrics on a large set of COVID-19 tweets. The obtained results show that CVS effectively facilitates negation handling (positive/negative tweet discrimination).