We studied the performance of fine-tuning open-source LLMs for a range of different chemical questions. We benchmark their performances against “traditional” machine learning models and find that, in most cases, the fine-tuning approach is superior.