The recent advancements in large language models (LLMs) have brought significant changes to the field of education, particularly in the generation and evaluation of feedback. LLMs are transforming education by streamlining tasks like content creation, feedback generation, and assessment, reducing teachers’ workload and improving online education efficiency. This study aimed to verify the consistency and reliability of LLMs as evaluators by conducting automated evaluations using various LLMs based on five educational evaluation criteria. The analysis revealed that while LLMs were capable of performing consistent evaluations under certain conditions, a lack of consistency was observed both among evaluators and across models for other criteria. Notably, low agreement among human evaluators correlated with reduced reliability in LLM evaluations. Furthermore, variations in evaluation results were influenced by factors such as prompt strategies and model architecture, highlighting the complexity of achieving reliable assessments using LLMs. These findings suggest that while LLMs have the potential to transform educational systems, careful selection and combination of models are essential to improve their consistency and align their performance with human evaluators in educational settings.

Controllable text generation is the primary technique for controlling specific attributes such as topic, keywords and obtaining augmented data. This work proposes a novel controllable text generation framework to improve the controllability of generation models. First, we introduce semantic control grammar, a controllable input format to generate sentences that satisfy the constraints. Second, we adopt a generation and rerank method to obtain semantically reranked controlled sentences. Extensive experiments and analyses are conducted on benchmark, natural language understanding, data-to-text generation, and text classification datasets. Through automatic evaluations, we show that our method leads to improvement over strong baselines. The results show that our model can control sentence and word attributes and semantically generate natural sentences. Furthermore, our techniques improve the generation quality of the model.

Summarization is one of the important tasks of natural language processing used to distill information. Recently, the sequence-to-sequence method was applied, in a general manner, to summarization tasks. The problem is that a large amount of information must be pre-trained for a specific domain, and information other than input statements cannot be utilized. To compensate for this shortcoming, controllable summarization has recently been in the spotlight. We introduced three properties into controllable summarization: 1) a new human-machine communication input format, 2) a robust constraint-sensitive summarization method for these formats, and 3) a practical interactive summarization interface available to the user. Experiments on the Wizard-of-Wikipedia dataset show that applying this input format and the constraint-sensitive method enhances summarization performance compared to the typical method. A user study shows that the interactive summarization interface is practical and that participants are evaluating it positively.

In recent years, methods that selectively fine-tune or reduce the number of layers in large language models (LLMs) have garnered attention as an efficient alternative to traditional fine-tuning, where all layers are trained. In this study, we revisit the classical concept of layer freezing and propose a simple, effective strategy that selectively fine-tunes only a portion of transformer layers. We show that freezing the bottom 25% or 50% of layers in small-scale LLMs with sub-3 billion parameters yields significant improvements in memory efficiency and training speed while maintaining, or even surpassing, the performance of full fine-tuning and Low-Rank Adaptation (LoRA). Through experiments on Natural Language Inference (NLI) tasks using LLMs with fewer than 3 billion parameters, our approach achieves up to 50% memory savings and 30% faster training. Notably, our method does not require architectural modifications or additional parameters, making it particularly suitable for resource-constrained environments.

Hyeonseok Kang, Hyein Seo, Jeesu Jung, Sangkeun Jung, Du-Seong Chang, Riwoo Chung. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.

Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, YoonHyung Roh. Findings of the Association for Computational Linguistics ACL 2024. 2024.