Large language models (LLMs) can store factual knowledge within their parameters and have achieved superior results in question-answering tasks. However, challenges persist in providing provenance for their decisions and keeping their knowledge up to date. Some approaches aim to address these challenges by combining external knowledge with parametric memory. In contrast, our proposed QA-RAG solution relies solely on the data stored within an external knowledge base, specifically a dense vector index database. In this paper, we compare RAG configurations using two LLMs—Llama 2b and 13b—systematically examining their performance in three key RAG capabilities: noise robustness, knowledge gap detection, and external truth integration. The evaluation reveals that while our approach achieves an accuracy of 83.3%, showcasing its effectiveness across all baselines, the model still struggles significantly in terms of external truth integration. These findings suggest that considerable work is still required to fully leverage RAG in question-answering tasks.

Large Language Models (LLMs), such as ChatGPT, have transformed the field of natural language processing with their capacity for language comprehension and generation of human-like, fluent responses for many downstream tasks. Despite their impressive capabilities, they often fall short in domain-specific and knowledge-intensive domains due to a lack of access to relevant data. Moreover, most state-of-art LLMs lack transparency as they are often accessible only through APIs. Furthermore, their application in critical real-world scenarios is hindered by their proclivity to produce hallucinated information and inability to leverage external knowledge sources. To address these limitations, we propose an innovative system that enhances LLMs by integrating them with an external knowledge management module. The system allows LLMs to utilize data stored in vector databases, providing them with relevant information for their responses. Additionally, it enables them to retrieve information from the Internet, further broadening their knowledge base. The research approach circumvents the need to retrain LLMs, which can be a resource-intensive process. Instead, it focuses on making more efficient use of existing models. Preliminary results indicate that the system holds promise for improving the performance of LLMs in domain-specific and knowledge-intensive tasks. By equipping LLMs with real-time access to external data, it is possible to harness their language generation capabilities more effectively, without the need to continually strive for larger models.

This paper presents a systematic evaluation of large language models (LLMs) and retrieval-augmented generation (RAG) approaches for question answering (QA) in the low-resource Kazakh language. We assess the performance of existing proprietary (GPT-4o, Gemini 2.5-flash) and open-source Kazakh-oriented models (KazLLM-8B, Sherkala-8B, Irbis-7B) across closed-book and RAG settings. Within a three-stage evaluation framework we benchmark retriever quality, examine LLM abilities such as knowledge-gap detection, external truth integration and context grounding, and measures gains from realistic end-to-end RAG pipelines. Our results show a clear pattern: proprietary models lead in closed-book QA, but RAG narrows the gap substantially. Under the Ideal RAG setting, KazLLM-8B improves from its closed-book baseline of 0.427 to reach answer correctness of 0.867, closely matching GPT-4o’s score of 0.869. In the end-to-end RAG setup, KazLLM-8B paired with Snowflake retriever achieved answer correctness up to 0.754, surpassing GPT-4o’s best score of 0.632. Despite improvements, RAG outcomes show an inconsistency: high retrieval metrics do not guarantee high QA system accuracy. The findings highlight the importance of retrievers and context grounding strategies in enabling open-source Kazakh models to deliver competitive QA performance in a low-resource setting.

The data descriptor introduces an open, high-resolution dataset of real-world bus operations in Astana, Kazakhstan, captured from GPS trajectories between July and September 2024. The data covers three high-frequency routes and have been processed into a GTFS format, enabling direct use with existing transit modeling tools. Unlike typical static GTFS feeds, this dataset provides empirically observed dwell times, run times, and travel times, offering a detailed snapshot of operational variability in urban bus systems. The dataset supports applications in machine learning–based travel time prediction, timetable optimization, and transit reliability analysis, especially in settings where live feeds are unavailable. By releasing this dataset publicly, we aim to promote transparent, data-driven transport research in emerging urban contexts.

To address the need for comprehensive terminology construction in rapidly evolving domains such as blockchain, this study examines how large language models (LLMs), particularly GPT, enhance automatic term extraction through human feedback. The experimental part involves 60 bachelor’s students interacting with GPT in a three-step iterative prompting process: initial prompt formulation, intermediate refinement, and final adjustment. At each step, the students’ prompts are evaluated by a teacher using a structured rubric based on 6C criteria (clarity, complexity, coherence, creativity, consistency, and contextuality), with their summed scores forming an overall grade. The analysis indicates that (1) students’ overall grades correlate with GPT’s performance across all steps, reaching the highest correlation (0.87) at Step 3; (2) the importance of rubric criteria varies across steps, e.g., clarity and creativity are the most crucial initially, while complexity, coherence and consistency influence subsequent refinements, with contextuality having no effect at all steps; and (3) the linguistic accuracy of prompt formulations significantly outweighs domain-specific factual content in influencing GPT’s performance. These findings suggest GPT has a robust foundational understanding of blockchain terminology, making clear, consistent, and linguistically structured prompts more effective than contextual domain-specific explanations for automatic term extraction.

This paper presents the MLOps strategy, which adapts the automation principles of DevOps to the deployment and lifecycle management of artificial intelligence (AI) models. By leveraging high-performance automation, MLOps ensures seamless AI development and operations integration, enabling efficient and reliable model deployment. The study demonstrates this approach by implementing the Astana Opinion Mining macro-service customized for sentiment analysis. This macro-service evaluates public opinions based on a criteria taxonomy for assessing the urban environment's sustainable development. As a smart city application, the system facilitates the collection and analysis of citizen feedback to assess the performance of city services and inform urban planning decisions. Technologically, the MLOps strategy employs containers and microservices to construct robust data and process pipelines. Four core pipelines were developed in this research: data collection, feature engineering, experimentation, deployment, and maintenance. The data collection pipeline is achieved through automated crawling from diverse sources such as social media and other internet platforms. The feature engineering pipeline ensures data preprocessing by removing noise, identifying message languages, categorizing topics, and preparing data for further analysis. The experimentation pipeline incorporates services for data labeling, model training, and performance evaluation customized to sentiment analysis tasks. Finally, the deployment pipeline and maintenance pipeline deliver trained models to end-users, ensuring their continual improvement and adaptation. Using this MLOps framework, four models of sentiment analysis were tested in Russian: "Blanchefort," "Sismetanin," "MonoHime," and "Dostoevsky." The "Blanchefort" showed an accuracy of 71,43%. The resulting MLOps framework is fault-tolerant, scalable, and enables real-time urban environment assessments. By automating workflows, the architecture enhances operational efficiency, offering practical applications for smart city initiatives and sustainable urban development, contributing to better decision-making.