Large Language Models (LLMs) are emerging as powerful tools for automating business process modeling, promising to streamline the translation of textual process descriptions into Business Process Model and Notation (BPMN) diagrams. However, the extent to which these Al systems can produce high-quality BPMN models has not yet been rigorously evaluated. This paper presents an early evaluation of five LLM-powered BPMN generation tools that automatically convert textual process descriptions into BPMN models. To assess the external quality of these Al-generated models, we introduce a novel structured evaluation framework that scores each BPMN diagram across three key process model quality dimensions: clarity, correctness, and completeness, covering both accuracy and diagram understandability. Using this framework, we conducted experiments where each tool was tasked with modeling the same set of textual process scenarios, and the resulting diagrams were systematically scored based on the criteria. This approach provides a consistent and repeatable evaluation procedure and offers a new lens for comparing LLM-based modeling capabilities. Given the focused scope of the study, the results should be interpreted as an exploratory benchmark that surfaces initial observations about tool performance rather than definitive conclusions. Our findings reveal that while current LLM-based tools can produce BPMN diagrams that capture the main elements of a process description, they often exhibit errors such as missing steps, inconsistent logic, or modeling rule violations, highlighting limitations in achieving fully correct and complete models. The clarity and readability of the generated diagrams also vary, indicating that these Al models are still maturing in generating easily interpretable process flows. We conclude that although LLMs show promise in automating BPMN modeling, significant improvements are needed for them to consistently generate both syntactically and semantically valid process models.
Business process redesign (BPR) has emerged as a widely accepted practice within organizations for delivering better products and services and ensuring measurable benefits. Despite the adoption of BPR, it largely remains the result of a creative process and there is a lack of approaches for evaluating the potential impact of a BPR method before investing in its implementation. This means that BPR initiatives are selected based on ‘black-box’ generic options that are not explicitly tailored to the culture, structure, and existing processes of the organization. This paper aims to address this gap by introducing the BPR: Assessment Framework, a conceptual model based on Design Science Research Process (DSRP). The framework is intended to assess the BPR capacity of process models based on their plasticity and external quality. The usability of the framework is presented in the following manner: (a) through the demonstration of the framework using a data-intensive workflow optimization method as an end-to-end paradigm, and (b) the BPR capability assessment of 15 business process models from literature, to better demonstrate the benefits of omitting ineligible models prior to BPR application. The categorization of the redesign capability of models was based on a tested cluster analysis method using the k-means algorithm. Based on the findings, a considerable number of process models proved to be either overly constrained for the application of redesign heuristics or did not possess the required external quality to be redesigned. The contribution of the approach lies in the fact that the framework can also be extended to apply systematic BPR to eligible BPs. The framework can serve as a reliable measurement of the redesign capacity of candidate process models and as an essential part of a systematic methodology for increased BPR effectiveness.
Large Language Models (LLMs) are emerging as powerful tools for automating business process modeling, promising to streamline the translation of textual process descriptions into Business Process Model and Notation (BPMN) diagrams. However, the extent to which these Al systems can produce high-quality BPMN models has not yet been rigorously evaluated. This paper presents an early evaluation of five LLM-powered BPMN generation tools, that automatically convert textual process descriptions into BPMN models. To assess the quality of these Al-generated models, we introduce a novel structured evaluation framework that scores each BPMN diagram across three key process model quality dimensions: clarity/readability, correctness, and completeness, covering both accuracy and diagram understandability. Using this framework, we conducted experiments where each tool was tasked with modeling the same set of textual process scenarios, and the resulting diagrams were systematically scored based on the criteria. This approach provides a consistent and repeatable evaluation procedure and offers a new lens for comparing LLM-based modeling capabilities. Our findings reveal that while current LLM-based tools can produce BPMN diagrams that capture the main elements of a process description, they often exhibit errors such as missing steps, inconsistent logic, or modeling rule violations, highlighting limitations in achieving fully correct and complete models. The clarity and readability of the generated diagrams also vary, indicating that these Al models are still maturing in generating easily interpretable process flows. We conclude that although LLMs show promise in automating BPMN modeling, significant improvements are needed for them to consistently generate both syntactically and semantically valid process models.