Speech recognition is a technique that recognizes words and sentences in audio form and converts them into text sentences. Currently, with the advancement of deep learning technologies, speech recognition has achieved very satisfactory results close to human abilities. However, there are still limitations in identification results such as lack of punctuation, capitalization, and standardized numerical data. Vietnamese also contains local words, homonyms, etc, which make it difficult to read and understand the identification results for users as well as to perform the next tasks in Natural Language Processing (NLP). In this paper, we propose to combine the transformer decoder with conditional random field (CRF) to restore punctuation and capitalization for the Vietnamese automatic speech recognition (ASR) output. By chunking input sentences and merging output sequences, it is possible to handle longer strings with greater accuracy. Experiments show that the method proposed in the Vietnamese post-speech recognition dataset delivers the best results.

Speech recognition is the process of converting the speech signal of a particular language into a sequence of corresponding content words in text format. The output text of automatic speech recognition (ASR) systems often lacks struc- ture, such as punctuation, capitalization of the first letter of a sentence, proper nouns, and names of locations. This absence of structure complicates compre- hension and restricts the utility of ASR-generated text in various applications, such as creating movie subtitles, generating transcripts for online meetings, and extracting customer information. Therefore, developing standardization solu- tions for the output text from ASR is necessary to improve the overall quality of ASR systems. In this article, we use the idea of multitask deep learning for the task of capitalization and punctuation recognition (CPR) for the output text of Vietnamese ASR, with the aim of the named entity recognition (NER) task as a supplement to help the CPR model perform better, and proposed to use text-to- speech (TTS) to create a dataset for CPR-NER multitask model training. The experiment results show that the multi-task deep learning model has improved CPR results by 6.2% of F1 score with ASR output and 7.1% on raw text.