18 December 2024 in IEEE Transactions on Audio, Speech and Language Processing
One promising aspect of using large language models (LLMs) for translation is their ability to adapt effectively to unseen domains without fine-tuning, through strategic prompting. However, the success of this approach largely depends on the selection of relevant examples from domain-specific databases to construct the few-shot prompt for the given input. This work propose a novel zero-shot technique that circumvents data limitations by generating synthetic translation examples from monolingual domain-specific target texts. Our preliminary experiments suggest that LLMs demonstrate increased sensitivity to noise on the target side of examples, while maintaining a tolerance for variations in example quality. Based on these findings, we generate pseudo-examples by using a cross-lingual sentence encoder to identify sentences in the target language that correspond to the input, and then pairing them with coarse translations derived from a dictionary. To improve the retrieval of better pseudo-examples for translation tasks, we enhance the sentence encoder by incorporating domain information through conditional layer normalization and introducing two innovative training objectives: LLM Preference Prediction and Windowed Sentence Prediction. Experiments conducted across various domains demonstrate that our approach not only significantly outperforms previous zero-shot methods but is also comparable to the robust NLLB-3.3B and ALMA-13B(-R) model.