Although large language models (LLMs) have the capacity to produce test cases rapidly, the accuracy of the test cases relies heavily on the specification given to the model. In this paper, we present Mirror Effect: an LLM is likely to reflect any level of clarity or ambiguity in natural-language requirements in subsequent test artifacts. For a comparative within-subjects experiment with five expert QA engineers, three commercial LLMs, four core requirements cases, and a five-domain transfer learning test, for each of the four core requirements, vague and measurable cases were formulated without altering the function, and then the two sets of prompts were compared against their human-generated baselines. As a result, four vague requirement prompts produced 127 test cases, 68% of which contained assumptions not present in the requirement statement, while four clear requirements produced 42 test cases with a hallucination rate of 12%. Coverage completeness improved from 42% to 92%, and mean QA review time reduced from 240 min to 85 min. As for the cross-domain experiments, the vague-prompt hallucination rate ranged from 68% to 88%, while for clear-requirement prompts, it ranged from 8% to 22%. Our findings are descriptive in nature due to the small sample size but demonstrate a consistent trend across all models and domains. In our contribution, we provide a definition of test hallucination using traceability, empirical evidence of ambiguity propagation, and a pre-prompt requirement gate.
Nowadays, coding agents are capable of generating code, writing tests to verify that code, and fixing problems on both the code and the tests side. If the same agent writes both the code and its tests, it may alter the test oracle to match what the code currently does, turning a bug into the new expected behavior. This problem, referred to as oracle drift, is of concern both in the area of quality assurance and in terms of security since it compromises the independence of the oracle and can result in integrity violations. We propose three methods for measuring drift, describe seven types of drift mechanisms and four types of adversaries, and draw on evidence from previous research. We also present the Oracle Custody Framework (OCF), which keeps track of where oracles originate, protects independent oracles from changes made by the agent, and forwards risky verdicts to humans for review. We applied our framework’s detectors to 1367 agent patches from SWE-bench Verified and to 500 developer patches. Direct changes to the test suite were rare (0.4%), but 80.2% of the patches to the agent had other files added, and 1.4% had build products added, such as certificate bundles. Having a hygiene rule resulted in an increase in automatic merges from 19.1% to 96.2%, and a decrease in blocking reviews from 2.6% to 0.6%.