First, Debug the Test Oracle
This research was sponsored in part by NSFC Program (Grant No. 91218302), National Key Technologies R&D Program (Grant No. 2012BAF16G01) of China, Postdoctoral Science Foundation of China (Grant No. 2014M560082), National Natural Science Foundation of China (Grant No. 61402248).
IEEE Transactions on Software Engineering
Debugging in computer science, Fault location (Engineering), Algorithms
Opposing to the oracle assumption, a trustworthy test oracle is not always available in real practice. Since manually written oracles and human judgements are still widely used, testers and programmers are in fact facing a high risk of erroneous test oracles. However, test oracle errors can bring much confusion thus causing extra time consumption in the debugging process. As substantiated by our experiment on the Siemens Test Suite, automatic fault localization algorithms suffer severely from erroneous test oracles, which impede them from reducing debugging time to the full extent. This paper proposes a simple but effective approach to debug the test oracle. Based on the observation that test cases covering similar lines of code usually generate similar results, we are able to identify suspicious test cases that are differently judged by the test oracle from their neighbors. To validate the effectiveness of our approach, experiments are conducted on both the Siemens Test Suite and grep. The results show that averagely over 75 percent of the highlighted test cases are actually test oracle errors. Moreover, performance of fault localization algorithms recovered remarkably with the debugged oracles
Locate the Document
PSU affiliates use Find in PSU library link at top.
Unaffiliated researchers can access the work here: http://dx.doi.org/10.1109/TSE.2015.2425392
Guo, X., Zhou, M., Song, X., Gu, M., & Sun, J. (2015). First, Debug the Test Oracle. Software Engineering, IEEE Transactions on, 41(10), 986-1000.