Email this article 
Open Access
Subscription or Fee Access
A Survey on Mutation Testing Methods
Abstract
Software testing is an important but expensive process because about fifty percent of the total development cost is spent for it. However, this part is the first one to miss by software developers if there is a limited time to complete the project. Tests are commonly generated from program source code, graphical models of software (such as control flow graphs), and specifications / requirements. Creating test cases manually is a huge work for software developers. It is time consuming and error prone. A solution which automatically generates test cases and test data can help the software developers to create test cases from software designs/models in early stage of the software development (before coding). High quality software can not be done without high quality testing. Heuristic techniques can be applied for creating quality test data. Exhaustive testing of a software is not possible so different techniques are invoked. Mutation testing is a powerful testing technique for generating software tests and evaluating the quality of software. Mutation testing measures how “good” our tests are by inserting faults into the program under test. This article presents a survey on various mutation testing tools available. Basic concepts and notations of mutation testing as well as how mutation testing works are described. Problems of automatic generation are identified and explained. Finally important and challenging future research topics are presented.
Keywords
References
J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In ICSE ’05: Proceedings of the 27th International Conference on Software Engineering, pages 402–411, New York, NY, USA, 2005. ACM.
P. J. Walsh. A measure of test case completeness (software, engineering). PhD thesis, State University of New York at Binghamton, Binghamton, NY, USA, 1985.
P. G. Frankl, S. N. Weiss, and C. Hu. All-uses versus mutation testing: An experimental comparison of effectiveness. Journal of Systems and Software, 38:235–253, 1997
B. J. M. Grün, D. Schuler, and A. Zeller. The impact of equivalent mutants. Technical report, Saarland University, 2009. Short paper submitted to Mutation 2009: International Workshop on Mutation Analysis.
D. Baldwin and F. Sayward. Heuristics for determining equivalence of program mutations. Technical Report 276, Yale University, Department of Computer Science, 1979.
J. Offutt and J. Pan. Detecting equivalent mutants and the feasible path problem. In COMPASS ’96: Proceedings 11th Conference on Computer Assurance, pages 224–236, Gathersburg, MD, 1996.
R. Hierons and M. Harman. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability, 9(4):233–262, 1999.
K. Adamopoulos, M. Harman, and R. M. Hierons. How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Genetic and Evolutionary Computation—GECCO 2004, volume 3103 of Lecture Notes in Computer Science, pages1338– 1349, Seattle, Washington, 2004. Springer
R. DeMillo, R. Lipton, and F. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, 1978, pp. 34–41.
B. Baudry et al., “Trustable Components: Yet Another Mutation-Based Approach,” Proc. 1st Symp. Mutation Testing, Kluwer Academic Publishers, 2000, pp. 69–76.
S.-W. Kim, J.A. Clark, and J.A. McDermid, “Investigating the Effectiveness of Object- Oriented Testing Strategies Using the Mutation Method,” Software Testing, Verification and Reliability, vol. 11, no. 4, 2001, pp. 207–225.
A.J. Offutt et al., “An Experimental Evaluation of Data Flow and Mutation Testing,” Software: Practice and Experience, vol. 26, no. 2, 1996, pp. 165–176.{Fault revealing power]
Elaine J. Weyuker. On Testing Non-testable Programs. The Computer Journal, 25(4):465{ 470, November 1982.
Timothy A. Budd and Dana Angluin. Two Notions of Correctness and Their Relation to Testing. Acta Informatica, 18(1):31{45, November 1982.
Jefferson Offutt and William Michael Craft. Using Compiler Optimization Techniques to Detect Equivalent Mutants. Technical Report 92-102, Department of Information and Software Systems Engineering, George Mason University, Fairfax, VA, November 1992
Timothy Alan Budd. Mutation Analysis of Program Test Data. Ph.D. dissertation, Yale University, May 1980.
B. J. Choi and A. P. Mathur. Use of fifth generation computers for high performance reliable software testing. Technical report SERC-TR-72- P, Software Engineering Research Center, Purdue University, West Lafayette IN, April 1990.
E. W. Krauser, A. P. Mathur, and V. Rego. High performance testing on SIMD machines. In Proceedings of the Second Workshop on Software.
Aditya P. Mathur and E. W. Krauser. Modeling mutation on a vector processor. In Proceedings of the 10th International Conference on Software Engineering, pages 154{161, Singapore, April 1988. IEEE Computer Society Press.
Aditya P. Mathur and E. W. Krauser. Mutant unification for improved vectorization. Technical report SERC-TR-14-P, Software Engineering Research Center, Purdue University, West Lafayette IN, April 1988.
M. R. Girgis and M. R. Woodward. An integrated system for program testing using weak mutation and data flow analysis. In Proceedings of the Eighth International Conference on Software Engineering, pages 313{319, London UK, August 1985. IEEE Computer Society.
R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software.
R. A. DeMillo and E. H. Spafford, `The Mothra software testing environment', Proceedings of the 11th NASA Software Engineering Laboratory Workshop, Goddard Space Center, December 1986.
R. A. DeMillo, E.W. Krauser, R. J. Martin, A. J. Offutt and E. H. Spafford, `The Mothra tool set', Proceedings of the Hawaii International Conference on System Sciences, Kailua-Kona, HI, January 1989.
J. Offutt, and E. J. Seaman, `Using symbolic execution to aid automatic test data generation', Proceedings of the IEEE 1990 Annual Conference on Computer Assurance (COMPASS 90), Gaithersburg, MD, June 1990.
W. M. Craft, `Detecting Equivalent Mutants Using Compiler Optimization Techniques', Master's thesis, Department of Computer Science, Clemson University, 1989.
R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt, `An extended overview of the Mothra software testing environment', Proceedings of the IEEE Second Workshop on Software Testing, Verification and Analysis, Banff Alberta, July 1988.
K. N. King and A. J. Offutt. A Fortran language system for mutationbased software testing. Software{Practice and Experience, 21(7):685{718, July 1991.
M. R. Woodward and K. Halewood. From weak to strong, dead or alive? An analysis of some mutation testing issues. In Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, pages 152{158, Ban_ Alberta, July 1988. IEEE Computer Society Press.Engineering, 3(4), July 1977.
L. J. Morell. A Theory of Error-Based Testing. PhD thesis, University of Maryland, College Park MD, 1984. Technical Report TR-1395.
B. Marick. The weak mutation hypothesis. In Proceedings of the Third Symposium on Software Testing, Analysis, and Verification, pages 190{199, Victoria, British Columbia, Canada, October 1991. IEEE Computer Society Press.
B. Marick. Using Weak Mutation with GCT. Testing Foundations, Champaign Illinois, 1993.
D. J. Richardson and M. C. Thompson. The relay model for error detection and its application. In Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, pages 223{230, Ban_ Alberta, July 1988. IEEE Computer Society Press.
J. R. Horgan and A. P. Mathur. Weak mutation is probably strong mutation. Technical report SERC-TR-83-P, Software Engineering Research Center, Purdue University, West Lafayette IN, December 1990.
R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900{910, September 1991.
J. Offutt. Automatic Test Data Generation. PhD thesis, Georgia Institute of Technology, Atlanta GA, 1988. Technical report GIT-ICS 88/28, (Also released as Purdue University Software Engineering Research Center technical report SERC-TR-25-P).
Y.-S. Ma, J. Offutt, and Y.-R. Kwon. MuJava home page. online, 2005. http://ise.gmu.edu/»offutt/mujava/,
http://salmosa.kaist.ac.kr/LAB/MuJava/, last access November 2005.
S. Kim, J. Clark, and J. McDermid. Class mutation: Mutation testing for object-oriented programs. In Net.ObjectDays Conference on Object- Oriented Software Systems, October 2000.
P. Chevalley. Applying mutation analysis for object oriented programs using a reflective approach. In Proceedings of the 8th Asia-Pacific Software Engineering Conference (APSEC 2001), pages 267{270, Macau SAR, China, December 2001. IEEE Computer Society.
P. Chevalley and P. Thevenod-Fosse. A mutation analysis tool for Java programs. Journal on Software Tools for Technology Transfer (STTT), pages 1{14, December 2002.
J. Offutt, R. Alexander, Y. Wu, Q. Xiao, and C. Hutchinson. A fault model for subtype inheritance and polymorphism. In Proceedings of the 12th International Symposium on Software Reliability Engineering, pages 84{93, Hong Kong China, November 2001. IEEE Computer Society Press.
Y. S. Ma, Y. R. Kwon, and J. Offutt. Inter-class mutation operators for Java. In 13th International Symposium on Software Reliability Engineering, pages 352{363, Annapolis MD, November 2002. IEEE Computer Society Press.
Y. S. Ma, A. J. Offutt, and Y. R. Kwon. MuJava: An automated class mutation system. Software Testing, Verification and Reliability, 15(2):97{133, June 2005.
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution 3.0 License.