Exploring the limitations of test coverage

Emma is a program coverage tool that can generate HTML based test coverage reports as part of our build system. JUnit test class checks the expected output and actural output of the original program. We can make more stable and solid program using both Emma and JUnit tools. However, is it possible a program has a bug but it still passes JUnit tests and also it has 100% coverage? You can find the answer below.

A BUG that inserted into Stack.java
I have been fixed the original Stack project so that the program passes all the automated QA tools and has 100% of coverage. My fixed Stack project, which has 100% coverage can be downloaded from here. I fixed toString() method that originally convert all elements in the stack into a string. the code I inserted is

return "[Stack [1, 2, 3]]";

This line simply return the string '[stack [1, 2, 3]]' no matter what is inside the stack. My JUnit test class checks toString() method after pushing 1, 2, 3. Therefore, there will be no error in JUnit test and coverage is still 100%.

Writing a new test case to catch the bug
In order to find the bug in the source program, I had to fix the test cases so that JUnit reports the error. The fixed case is

@Test
public void testToString() {
  Stack stack = new Stack();
  stack.push(one);
  stack.push(two);
  stack.push(three);
  stack.push(four);
  assertEquals("Testing stack toString", "[Stack [1, 2, 3, 4]]",stack.toString());
}

This test case failed because the actural output "[Stack [1, 2, 3]]" did not match to expected out put "[Stack [1, 2, 3, 4]". This source can be downloaded from http://www2.hawaii.edu/~jeho/ICS413/stack-JUnit-Lab-jeho-6.0.930.zip

Lessons from Emma and JUnit
Although Emma and JUnit are great tools to find bugs and check expected out put of the source code, there still a posiblities that a code has logical bugs but can pass those tools. Therefore both Automated and Manual QA are still important and need to be checked together.