About Code Coverage Testing
Abstract. Software Testing has evolved and matured over the past few years on all the fronts, be it process or technical. When it comes to Code Coverage Testing, there always remain debates with novel & professionals alike each claiming its own definition of Code Coverage Testing. The biggest universally accepted myth is that 100% code coverage guarantees exceptional quality. There are books and material available on the subject which are not wrong, but lack the clarity which is needed for actual implementation. The current paper is an attempt to tailor the entire subject in one crisp document and so goes the title, “Nuts and Bolts of Code Coverage Testing”. The paper starts with defining Code Coverage along with clearing out myths on Code Coverage. It demonstrates a real-time example of Code Coverage implementation and how/where other testing activities are involved and their relation with Code Coverage. It presents a methodical way to implement Code Coverage Testing (highlighting the best practices along with pit falls to check out of). It also presents a real time case study of one of the heaviest products in the Industry, where it reached to a point where testing became almost untrack able and how Code Coverage Implementation helped to bring the Testing activity back on track.
What is Code Coverage Testing?
Code Coverage Testing is one of the measurement criterion used to identify the Test Coverage of Application Under Test. This is typically used to publish the numbers to the management/stakeholders so that they can make more informed decisions before releasing a product.
Code Coverage Testing is performed to know the amount of Application Code being exercised during testing (regardless of Black Box, White Box, or Grey Box) Testing.
This term is often confused with Statistical Code Analysis which is a completely different activity done typically by developers, dev. leads and some of the most commonly techniques include code inspections, walkthroughs etc.
Why Code Coverage Testing?
Over the past few years, the software development tools have evolved in terms of maturity levels (simplicity from the development perspective but at the cost of increasing internal complexities). Most of the development Tools are wizard driven & feature rich but they come with a lot of overhead. The codebase of the application not only includes its own codebase, but also of OS, support/3rd party libraries. It makes Testing a daunting task as the size of codebases associated with the product remains complex and bulky. Development Team does have Unit Test cases where the scope is limited. The final gate always remains to be the Testing Team and no matter how hard the Team tries, there would always remain areas where the code will be running multiple times and on the other side there will be areas where the application code will never be tested.
Historical data proves that it’s impossible to release bug-free products which are destined to come when the Product goes live. Manual Test Team cannot be blamed as their job is to execute the tests based on the Test Cases generated from Functional Requirements documentation. At the same time, Automated Regression Tests can’t be blamed as their scope is limited to ensure nothing is broken where inputs are again in terms of black and white(0’s and 1’s). And similar is the case with system, integration, grey box, and performance testing.
Code Coverage: is the Test execution performed on the Product with the internal binaries hooked to a Coverage Tool so that an accurate analysis can be drawn as to how much code was actually exercised during test execution. It draws a measurement criterion of the untested code of the application.
What Code Coverage Testing is not?
It is never intended to replace:
- Manual Testing
- Automated Testing
- Grey Box Testing
- Unit Testing
- Static Analysis – which includes code walkthroughs, reviews etc.
Implementing Code Coverage Testing:
Following are the high level objectives that should be thoroughly studied and put together in order to implement effective Code Coverage Testing.
Draw the right expectations – It’s always important to clearly set expectations so that there is no scope of ambiguities in future. Following are the key expectations to be considered —
- Objective Indicator of Test Coverage of application code
- Links to uncovered Packages / Classes / Methods / Branches
- Links to uncovered Folders / Files / Lines
- Drill down support right from the namespace to individual line of code
- Early detections of uncovered Test Coverage to avoid later investments (which are higher multiple times as the Project Progresses)
- Remove redundancy(same code being exercised in multiple tests) in testing
- Increased Confidence for Releases
- Selecting the Tool: What to look for while selecting a Code Coverage Tool – there are many factors that need to be considered when it comes to selecting the Code Coverage Tool. The most important key factors are mentioned below:
Table 1 – Code Coverage Testing – Tools selection matrix
|SNO Deciding Factor for Tool Selection||Code Coverage tool|
|Uncovering Untested Areas||Lines/nodes that remain untested|
|Execution modes||For example, support for debug/run mode.|
|Identifying Repetitive Tested Areas|
|Quality of Test Reporting Metrics||Should be intuitive/meaningful|
|Capacity to drill down from the top most module to pin pointing the code/function|
|Accuracy of results during evaluation of the Tool before making a recommendation|
|User Friendliness||Should be intuitive/meaningful|
|Logging levels||For example, basic, verbose etc.|
|Merging of Results||Merging can be done manually as well as automated. Popular Tools support both the modes of merging|
Code Coverage Testing Strategy (Short Term and Long Term)
Execution and Reporting
This is one of the most critical part which is often neglected by the Engineering teams. The product owners/stake-holders need this information. The information not only needs to be accurate, but at the same time crisp (with meaningful data) and presentable.
Code Coverage Testing – A Real World Implementation
For the Purpose of demonstration, let’s do a real-time implementation and see how Code Coverage Testing is different from other types of Testing.
Application Under Test: Consider a web or UI form (as displayed in Figure 1) with only 3 controls, User ID, Password, and Login with an option to minimize, maximize and close.
Black Box Testing: Enter a legitimate User ID and the Password and click the submit button. Expected outcome would be getting to the home page.
Grey Box Testing: Would include going to the database and validating that the User ID and Password are authentic. Expected behavior would be accurate credentials.
Load/Performance Testing: Would include multiple users logging in to the form (with some ramp up patterns) using their respective credentials, followed by constant load and finally ramp down. Expected outcome would be conformance to SLA’s (be it response time, CPU/memory utilizations etc.).
Automated Testing: A script would add user id from an external file and then hit the submit button in un-attended mode. Expectation would be the execution report marking the result as pass.
Code Coverage Testing: Code Coverage can be generated for each of the Tests mentioned above. The expectation would be to publish the results and increase tests to increase coverage. A typical Code Coverage Execution Report looks like shown in Figure 2 and 3 below.
Interpreting Code Coverage Testing Reports
Code Coverage Testing has a typical jargon associated with it. It is not as hard as it sounds at the first place. For example, terms like Branch Coverage, Sequence Point Coverage and so on. The key terms are explained below
Line Coverage– Line coverage captures the number of lines that were executed during testing activity. Later, the numbers are compared with the total number of executable lines (codebase of the Application under Test). There is a downside of Line Coverage because it measures the line formats rather than the code. For example, it is possible to format a program in a single line and achieve 100% line coverage with only one test.
Statement/Branch Coverage — Statement and branch testing are relatively stronger than Line Coverage, but have the weakness that interactions between decision outcomes can mask errors during testing.
Decision Coverage– By requiring decision outcomes to be performed independently during testing, basis path testing can expose additional errors.
For example, please refer to the following code-
a = a + 1;
a = a -1;
If the expectation from the function is to have the value of variable “a” unchanged under all circumstances, it makes testing a challenging task. Now let’s consider all the Tests for testing this piece of code.
Branch Testing — The branch testing can be accomplished by executing two tests that do not detect error. The tests would be —
The first test to make both decision outcomes to be false, in which case the value of “a” will not get affected.
The second test to make both decision outcomes to be true.
Statement Testing — The statement testing can be accomplished only by the later test
Basis Path Testing – From the above example, it can be concluded that neither statement nor branch testing is sufficient to detect the error. In this particular example, Basis Path Test will successfully detect the error.
Code Coverage Testing is measured as percentage of application code executed during the testing activities. Code Coverage can be measured at various levels – in terms of programming language like – Namespaces, Classes, Methods, and Branches or in terms of physical parameters like – Folders, Files and Lines. And the idea is simple, generate coverage add tests to increase coverage till you reach the targeted coverage.
Note: 100% Code Coverage can never claim that the product is bug free. It can help ensuring that the code is 100% tested.
A Case Study of Code Coverage Implementation for Testing an Application
Summary/Challenge: The codebase of the Application was increasing from one version to the next. This was primarily because it had to support backward compatibility (for legacy versions) as well as add new features, do patch/SP releases. The increasing complexities and the bulkiness of product made testing a challenge as well as an expensive task. It needed around 8-9 months of effort to just to test the regression suites manually (with a team size of 70 engineers). In order to control the situation, the test suites were automated via as is basis, and the execution time was reduced to 7 days for unattended test execution. The team could spend qualitative time in enhancing the existing automation suites and testing for new feature/patch/bug fix releases. Later it was discovered that there was no actual criterion to understand as to how much application code was actually tested. There were high possibilities of –
1. Cases where code was hitting the same sequence hundreds of times (maybe more)
2. Cases where the code was never reaching out during Testing
A couple of popular Code Coverage Testing Tools were evaluated and the Application was tested after configuring the Code Coverage. The Test Results of various Tools were compared. The results of the Code Coverage were alarming but accurate. The best fit Code Coverage Tool was recommended, a Code Coverage strategy was put together along with highlighting the current as well as future targets.
The table mentioned below was put together and shared in common dashboard. The high level objectives included –
Highlighting and publishing the current test coverage metrics of the Product
Use the number as a benchmark to set the targets for subsequent releases and
To bring back the clarity in Testing with a continuous focus on improvement
Highlight/bring in clarity in the Test Organization as well as Product Management team
The table below uses 10 %( a random number) as targeted increased coverage for subsequent releases
Table 2: Code Coverage Testing – Dashboard Metrics
|Release Categorization||Functional/Manual Testing||Automated Test Execution||Performance and Load Testing||Total % of Code Coverage Achieved|
|Release X(baseline Coverage)||A% — Benchmark||B% — Benchmark||C% — Benchmark|
|Release YTargeted CoverageActual Coverage||Targeted Coverage – A %+10%||Targeted Coverage – B %+10%||Targets arenormally not forthe increasedcoverage but forapplication stability|
|Release ZTargeted CoverageActual Coverage|
Recommended Best Practices for Code Coverage Testing
1. Ensure covering all application paths/workflows in terms of decision trees in the code
2. Ensure covering all data values – this can be done by patterns with different sets of data and can avoid extra fat/tests.
3. Code Coverage Testing Tool should determine where the controls/sequences are being tested/not being tested, which can save a lot of fat/extra tests being tested by automated suites. It can cover the classes, methods, branches but not the business logic.
4. Give graphical representation of the results in terms of real time charts & metrics.
5. Education – Ensure that the stake-holders are aware of the shortcomings and the benefits of implementing Code Coverage so that the expectations are clearly set and visible.
6. Testing is Proportional to Complexity. It is a known fact that 80% of the bugs are always found in 20% of the code. Identification, Planning, and execution of Effective strategy can make a huge difference and bring in value to the Organization.
Analysis on Code Coverage Testing:
Some Important Points to recap –
- Code Coverage Testing can/should never replace any other form of Testing
- Testing activity is always proportional to complexity of the application
- Testing efforts should be focused on the error-prone software and/or error-prone components.
- Making informed decisions; whether it is taking calculated risks or confidence in application stability. A couple of examples include –
- Facilitating Quicker Release Cycles
- Shrinking down the execution time
- Percentage of un-tested code in Application
- How have been the Code Coverage’s from x Release x to y release and what’s the target benchmark set for forthcoming release?
Take Away/Where to go from Here: The Code Coverage Testing is typically done by the development Team. This is an attempt to help manual/functional/automation testers to get started on Code Coverage Testing. I strongly recommend starting with a real-time implementation taking the sample project for getting started. But before doing some tests, download a couple of Code Coverage Tools and learn how to hook on to the binaries of the Application under Test. Aim for the Tool that works best for your product and justifies business needs. Go through the internals of the Application under Test (an area which is always a black box for a tester). This should be followed by going through the Tools to narrow down for selection. Once a Tool is selected and basic Tests and some benchmarking is done. You are good to go. There is enough information available in the websites, Books, journals to help you make a baseline, creating a long-term strategy to make a significant impact on the whole testing life cycle and eventually the Product.