Infinite Loop

Covering it all up

In the previous two posts I described how you can create unit tests for threaded code as well as how you can de-couple your tests from external errors on the internet.

In this post I’ll show how to use GCOV and CoverStory to ensure that you are actually testing all parts of your code. Or at least obtain knowledge about which parts of your code that are being tested and which parts that are not being tested by your unit tests.

Having a high degree of code coverage in your unit tests helps to ensure that your code will continue working as expected even after you change parts of it, e.g. when fixing bugs, re-factoring it, adding new features, etc.

The coverage data can also be used to show when you have added enough tests to your code. Once you have obtained full test coverage of a certain function, adding more tests for that same function will probably not help you find more bugs in the code. Instead you can now focus your efforts on adding new functionality without having to worry about it breaking behind your back.


[AdSense - blog banner]

Adding Code Coverage to Xcode

First we’ll need to setup the Xcode project to generate GCOV code coverage data. The Apple Technical Q&A QA1514 details how to setup this for Xcode 3 but at the time of writing this post it has not been updated for Xcode 4.

Enabling gcov generation The setup is however very similar as shown in the screen dump to the left. For now, we’ll just enable GCOV for the default UnitTest target.

Update: If you are running Xcode 4.2 or newer you can skip the next section regarding GCC as detailed here.

Build using gcc For whatever reason the LLVM compiler doesn’t support generation of code coverage files so we’ll need to use the “old” GCC compiler instead. This is done by adding a custom build rule to the UnitTest target specifying that GCC shall be used for C source files.

This setup means that code coverage data is now generated every time the unit tests are executed. This doesn’t significantly affects the performance of the unit tests since GCOV only adds little overhead to the execution time.

Build output hierarchy

When we now run the unit test target we should hopefully see that GCOV have created a .gcno and .gcda file for each of the .o object files. These files are normally stored deeply nested inside your build output folder.

On Xcode 4 it can be a bit tricky to locate the Build folder. If you open the Organizer window and select the Projects icon, each of your projects will be listed along with their various folders, snapshots, etc. Click on the small icon to the right of the Derived Data path and the Build folder should open in the Finder.

If you do not see the .gcda and .gcno files in the folder make sure you have defined the correct build parameters and are actually using GCC as the compiler. Then clean the target and do a fresh build of the UnitTest target.

Displaying the Coverage Data

We can display a summary of the coverage data from the unit tests by running the following commands in the Terminal:

$ cd very-long-path-to/Objects-normal/i386
$ gcov --branch-probabilities --no-output ILGeoNamesLookup.m

This should produce a summary of all the code lines being executed during the unit test. At the end of the list we should see some output similar to this:

File '..../Source/ILGeoNamesLookup.m'
Lines executed:35.90% of 117
Branches executed:39.13% of 46
Taken at least once:19.57% of 46
Calls executed:25.64% of 117

Overall we can see that only 35.9% of the code lines in ILGeoNamesLookup.m are actually being tested by the unit tests. That’s really not that impressive so let’s have a closer look at which lines are actually being tested and which are not.

It’s possible to have GCOV create a detailed output of which lines are covered by omitting the --no-output parameter in the command. That output format is however not very user friendly so instead we’ll use a tool called CoverStory.

CoverStory is a Mac OS X application that enables you to easily view the code coverage of your unit tests. CoverStory is available as Open Source and can be downloaded from Google Code. Once you have downloaded it, just launch it and point it to the GCOV output folder.

It should be no surprise that the methods -sendRequestWithURLString: and -threadedRequestWithURLString: which have been replaced by the mock object are not being covered by the unit tests. What’s more interesting is that only a very small part of the -connectionDidFinishLoading: method is actually being tested:

Initial code coverage This method is responsible for all the parsing and error handling of the results received from the geonames.org services.

Currently only the main success path is being tested. There are no tests of the various error conditions that may occur:

- Our request being rejected by the geonames.org service
- Empty search results returned from the geonames.org service
- Malformed response, i.e. invalid JSON

Let’s have a look at how to improve this.

Improving the Test Coverage

The first error condition can be triggered by providing an invalid request to the geonames.org service, e.g. by providing invalid latitude/longitude. Since we already prepared the test suite for injection of test data we can just add another unit test which returns the canned error response:

-(void)testInvalidPosition {
    // Mock the ILGeoNamesLookup so no actual network access is performed
    [self loadCannedResultWithName:@"InvalidPosition"];
    mockParser = [OCMockObject partialMockForObject:parser];
    [[[mockParser stub] andCall:@selector(returnCannedResultForRequest:)
                       onObject:self]
        sendRequestWithURLString:[OCMArg any]];
 
    // Perform code under test
    [parser findNearbyPlaceNameForLatitude:-100.0
                                 longitude:-10.0];
    // Validate result
    ...
}

The test data is generated by issuing the following command in the Terminal:

$ curl "http://api.geonames.org/findNearbyJSON?lat=-100.0000000&lng=-10.00000000&style=FULL&username=unittest" -o InvalidPositions.json

The second error condition can be triggered by manually crafting an empty response. This would be difficult to test in a live scenario since it would require the geonames.org service to malfunction:

-(void)testEmptyResponse {
    // Mock the ILGeoNamesLookup so no actual network access is performed
    [self loadCannedResultWithName:@"EmptyResponse"];
    mockParser = [OCMockObject partialMockForObject:parser];
    [[[mockParser stub] andCall:@selector(returnCannedResultForRequest:)
                       onObject:self]
        sendRequestWithURLString:[OCMArg any]];
 
    // Perform code under test
    [parser findNearbyPlaceNameForLatitude:37.3
                                 longitude:-122.0];
    // Validate result
    ...
}

The test data is generated by handcrafting an empty but otherwise valid JSON file. The third test case is created in the same way, but this time the canned response contains an invalid JSON file that will cause the parser to fail.

When we now run the unit test we can see that the overall code coverage have increased to 49.6%. But what’s even more important is that we now have 100% coverage of the -connectionDidFinishLoading: method in our unit tests:

Improved code coverage

Conclusion

As I have shown above it is possible to dramatically increase the coverage of unit tests by using GCOV and CoverStory to pinpoint where the tests are currently lacking. This will make it a lot easier to obtain a high degree of coverage without adding a lot of redundant or superfluous tests which may not improve the code quality, but instead only end up contributing to waste of precious development time.

Generally it should be an overall goal to have as high a code coverage as feasible and practically possible. But, even though you end up having full code coverage in your unit tests it will not guarantee that every single bug is found by the automated tests. But it will do a very good job of keeping an eye out for many of the bugs that try to sneak into your code while you are not watching.

The Xcode project used in this post as well as the complete unit tests can be downloaded from GitHub.

Comments (13) | Trackback

13 Responses to “Covering it all up”

  1. Martijn The says:

    Hi Claus and thanks for the article on GCov / Xcode 4.
    I was wondering why you add a custom build rule for C files, instead of just setting GCC in the build settings (“Compiler Version”).

    Martijn

    • Claus Broch says:

      Martijn you got a point. I think the reason was that I struggled a bit to get Xcode to select GCC when I original wrote this tutorial.
      I’ve later had it working too just by setting the “Compiler for C/C++/Objective-C” (GCC_VERSION) to “4.2″

      • Philippe Bernery says:

        Hi Claus,

        Thanks for this article. I wanted to let you know that latest LLVM version include in Xcode can generate the appropriate files. You just need to link to profile_rt instead of gcov. The file is in /Developer/Platforms/iPhoneSimulator.platform/Developer/usr/lib

        • Claus Broch says:

          Thanks Philippe for your suggestion. I’ll have a look at it shortly, especially since gcc 4.2 doesn’t seem to be supported any longer on Lion.

  2. [...] Covering it all up 13 OCT 0 Share Categories: iPhone, Programming Tags: iPhone, Programming /* [...]

  3. Hernan says:

    Hi, thanks for the post. I cannot get it to work with latest XCode (4C199). Seems GCC is not there, only LLVM GCC 4.2, so I guess that’s why I only see .gcno files generated. Any ideas? Thanks.

  4. Praveen says:

    Hi all,

    I am facing one issue regarding code coverage setup for Unit tests. I created unit test target for my project. Generate Test Coverage Files and Instrument Program Flow i made it YES for the Unit test target. Now i’m able to Test the project. Now my issue is after Test done, i’m getting only .gcno. not .gcda. Without .gcda file, we can’t get the percentage coverage. Any ideas appreciated.

    Thanks in Advance
    Praveen

  5. [...] author in the first link mentions “Just run your unit tests and view the code coverage data as usual”; however, I cannot find [...]

  6. [...] has built-in functionality to record coverage reports. How to set this up is described in detail in this blog post. To get this to work for me, I had to enable “Generate Test Coverage Files” and [...]

  7. [...] background information about how to customize the configuration for coverage from the articles Covering it all up and How to Measure Code Coverage in [...]

Leave a Reply

*