How do you know that your code is well tested ?

The test coverage is the proportion of source code lines that are executed (covered) when running the tests. It is useful to find the parts of your code that are no exercised no matter how many test you add. It can also prove useful to spot dead code.

I implemented of proof of concept of code coverage, as a patch for R-3.0.2. To test i, I tried to evaluate the test coverage of the 10 most downloaded R packages in 2014 on the rstudio CRAN mirror (I got the data using the installr package).


These results are to be taken with caution:


plot of chunk plot

pkg nb_srcs covered lines coverage
colorspace 4 0 694 0
digest 3 83 89 93
ggplot2 163 1432 3769 38
plyr 66 674 903 75
RColorBrewer 1 0 57 0
Rcpp 18 394 1018 39
reshape2 11 169 190 89
scales 25 123 300 41
stringr 18 169 202 84
zoo 33 670 1614 42

We can note that packages digest, plyr, reshape2 and stringr are pretty well covered by their tests. colorspace and RColorBrewer do not contain any tests. Rcpp has a lot of tests, but there are two main problems:

getting deeper

Looking at digest, that has a high coverage of its R code (note most of its code is written in C and as such not covered nor coverable), one can wonder how to improve its test coverage. The test coverage output (not shown) reveals that lines 13:14,29,36,56,59 are missed for source file digest.R. Here is an excerpt of lines 5:15.

 digest <- function(object, algo=c("md5", "sha1", "crc32", "sha256", "sha512"),
                    serialize=TRUE, file=FALSE, length=Inf,
                    skip="auto", ascii=FALSE, raw=FALSE) {
   algo <- match.arg(algo)
   if (is.infinite(length)) {
     length <- -1               # internally we use -1 for infinite len

   if (is.character(file) && missing(object)) {
     object <- file  # line 13
     file <- TRUE    # line 14

The missed (non-covered) lines correspond to the case where no object is given but a filename. To improve the tests, we just have to add a test for this use case.

the test coverage current implementation

What I call the current test coverage implementation is actually just a proof of concept, it is a patch for R-3.0.1 and R-3.0.2, that I submitted to the R-devel list, and I must confess that I got absolutely no feedback.

I have no experience nor understanding of the internals of R. I had the idea of this hack trying to understand how the R profiler is implemented. It works by hooking in the R evaluation engine. It is not perfect, for instance functions whose body is not enclosed by braces are not covered, i.e this function:

not_covered <- function(x) x *x 

will not be covered, but this one will:

covered <- function(x) { x * x }

The overhead should be negligible when the coverage is not activated (by calling Rcov_start()), and is very low when activated.

The patch will provide the listing (and frequency) of the lines of code executed. But to be able to compute the coverage rate we also need to compute the coverable lines, this is currently done using parse() and getParseData(), and replicating the behaviour of the code coverage patch.

the future

We are successfully using this patch to increase the testing of our internal packages and as such of our services.

But once again this is a only proof of concept. I would like to contribute this feature to the community if someone manifests some interest. Having this feature as a patch is far from optimal, it may break for any new version of R, and is not really tested and reviewed. I would really appreciate advice and proper guidance to move on.

I believe that test coverage is a useful tool to improve the quality of software.