covr: Test coverage for packages

Description

covr tracks and reports code coverage for your package and (optionally) upload the results to a coverage service like 'Codecov' https://about.codecov.io or 'Coveralls' https://coveralls.io. Code coverage is a measure of the amount of code being exercised by a set of tests. It is an indirect measure of test quality and completeness. This package is compatible with any testing methodology or framework and tracks coverage of both R code and compiled C/C++/FORTRAN code.

Details

A coverage report can be used to inspect coverage for each line in your package. Using report() requires the additional dependencies DT and htmltools.

# If run with no arguments `report()` implicitly calls `package_coverage()`
report()

Package options

covr uses the following options() to configure behaviour:

• covr.covrignore: A filename to use as an ignore file, listing glob-style wildcarded paths of files to ignore for coverage calculations. Defaults to the value of environment variable COVR_COVRIGNORE, or ".covrignore" if the neither the option nor the environment variable are set.

• covr.exclude_end: Used along with covr.exclude_start, an optional regular expression which ends a line-exclusion region. For more details, see ?exclusions.

• covr.exclude_pattern: An optional line-exclusion pattern. Lines which match the pattern will be excluded from coverage. For more details, see ?exclusions.

• covr.exclude_start: Used along with covr.exclude_end, an optional regular expression which starts a line-exclusion region. For more details, see ?exclusions.

• covr.filter_non_package: If TRUE (the default behavior), coverage of files outside the target package are filtered from coverage output.

• covr.fix_parallel_mcexit:

• covr.flags:

• covr.gcov: If the appropriate gcov version is not on your path you can use this option to set the appropriate location. If set to "" it will turn off coverage of compiled code.

• covr.gcov_additional_paths:

• covr.gcov_args:

• covr.icov:

• covr.icov_args:

• covr.icov_flags:

• covr.icov_prof:

• covr.rstudio_source_markers: A logical value. If TRUE (the default behavior), source markers are displayed within the RStudio IDE when using zero_coverage.

• covr.record_tests: If TRUE (default NULL), record a listing of top level test expressions and associate tests with covr traces evaluated during the test's execution. For more details, see ?covr.record_tests.

• covr.showCfunctions:

Author(s)

Maintainer: Jim Hester james.f.hester@gmail.com

Other contributors:

• Willem Ligtenberg [contributor]

• Kirill Müller [contributor]

• Henrik Bengtsson [contributor]

• Steve Peak [contributor]

• Kirill Sevastyanenko [contributor]

• Jon Clayden [contributor]

• Robert Flight [contributor]

• Eric Brown [contributor]

• Brodie Gaslam [contributor]

• Will Beasley [contributor]

• Robert Krzyzanowski [contributor]

• Markus Wamser [contributor]

• Karl Forner [contributor]

• Gergely Daróczi [contributor]

• Jouni Helske [contributor]

• Kun Ren [contributor]

• Jeroen Ooms [contributor]

• Ken Williams [contributor]

• Chris Campbell [contributor]

• David Hugh-Jones [contributor]

• Qin Wang [contributor]

• Doug Kelkhoff [contributor]

• Ivan Sagalaev (highlight.js library) [contributor, copyright holder]

• Mark Otto (Bootstrap library) [contributor]

• Jacob Thornton (Bootstrap library) [contributor]

• Bootstrap contributors (Bootstrap library) [contributor]

• Twitter, Inc (Bootstrap library) [copyright holder]

See Also

Useful links:

• https://covr.r-lib.org

• https://github.com/r-lib/covr

• Report bugs at https://github.com/r-lib/covr/issues

Convert a counters object to a coverage object

Description

Convert a counters object to a coverage object

Usage

as_coverage(counters = NULL, ...)

Arguments

Clean and restructure counter tests for a coverage object

Description

For tests produced with options(covr.record_tests), prune any unused records in the $tests$tally matrices of each trace and get rid of the wrapping $tests environment (reassigning with value of $tests$tally)

Usage

as_coverage_with_tests(counters)

Arguments

Run covr on a package and output the result so it is available on Azure Pipelines

Description

Run covr on a package and output the result so it is available on Azure Pipelines

Usage

azure(
  ...,
  coverage = package_coverage(..., quiet = quiet),
  filename = "coverage.xml",
  quiet = TRUE
)

Arguments

clear all previous counters

Description

clear all previous counters

Usage

clear_counters()

Calculate coverage of code directly

Description

This function is useful for testing, and is a thin wrapper around file_coverage() because parseData is not populated properly unless the functions are defined in a file.

Usage

code_coverage(
  source_code,
  test_code,
  line_exclusions = NULL,
  function_exclusions = NULL,
  ...
)

Arguments

Examples

source <- "add <- function(x, y) { x + y }"
test <- "add(1, 2) == 3"
code_coverage(source, test)

Run covr on a package and upload the result to codecov.io

Description

Run covr on a package and upload the result to codecov.io

Usage

codecov(
  ...,
  coverage = NULL,
  base_url = "https://codecov.io",
  token = NULL,
  commit = NULL,
  branch = NULL,
  pr = NULL,
  flags = NULL,
  quiet = TRUE
)

Arguments

Examples

## Not run: 
codecov(path = "test")

## End(Not run)

increment a given counter

Description

increment a given counter

Usage

count(key)

Arguments

Append a test trace to a counter, updating global current test

Description

Append a test trace to a counter, updating global current test

Usage

count_test(key)

Arguments

Convert a coverage dataset to a list

Description

Convert a coverage dataset to a list

Usage

coverage_to_list(x = package_coverage())

Arguments

Value

A list containing coverage result for each individual file and the whole package

Run covr on a package and upload the result to coveralls

Description

Run covr on a package and upload the result to coveralls

Usage

coveralls(
  ...,
  coverage = NULL,
  repo_token = Sys.getenv("COVERALLS_TOKEN"),
  service_name = Sys.getenv("CI_NAME", "travis-ci"),
  quiet = TRUE
)

Arguments

Record Test Traces During Coverage Execution

Description

By setting options(covr.record_tests = TRUE), the result of covr coverage collection functions will include additional data pertaining to the tests which are executed and an index of which tests, at what stack depth, trigger the execution of each trace.

Details

This functionality requires that the package code and tests are installed and sourced with the source. For more details, refer to R options, keep.source, keep.source.pkgs and keep.parse.data.pkgs.

Additional fields

Within the covr result, you can explore this information in two places:

• attr(,"tests"): A list of call stacks, which results in target code execution.

• ⁠$<srcref>$tests⁠: For each srcref count in the coverage object, a ⁠$tests⁠ field is now included which contains a matrix with three columns, "test", "depth" and "i" which specify the test number (corresponding to the index of the test in attr(,"tests"), the stack depth into the target code where the trace was executed, and the order of execution for each test.

Test traces

The content of test traces are dependent on the unit testing framework that is used by the target package. The behavior is contingent on the available information in the sources kept for the testing files.

Test traces are extracted by the following criteria:

1. If any srcref files are are provided by a file within covr's temporary library, all calls from those files are kept as a test trace. This will collect traces from tests run with common testing frameworks such as testthat and RUnit.

2. Otherwise, as a conservative fallback in situations where no source references are found, or when none are from within the temporary directory, the entire call stack is collected.

These calls are subsequently subset for only those up until the call to covr's internal count function, and will always include the last call in the call stack prior to a call to count.

Examples

fcode <- '
f <- function(x) {
  if (x)
    f(!x)
  else
    FALSE
}'

options(covr.record_tests = TRUE)
cov <- code_coverage(fcode, "f(TRUE)")

# extract executed test code for the first test
tail(attr(cov, "tests")[[1L]], 1L)
# [[1]]
# f(TRUE)

# extract test itemization per trace
cov[[3]][c("srcref", "tests")]
# $srcref
# f(!x)
#
# $tests
#      test depth i
# [1,]    1     2 4

# reconstruct the code path of a test by ordering test traces by [,"i"]
lapply(cov, `[[`, "tests")
# $`source.Ref2326138c55:4:6:4:10:6:10:4:4`
#      test depth i
# [1,]    1     1 2
#
# $`source.Ref2326138c55:3:8:3:8:8:8:3:3`
#      test depth i
# [1,]    1     1 1
# [2,]    1     2 3
#
# $`source.Ref2326138c55:6:6:6:10:6:10:6:6`
#      test depth i
# [1,]    1     2 4

Retrieve the path name (filename) for each coverage object

Description

Retrieve the path name (filename) for each coverage object

Usage

display_name(x)

Arguments

Calculate coverage of an environment

Description

Calculate coverage of an environment

Usage

environment_coverage(
  env = parent.frame(),
  test_files,
  line_exclusions = NULL,
  function_exclusions = NULL
)

Arguments

Exclusions

Description

covr supports a couple of different ways of excluding some or all of a file.

Line Exclusions

The line_exclusions argument to package_coverage() can be used to exclude some or all of a file. This argument takes a list of filenames or named ranges to exclude.

Function Exclusions

Alternatively function_exclusions can be used to exclude R functions based on regular expression(s). For example ⁠print\\\.*⁠ can be used to exclude all the print methods defined in a package from coverage.

Exclusion Comments

In addition you can exclude lines from the coverage by putting special comments in your source code. This can be done per line or by specifying a range. The patterns used can be specified by the exclude_pattern, exclude_start, exclude_end arguments to package_coverage() or by setting the global options covr.exclude_pattern, covr.exclude_start, covr.exclude_end.

Examples

## Not run: 
# exclude whole file of R/test.R
package_coverage(exclusions = "R/test.R")

# exclude lines 1 to 10 and 15 from R/test.R
package_coverage(line_exclusions = list("R/test.R" = c(1:10, 15)))

# exclude lines 1 to 10 from R/test.R, all of R/test2.R
package_coverage(line_exclusions = list("R/test.R" = 1:10, "R/test2.R"))

# exclude all print and format methods from the package.
package_coverage(function_exclusions = c("print\\.", "format\\."))

# single line exclusions
f1 <- function(x) {
  x + 1 # nocov
}

# ranged exclusions
f2 <- function(x) { # nocov start
  x + 2
} # nocov end

## End(Not run)

Calculate test coverage for sets of files

Description

The files in source_files are first sourced into a new environment to define functions to be checked. Then they are instrumented to track coverage and the files in test_files are sourced.

Usage

file_coverage(
  source_files,
  test_files,
  line_exclusions = NULL,
  function_exclusions = NULL,
  parent_env = parent.frame()
)

Arguments

Examples

# For the purpose of this example, save code containing code and tests to files
cat("add <- function(x, y) { x + y }", file="add.R")
cat("add(1, 2) == 3", file="add_test.R")

# Use file_coverage() to calculate test coverage
file_coverage(source_files = "add.R", test_files = "add_test.R")

# cleanup
file.remove(c("add.R", "add_test.R"))

A coverage report for a specific file

Description

A coverage report for a specific file

Usage

file_report(
  x = package_coverage(),
  file = NULL,
  out_file = file.path(tempdir(), paste0(get_package_name(x), "-file-report.html")),
  browse = interactive()
)

Arguments

Calculate test coverage for a specific function.

Description

Calculate test coverage for a specific function.

Usage

function_coverage(fun, code = NULL, env = NULL, enc = parent.frame())

Arguments

Examples

add <- function(x, y) { x + y }
function_coverage(fun = add, code = NULL) # 0% coverage
function_coverage(fun = add, code = add(1, 2) == 3) # 100% coverage

Run covr on package and create report for GitLab

Description

Utilize internal GitLab static pages to publish package coverage. Creates local covr report in a package subdirectory. Uses the pages GitLab job to publish the report.

Usage

gitlab(..., coverage = NULL, file = "public/coverage.html", quiet = TRUE)

Arguments

Is the source bound to the expression

Description

Is the source bound to the expression

Usage

has_srcref(expr)

Arguments

Value

A logical value indicating whether the language object has source

Determine if code is being run in covr

Description

covr functions set the environment variable R_COVR when they are running. in_covr() returns TRUE if this environment variable is set and FALSE otherwise.

Usage

in_covr()

Examples

if (require(testthat)) {
  testthat::skip_if(in_covr())
}

Is the expression a call to covr:::count

Description

Is the expression a call to covr:::count

Usage

is_covr_count_call(expr)

Arguments

Value

A logical value indicating whether the object is a call to covr:::count.

Returns TRUE if we've moved on from test reflected in .current_test

Description

Quickly dismiss the need to update the current test if we can. To test if we're still in the last test, check if the same srcref (or call, if source is not kept) exists at the last recorded calling frame prior to entering a covr trace. If this has changed, do a more comprehensive test to see if any of the test call stack has changed, in which case we are onto a new test.

Usage

is_current_test_finished()

Generate a key for a call

Description

Generate a key for a call

Usage

key(x)

Arguments

initialize a new counter

Description

initialize a new counter

Usage

new_counter(src_ref, parent_functions)

Arguments

Initialize a new test counter for a coverage trace

Description

Initialize a test counter, a matrix used to tally tests, their stack depth and the execution order as the trace associated with key is hit. Each test trace is an environment, which allows assignment into a pre-allocated tests matrix with minimall reallocation.

Usage

new_test_counter(key)

Arguments

Details

The tests matrix has columns tests, depth and i, corresponding to the test index (the index of the associated test in .counters$tests), the stack depth when the trace is evaluated and the number of traces that have been hit so far during test evaluation.

Calculate test coverage for a package

Description

This function calculates the test coverage for a development package on the path. By default it runs only the package tests, but it can also run vignette and example code.

Usage

package_coverage(
  path = ".",
  type = c("tests", "vignettes", "examples", "all", "none"),
  combine_types = TRUE,
  relative_path = TRUE,
  quiet = TRUE,
  clean = TRUE,
  line_exclusions = NULL,
  function_exclusions = NULL,
  code = character(),
  install_path = temp_file("R_LIBS"),
  ...,
  exclusions,
  pre_clean = TRUE
)

Arguments

Details

This function uses tools::testInstalledPackage() to run the code, if you would like to test your package in another way you can set type = "none" and pass the code to run as a character vector to the code parameter.

Parallelized code using parallel's mcparallel() needs to use a patched parallel:::mcexit. This is done automatically if the package depends on parallel, but can also be explicitly set using the environment variable COVR_FIX_PARALLEL_MCEXIT or the global option covr.fix_parallel_mcexit.

See Also

exclusions() For details on excluding parts of the package from the coverage calculations.

Provide percent coverage of package

Description

Calculate the total percent coverage from a coverage result object.

Usage

percent_coverage(x, ...)

Arguments

Value

The total percentage as a numeric(1).

Print a coverage object

Description

Print a coverage object

Usage

## S3 method for class 'coverage'
print(x, group = c("filename", "functions"), by = "line", ...)

Arguments

Value

The coverage object (invisibly).

Display covr results using a standalone report

Description

Display covr results using a standalone report

Usage

report(
  x = package_coverage(),
  file = file.path(tempdir(), paste0(get_package_name(x), "-report.html")),
  browse = interactive()
)

Arguments

Examples

## Not run: 
x <- package_coverage()
report(x)

## End(Not run)

Run a system command and check if it succeeds.

Description

This function automatically quotes both the command and each argument so they are properly protected from shell expansion.

Usage

system_check(
  cmd,
  args = character(),
  env = character(),
  quiet = FALSE,
  echo = FALSE,
  ...
)

Arguments

Value

TRUE if the command succeeds, an error will be thrown if the command fails.

Run a system command and capture the output.

Description

This function automatically quotes both the command and each argument so they are properly protected from shell expansion.

Usage

system_output(
  cmd,
  args = character(),
  env = character(),
  quiet = FALSE,
  echo = FALSE,
  ...
)

Arguments

Value

command output if the command succeeds, an error will be thrown if the command fails.

Tally coverage by line or expression

Description

Tally coverage by line or expression

Usage

tally_coverage(x, by = c("line", "expression"))

Arguments

Value

a data.frame of coverage tallied by line or expression.

Create a Cobertura XML file

Description

Create a cobertura-compliant XML report following this DTD. Because there are two DTDs called ⁠coverage-04.dtd⁠ and some tools do not seem to adhere to either of them, the parser you're using may balk at the file. Please see this github discussion for context. Where covr doesn't provide a coverage metric (branch coverage, complexity), a zero is reported.

Usage

to_cobertura(cov, filename = "cobertura.xml")

Arguments

Details

Note: This functionality requires the xml2 package be installed.

Create a SonarQube Generic XML file for test coverage according to https://docs.sonarqube.org/latest/analysis/generic-test/ Based on cobertura.R

Description

This functionality requires the xml2 package be installed.

Usage

to_sonarqube(cov, filename = "sonarqube.xml")

Arguments

Author(s)

Talkdesk Inc.

trace each call with a srcref attribute

Description

This function calls itself recursively so it can properly traverse the AST.

Usage

trace_calls(x, parent_functions = NULL, parent_ref = NULL)

Arguments

Value

a modified expression with count calls inserted before each previous call.

See Also

http://adv-r.had.co.nz/Expressions.html

Truncate call objects to limit the number of arguments

Description

A helper to circumvent R errors when deserializing large call objects from Rds. Trims the number of arguments in a call object, and replaces the last argument with a ⁠<truncated>⁠ symbol.

Usage

truncate_call(call_obj, limit = 10000)

Arguments

Value

The call_obj with arguments trimmed

Update current test if unit test expression has progressed

Description

Updating a test logs some metadata regarding the current call stack, noteably trying to capture information about the call stack prior to the covr::count call being traced.

Usage

update_current_test()

Details

There are a couple patterns of behavior, which try to accommodate a variety of testing suites:

• testthat: During execution of testthat's ⁠test_*⁠ functions, files are sourced and the working directory is temporarily changed to the package ⁠/tests⁠ directory. Knowing this, calls in the call stack which are relative to this directory are extracted and recorded.

• RUnit:

• custom: Any other custom test suites may not have source kept with their execution, in which case the entire test call stack is kept.

checks to see if the current call stack has the same srcref (or expression, if no source is available) at the same frame prior to entering into a package where covr:::count is called.

Retrieve the value from an object

Description

Retrieve the value from an object

Usage

value(x, ...)

Arguments

Provide locations of zero coverage

Description

When examining the test coverage of a package, it is useful to know if there are any locations where there is 0 test coverage.

Usage

zero_coverage(x, ...)

Arguments

Details

if used within RStudio this function outputs the results using the Marker API.

Value

A data.frame with coverage data where the coverage is 0.