elm-coverage¶

elm-coverage is a tool for calculating code coverage for Elm code tested with elm-test.

The goal of the reports generated by elm-coverage is to help you visualize what parts of your code are being evaluated when running your tests, and to try and guide you towards writing tests that focus specifically on the more complex functions in your codebase.

The goal is not to condense that information down into a single metric. It is too easy to write tests that don’t make meaningful assertions about your code and its behaviour, but only serve to increase the coverage.

The only thing worse than having no tests is having tests that provide a false sense of security.

Installation¶

Installing elm-coverage works much the same way as installing other tools in the Elm ecosystem:

npm i -g elm-coverage

Usage¶

The simplest invocation of elm-coverage is to simply invoke elm-coverage in the root of your project:

elm-coverage

By default, elm-coverage assumes that your sources exist in a separate src/ directory, that you have elm-test installed globally, and that elm-test needs no further, special flags.

You can specify an alternative path to crawl for sources to instrument:

elm-coverage elm_src/

If you don’t want to use a globally installed elm-test, you can specify the path to an elm-test executable:

elm-coverage --elm-test ./node_modules/.bin/elm-test

Parameters following -- are passed through to elm-test, for example to specify the initial seed and number of fuzz testruns:

elm-coverage -- --seed 12345 --fuzz 99

elm-coverage will write an HTML report to .coverage/coverage.html. It is not recommended to version control this directory. In order to open the report once it is generated, you can specify the --open option:

elm-coverage --open

Contribute¶

Issues and source code is available on github. The source for elm-instrument which is used to actually instrument the sources in order to calculate coverage is also available.

License¶

elm-coverage is licensed under the BSD-3 license. elm-instrument is licensed under the BSD-3 license.

Further reading¶

Reports¶

TODO: Give examples and information.

Cyclomatic Complexity¶

Cyclomatic complexity gives an indication of how complex a piece of code is. Essentially, it counts the number of paths through code.

The goal of including complexity information is to help prioritizing which functions to write tests for. A function (or module) with a very high cyclomatic complexity and low test-coverage is a good place to start testing.

Expression complexity¶

There are only two expressions that - by themselves - increment cyclomatic complexity: if <cond> then and branches in case <expr> of.

Let’s look at some examples:

if a == 12 then
    "It was twelve!"
else
    "It wasn't twelve..."

The above example has a cyclomatic complexity of 1, as there is 1 extra flow through the code added. Similarly:

if a == 6 then
    "It was six."
else if a == 12 then
    "It was twelve!"
else
    "It was just a random number..."

This expression has a cyclomatic complexity of 2.

For case <expr> of, similar rules apply. A single branch (which can either be a catch-all or a destructuring) does not increase complexity; as all pattern matches in Elm must be exhaustive, we know that this branch was the only option and as such, does not introduce a decision point.

As such, and easy way to compute the complexity of an expression is to count the number of if branches, add the number of case branches and subtract the number of case <expr> of expressions.

Declaration complexity¶

The total complexity of a top-level declaration is simply the total complexity of its body + 1. Note that let-bindings do not - by themselves - increase the complexity of a declaration. If they contain case <expr> of expressions of if <cond> then expressions, however, they will count towards the complexity of the surrounding declaration. The same rule is applied to anonymous functions.

Module complexity¶

The complexity of a module is calculated by taking the sum of the calculated complexity of each declaration, subtracting the number of declarations and adding one.

As such, no matter how many declarations are defined in a module, if they all have complexity 1, the module will also have complexity 1. If there is one declaration with complexity 2 and one declaration with complexity 3, the total complexity of that module will be 4.

Under the hood¶

elm-coverage consists of a fair number of moving parts. This is my attempt to document how those parts work together, and which part is responsible for what.

The runner¶

The runner or supervisor is the main entrypoint and is responsible for glueing all the pieces together to a coherent whole.

Its responsibilities are roughly these:

Parse the commandline arguments
Traverse the source-path, looking for Elm-files
Create a backup of all of these and instrument the originals in-place using elm-instrument
Modify tests/elm-package.json to know where the Coverage module is
Run elm-test
Restore all the backups (sources and tests/elm-package.json
Instruct the analyzer to analyze the generated coverage files and create a report
Optionally, try to open the generated report in the user’s browser

Instrumenting with `elm-instrument`¶

Instrumentation is handled by an AST->AST transformation implemented in a fork of elm-format - since that project happens to have the highest quality parser+writer in the ecosystem, battle-tested on hundreds of projects.

The AST is traversed and modified while also keeping track of a few bit of information. Certain expressions (specifically the bodies of declarations, let-declarations, lambda’s, if/else branches and case..of branches) are instrumented with a let _ = Coverage.track <moduleIdentifier> <expressionIdentifier> in expression. Whenever instrumentation is added, some information about the instrumented expression is tracked.

	Source location	Cyclomatic complexity	Name
Declaration	x	x	x
Let declaration	x	x
Lambda body	x	x
if/else branch	x
case..of branch	x

The recorded information is accumulated for all instrumented modules and persisted to .coverage/info.json.

The `Coverage` module¶

The Coverage module, which is “linked in” by the runner, exposes a single function:

Coverage.track : String -> Int -> Never -> a

It is passed the module-name and an offset in the total list of track expressions of a module, and returns a function that can never be called. When evaluated, the internal coverage-data is updated; incrementing a simple counter based on the module-name and offset.

When the active process signals elm-test that all of its tests have finished running, the coverage data is persisted to disk in a coverage-{{pid}}.json file.

The analyzer¶

The analyzer is a thin wrapper around an Elm module. The wrapper reads in the info.json file created by the instrumenter, all the coverage files created by the elm-test run, and all the sources of the referenced files. Once all the data is read, it is bundled up and sent off to an Elm module for further processing.

The Elm module parses all that data and creates the HTML report, returning the generated report as a String over a port.