[Go][sum up] go test

Golang test:
https://golang.org/cmd/go/#hdr-Testing_flags
https://godoc.org/github.com/golang/go/src/cmd/go
$ go help testflag
$ go tool cover -help

Go test runs in two different modes:

• local directory mode, occurs when go test is invoked with no package arguments (for example, 'go test' or 'go test -v').
  In this mode, go test compiles the package sources and tests found in the current directory and then runs the resulting test binary.
  In this mode, caching is disabled. After the package test finishes, go test prints a summary line showing the test status ('ok' or 'FAIL'), package name, and elapsed time.
• package list mode, occurs when go test is invoked with explicit package arguments (for example 'go test math', 'go test ./...', and even 'go test .').
  In this mode, go test compiles and tests each of the packages listed on the command line.
  If a package test passes, go test prints only the final 'ok' summary line.
  If a package test fails, go test prints the full test output.
  If invoked with the -bench or -v flag, go test prints the full output even for passing package tests, in order to display the requested benchmark results or verbose logging.

Test cmd:
Provides chatty output for the testing.
$ go test -v

Disable test cache in 'go test' cache mode
$ go test -v -count=1

Go test also supports this flag and reports races.
Use this flag during development to detect the races.
$ go test -race

Filter tests to run by regex and the -run flag.
The following command will only test examples.
-run regexp
    Run only those tests and examples matching the regular expression.
    For tests, the regular expression is split by unbracketed slash (/)
    characters into a sequence of regular expressions, and each part
    of a test's identifier must match the corresponding element in
    the sequence, if any. Note that possible parents of matches are
    run too, so that -run=X/Y matches and runs and reports the result
    of all tests matching X, even those without sub-tests matching Y,
    because it must run them to look for those sub-tests.
$ go test -run=Example

This flag allows you to delegate some work to an
external program from the Go tool.
$ go test -exec


Converage cmd:
$ go test -cover

-coverprofile flag automatically sets -cover to enable coverage analysis.
$ go test -coverprofile=coverage.out

Counting statement execution for a standard package, the fmt formatting package.
$ go test -covermode=count -coverprofile=count.out fmt
$ go tool cover -func=coverage.out
$ go tool cover -html=coverage.out


go test cmd:

• The 'go test' runs "*_test.go" files corresponding to the package under test.
• Files whose names begin with "_" (including "_test.go") or "." are ignored.
• Test files that declare a package with the suffix "_test" will be compiled as a separate package, and then linked and run with the main test binary.
• The go tool will ignore a directory named "testdata", making it available to hold ancillary data needed by the tests.
• 'go test' runs 'go vet' on the package and its test source files to identify significant problems.
• To disable the running of go vet, use the -vet=off flag.
• All test output and summary lines are printed to the go command's standard output, even if the test printed them to its own standard error.
  (The go command's standard error is reserved for printing errors building the tests.)

Test function:
func TestXxx(t *testing.T) { ... }

Benchmark function:
func BenchmarkXxx(b *testing.B) { ... }

Example functions:

• An example function is similar to a test function but, instead of using *testing.T to report success or failure, prints output to os.Stdout.
• If the last comment in the function starts with "Output:" then the output is compared exactly against the comment.
• If the last comment begins with "Unordered output:" then the output is compared to the comment, however the order of the lines is ignored.
• An example with no such comment is compiled but not executed.
• An example with no text after "Output:" is compiled, executed, and expected to produce no output.
• Godoc displays the body of ExampleXxx to demonstrate the use of the function, constant, or variable Xxx.
• An example of a method M with receiver type T or *T is named ExampleT_M.
• There may be multiple examples for a given function, constant, or variable, distinguished by a trailing _xxx,where xxx is a suffix not beginning with an upper case letter.
• The entire test file is presented as the example when it contains a single example function, at least one other function, type, variable, or constant declaration, and no test or benchmark functions.

func ExamplePrintln() {
                Println("The output of\nthis example.")
                // Output: The output of
                // this example.
}

func ExamplePerm() {
                for _, value := range Perm(4) {
                        fmt.Println(value)
                }

                // Unordered output: 4
                // 2
                // 1
                // 3
                // 0
}


Coverage:
Reference:
https://blog.golang.org/cover
http://gcc.gnu.org/onlinedocs/gcc/Gcov.html

tl;dr
Golang converage uses code injection to rewrite the source code base on blocks for testing code coverage.


Excerpts from official document:
The usual way to compute test coverage is to instrument the binary.
For instance, the GNU gcov program sets breakpoints at branches executed by the binary.
As each branch executes, the break-point is cleared and the target statements of the branch are marked as 'covered'.

This approach is successful and widely used.
An early test coverage tool for Go even worked the same way.
But it has problems. It is difficult to implement, as analysis of the execution of binaries is challenging.
It also requires a reliable way of tying the execution trace back to the source code, which can also be difficult, as any user of a source-level debugger can attest.

Problems there include inaccurate debugging information and issues such as in-lined functions complicating the analysis.

Most important, this approach is very non-portable.
It needs to be done afresh for every architecture,
and to some extent for every operating system since debugging support varies greatly from system to system.


Another approach:
For the new test coverage tool for Go, we took a different approach that avoids dynamic debugging.
The idea is simple: Rewrite the package's source code before compilation to add instrumentation,compile and run the modified source, and dump the statistics.
The rewriting is easy to arrange because the go command controls the flow from source to test to execution.


How it works?
When test coverage is enabled, go test runs the "cover" tool,
a separate program included with the distribution, to rewrite the source code before compilation.


Original code:

package size

func Size(a int) string {
    switch {
    case a < 0:
        return "negative"
    case a == 0:
        return "zero"
    case a < 10:
        return "small"
    case a < 100:
        return "big"
    case a < 1000:
        return "huge"
    }
    return "enormous"
}

Modified code:

func Size(a int) string {
    GoCover.Count[0] = 1
    switch {
    case a < 0:
        GoCover.Count[2] = 1
        return "negative"
    case a == 0:
        GoCover.Count[3] = 1
        return "zero"
    case a < 10:
        GoCover.Count[4] = 1
        return "small"
    case a < 100:
        GoCover.Count[5] = 1
        return "big"
    case a < 1000:
        GoCover.Count[6] = 1
        return "huge"
    }
    GoCover.Count[1] = 1
    return "enormous"
}

Although that annotating assignment might look expensive, it compiles to a single "move" instruction. Its run-time overhead is therefore modest, adding only about 3% when running a typical (more realistic) test.

That makes it reasonable to include test coverage as part of the standard development pipeline.


Heat maps:
A big advantage of this source-level approach to test coverage is that it's easy to instrument the code in different ways.
For instance, we can ask not only whether a statement has been executed, but how many times.


Basic blocks:
Coverage annotations is demarcated by branches in the program.
It's hard to do that by rewriting the source, though, since the branches don't appear explicitly in the source.

What the coverage annotation does is instrument blocks, which are typically bounded by brace brackets.
e.g
f() && g()

Coverage has no attempt to separately instrument the calls to f and g, regardless of the facts it will always look like they both ran the same number of times, the number of times f ran.