[Go] cmd to compile to assembly with cgo embedded

Reference:

https://groups.google.com/g/golang-dev/c/sJU3GHYmQO0/m/3O-VMpvpCAAJ

$ go build -gcflags=-S hello.go

[Go] Build constraints

 https://golang.org/cmd/go/#hdr-Build_constraints

[Go] inline gotya

Reference:
https://vsdmars.blogspot.com/2020/02/golang-tooling-update.html

https://github.com/golang/go/wiki/CompilerOptimizations

Proposal:

https://go.googlesource.com/proposal/+/refs/heads/master/design/34481-opencoded-defers.md


Function Inlining: (compiled with -gcflags="-m")
Only short and simple functions are inlined.
To be inlined a function must contain less than ~40 expressions(80 AST nodes)  and does not contain complex things like 

• loops (for), 
• labels, 
• closures,
• panic's,
• recover's,
• select's,
• switch
• defer (now up-lifted thanks to Proposal: Low-cost defers through inline code, and extra funcdata to manage the panic case)
• go (goroutine creation)

Visualize with flag:

-gcflags="-d pctab=pctoinline"

[Go] Inline optimization flags

https://github.com/golang/go/blob/62ccee49d6d9bdb63841a259d835703ff85ab0b7/src/cmd/compile/internal/gc/inl.go#L10

disabling optimisations
$ -gcflags='-l -N'

More aggressive opt
$ -gcflags='-l -l'

// The debug['l'] flag controls the aggressiveness. Note that main() swaps level 0 and 1,
// making 1 the default and -l disable. Additional levels (beyond -l) may be buggy and
// are not supported.
//      0: disabled
//      1: 80-nodes leaf functions, oneliners, panic, lazy typechecking (default)
//      2: (unassigned)
//      3: (unassigned)
//      4: allow non-leaf functions

[Go] tooling update (compile time optimization, linker, compiler)

Reference:
https://rakyll.org/go-tool-flags/
https://golang.org/cmd/go/
https://github.com/golang/go/wiki/CompilerOptimizations
https://www.alexedwards.net/blog/an-overview-of-go-tooling#building-an-executable



Lists all the commands go build invokes
$ go build -x

Show flags that can be passed to compiler
$ go tool compile -help

e.g. to disable compiler optimizations and inlining, you can use the following the gcflags.
$ go build -gcflags="-N -l"


Disable bounds checking
$ go build -gcflags="-B"

Show compiler debug list:
$ go tool compile -d help 2>&1|bat

Show compiler -d ssa list:
compile: PhaseOptions usage:
$ go tool compile -d=ssa/<phase>/<flag>[=<value>|<function_name>]
$ go tool compile -d ssa/help 2>&1|bat

e.g If we want to check which line has the  "Bounds Check Elimination"
applied:
https://golang.org/src/cmd/compile/internal/ssa/checkbce.go
$ go build -gcflags="-d=ssa/check_bce/debug=1"



Linker help:
$ go tool link -help

"burn in" a (string) value to a specific variable in your application:
$ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo .

Strip debugging information from the binary.
$ go build -ldflags="-s -w" -o=/tmp/foo .  # Strip debug information from the binary



Escape analysis:
Gc compiler does global escape analysis across function and package boundaries. However, there are lots of cases where it gives up.
For example, anything assigned to any kind of indirection (*p = ...) is considered escaped.
Other things that can inhibit analysis are: function calls, package boundaries, slice literals, sub-slicing and indexing, etc.
Full rules are too complex to describe, so check the -m output.
$ go build -gcflags -m


Function Inlining:
Only short and simple functions are inlined.
To be inlined a function must contain less than ~40 expressions and does not contain complex things like loops, labels, closures, panic's, recover's, select's, switch'es, etc.


Non-scannable objects:
Garbage collector does not scan underlying buffers of slices, channels and maps when element type does not contain pointers (both key and value for maps)(this is called tracing).

This allows to hold large data sets in memory without paying high price during garbage collection.

For example, the following map won't visibly affect GC time:

type Key [64]byte // SHA-512 hash
type Value struct {
 Name      [32]byte
 Balance   uint64
 Timestamp int64
}
m := make(map[Key]Value, 1e8)

[Go][bookmark] toolchain tricks

Bookmark:
https://github.com/golang/go/wiki/GcToolchainTricks
https://github.com/golang/go/wiki

[Go] race detector

Reference:
https://golang.org/doc/articles/race_detector.html



Usage:
$ go test -race mypkg // to test the package

$ go run -race mysrc.go // to run the source file

$ go build -race mycmd // to build the command

$ go install -race mypkg // to install the package

Options:

The GORACE environment variable sets race detector options.

The format is:

$ GORACE="option1=val1 option2=val2"

The options are:

• log_path (default stderr):
  The race detector writes its report to a file named log_path.pid. The special names stdout and stderr cause reports to be written to standard output and standard error, respectively.
• exitcode (default 66):
  The exit status to use when exiting after a detected race.
• strip_path_prefix (default ""):
  Strip this prefix from all reported file paths, to make reports more concise.
• history_size (default 1):
  The per-goroutine memory access history is 32K * 2**history_size elements.
  Increasing this value can avoid a "failed to restore the stack" error in reports, at the cost of increased memory usage.
• halt_on_error (default 0):
  Controls whether the program exits after reporting first data race.

e.g

$ GORACE="log_path=/tmp/race/report strip_path_prefix=/my/go/sources/" go test -race

Excluding Tests with build tag:

// +build !race

package foo

// The test contains a data race. See issue 123.
func TestFoo(t *testing.T) {
 // ...
}

// The test fails under the race detector due to timeouts.
func TestBar(t *testing.T) {
 // ...
}

// The test takes too long under the race detector.
func TestBaz(t *testing.T) {
 // ...
}

[Go] tooling update

https://github.com/dominikh/go-tools

[Go][tricks] API compatibility

Reference:
GopherCon 2019: Jonathan Amsterdam - Detecting Incompatible API Changes
https://www.youtube.com/watch?v=JhdL5AkH-AQ
Slides:
https://about.sourcegraph.com/go/gophercon-2019-detecting-incompatible-api-changes
Spec:
https://go.googlesource.com/exp/+/refs/heads/master/apidiff/README.md
Tools binary:
$ go get golang.org/x/exp/cmd/apidiff  # will be replaced by gorelease
Golang /x/tools/go/packages
https://godoc.org/golang.org/x/tools/go/packages
Golang types package
https://golang.org/pkg/go/types/
Golang constant package
https://golang.org/pkg/go/constant/
Golang gcexportdata package
https://godoc.org/golang.org/x/tools/go/gcexportdata

A comparison of two interface values with identical dynamic types causes a run-time panic if values of that type are not comparable.
This behavior applies not only to direct interface value comparisons but also when comparing arrays of interface values or structs with interface-valued fields.

Go v1.15

Use an unexported, zero-width, non-comparable field
(Function values, Slice values and Map values are not comparable
https://golang.org/ref/spec#Comparison_operators ), to

prevent clients from comparing a struct and shrink binary size.
(Using 0 size of array instead of pure Function/Slice/Map which holds a size
of pointer points to memory):

type Point struct {
 _ [0] func()
 X, Y int
}

[Go] Optimizing Go

Reference:
Lock Laptop CPU performance (linux only) to avoid throttling.
https://github.com/aclements/perflock
High Performance Go Workshop
https://dave.cheney.net/high-performance-go-workshop/gopherchina-2019.html
cmd/compile
https://golang.org/pkg/cmd/compile/
go build -gcflags doc
https://github.com/golang/go/blob/master/src/cmd/compile/internal/ssa/compile.go
SSA:
https://godoc.org/golang.org/x/tools/go/ssa



Which function is inlined or not/heap alloc or not:
$ go build -gcflags='-m -m' *.go
$ go build -gcflags=-d=ssa/check_bce/debug=1 *.go



Prove path (SSA aka. Static Single-Assignment):
$ go build -gcflags=-d=ssa/prove/debug=1 *.go
$ go build -gcflags=-d=ssa/prove/debug=2 *.go


code snippet:
internally, string as key of a map internally converted to []byte:

m = make(map[string]string)
# clear it, it's fast after Go v1.11
for k := range m {
 delete(m, k)
}

Count string size::

n := len([]rune(str))

Tooling, dump SSA to ssa.html
$ GOSSAFUNC=FuncName go build && open ssa.html

[Go][sum up] profiling

Reference:
https://blog.golang.org/profiling-go-programs
https://golang.org/pkg/runtime/pprof/
https://golang.org/pkg/runtime/pprof/#Profile
https://rakyll.org/custom-profiles/
https://rakyll.org/mutexprofile/
https://github.com/google/pprof
https://github.com/golang/go/issues/13841


Definition:
Profiles are only as good as the kernel support used to generate them.

A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation.

Packages can create and maintain their own profiles; the most common use is for tracking resources that must be explicitly closed, such as files or network connections.

A Profile's methods can be called from multiple goroutines simultaneously.

Each Profile has a unique name.

A few profiles are predefined:

• cpu
  CPU profile shows where a program spends its time while actively consuming CPU cycles (opposed to sleep or wait for I/O).
  CPU profile is not enabled by default; use StartCPUProfile to enable it (StopCPUProfile to stop it in the program).
  If the sample turns out 0, means your program isn't run long enough, or you could set the CPU profile rate:
  https://golang.org/pkg/runtime/#SetCPUProfileRate
• goroutine
  Goroutine profile reports stack traces of all current goroutines.
• heap
  Heap profile reports the current live allocations; monitors current memory usage and check for memory leaks.
• allocs
  Allocs profile samples all past memory allocations.
• threadcreate
  Thread creation profile stack traces that led to the creation of new OS threads
• block
  Block profile reports where goroutines block waiting on synchronization primitives (including timer channels).
  Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it.
• mutex
  Mutex profile reports the lock contentions. Useful for debugging mutex contention. Mutex profile is not enabled by default, use runtime.SetMutexProfileFraction to enable it.

Main URL:
http://localhost:4242/debug/pprof/


Sub URL:
http://localhost:4242/debug/pprof/goroutine
http://localhost:4242/debug/pprof/heap
http://localhost:4242/debug/pprof/allocs
http://localhost:4242/debug/pprof/threadcreate
http://localhost:4242/debug/pprof/block
http://localhost:4242/debug/pprof/mutex
http://localhost:4242/debug/pprof/profile
http://localhost:4242/debug/pprof/trace?seconds=5

Use "go tool pprof" to analyze these profiles (lists of stack traces),
Use "go tool trace" to analyze trace endpoint (/debug/pprof/trace?seconds=5)


cmd:
$ go tool pprof --help 

$ go tool pprof http://localhost:4242/debug/pprof/heap
$ go tool pprof http://localhost:4242/debug/pprof/profile?seconds=30

Look at the goroutine blocking profile,
after calling runtime.SetBlockProfileRate in the program:
$ go tool pprof http://localhost:4242/debug/pprof/block

Look at the holders of contended mutexes,
after calling runtime.SetMutexProfileFraction in your program:
$ go tool pprof http://localhost:4242/debug/pprof/mutex

Read heap information
$ go tool pprof -top http://localhost:4242/debug/pprof/heap

Generate png
$ go tool pprof -png http://localhost:4242/debug/pprof/heap > out.png

Profile benchmarks and the contention on your mutexes.
$ go test bench=. -mutexprofile=mutex.out

analyzing the Mutex contention ropfile
$ go tool pprof runtime.test mutex.out

[Go][sum up] go test

Golang test:
https://golang.org/cmd/go/#hdr-Testing_flags
https://godoc.org/github.com/golang/go/src/cmd/go
$ go help testflag
$ go tool cover -help

Go test runs in two different modes:

• local directory mode, occurs when go test is invoked with no package arguments (for example, 'go test' or 'go test -v').
  In this mode, go test compiles the package sources and tests found in the current directory and then runs the resulting test binary.
  In this mode, caching is disabled. After the package test finishes, go test prints a summary line showing the test status ('ok' or 'FAIL'), package name, and elapsed time.
• package list mode, occurs when go test is invoked with explicit package arguments (for example 'go test math', 'go test ./...', and even 'go test .').
  In this mode, go test compiles and tests each of the packages listed on the command line.
  If a package test passes, go test prints only the final 'ok' summary line.
  If a package test fails, go test prints the full test output.
  If invoked with the -bench or -v flag, go test prints the full output even for passing package tests, in order to display the requested benchmark results or verbose logging.

Test cmd:
Provides chatty output for the testing.
$ go test -v

Disable test cache in 'go test' cache mode
$ go test -v -count=1

Go test also supports this flag and reports races.
Use this flag during development to detect the races.
$ go test -race

Filter tests to run by regex and the -run flag.
The following command will only test examples.
-run regexp
    Run only those tests and examples matching the regular expression.
    For tests, the regular expression is split by unbracketed slash (/)
    characters into a sequence of regular expressions, and each part
    of a test's identifier must match the corresponding element in
    the sequence, if any. Note that possible parents of matches are
    run too, so that -run=X/Y matches and runs and reports the result
    of all tests matching X, even those without sub-tests matching Y,
    because it must run them to look for those sub-tests.
$ go test -run=Example

This flag allows you to delegate some work to an
external program from the Go tool.
$ go test -exec


Converage cmd:
$ go test -cover

-coverprofile flag automatically sets -cover to enable coverage analysis.
$ go test -coverprofile=coverage.out

Counting statement execution for a standard package, the fmt formatting package.
$ go test -covermode=count -coverprofile=count.out fmt
$ go tool cover -func=coverage.out
$ go tool cover -html=coverage.out


go test cmd:

• The 'go test' runs "*_test.go" files corresponding to the package under test.
• Files whose names begin with "_" (including "_test.go") or "." are ignored.
• Test files that declare a package with the suffix "_test" will be compiled as a separate package, and then linked and run with the main test binary.
• The go tool will ignore a directory named "testdata", making it available to hold ancillary data needed by the tests.
• 'go test' runs 'go vet' on the package and its test source files to identify significant problems.
• To disable the running of go vet, use the -vet=off flag.
• All test output and summary lines are printed to the go command's standard output, even if the test printed them to its own standard error.
  (The go command's standard error is reserved for printing errors building the tests.)

Test function:
func TestXxx(t *testing.T) { ... }

Benchmark function:
func BenchmarkXxx(b *testing.B) { ... }

Example functions:

• An example function is similar to a test function but, instead of using *testing.T to report success or failure, prints output to os.Stdout.
• If the last comment in the function starts with "Output:" then the output is compared exactly against the comment.
• If the last comment begins with "Unordered output:" then the output is compared to the comment, however the order of the lines is ignored.
• An example with no such comment is compiled but not executed.
• An example with no text after "Output:" is compiled, executed, and expected to produce no output.
• Godoc displays the body of ExampleXxx to demonstrate the use of the function, constant, or variable Xxx.
• An example of a method M with receiver type T or *T is named ExampleT_M.
• There may be multiple examples for a given function, constant, or variable, distinguished by a trailing _xxx,where xxx is a suffix not beginning with an upper case letter.
• The entire test file is presented as the example when it contains a single example function, at least one other function, type, variable, or constant declaration, and no test or benchmark functions.

func ExamplePrintln() {
                Println("The output of\nthis example.")
                // Output: The output of
                // this example.
}

func ExamplePerm() {
                for _, value := range Perm(4) {
                        fmt.Println(value)
                }

                // Unordered output: 4
                // 2
                // 1
                // 3
                // 0
}


Coverage:
Reference:
https://blog.golang.org/cover
http://gcc.gnu.org/onlinedocs/gcc/Gcov.html

tl;dr
Golang converage uses code injection to rewrite the source code base on blocks for testing code coverage.


Excerpts from official document:
The usual way to compute test coverage is to instrument the binary.
For instance, the GNU gcov program sets breakpoints at branches executed by the binary.
As each branch executes, the break-point is cleared and the target statements of the branch are marked as 'covered'.

This approach is successful and widely used.
An early test coverage tool for Go even worked the same way.
But it has problems. It is difficult to implement, as analysis of the execution of binaries is challenging.
It also requires a reliable way of tying the execution trace back to the source code, which can also be difficult, as any user of a source-level debugger can attest.

Problems there include inaccurate debugging information and issues such as in-lined functions complicating the analysis.

Most important, this approach is very non-portable.
It needs to be done afresh for every architecture,
and to some extent for every operating system since debugging support varies greatly from system to system.


Another approach:
For the new test coverage tool for Go, we took a different approach that avoids dynamic debugging.
The idea is simple: Rewrite the package's source code before compilation to add instrumentation,compile and run the modified source, and dump the statistics.
The rewriting is easy to arrange because the go command controls the flow from source to test to execution.


How it works?
When test coverage is enabled, go test runs the "cover" tool,
a separate program included with the distribution, to rewrite the source code before compilation.


Original code:

package size

func Size(a int) string {
    switch {
    case a < 0:
        return "negative"
    case a == 0:
        return "zero"
    case a < 10:
        return "small"
    case a < 100:
        return "big"
    case a < 1000:
        return "huge"
    }
    return "enormous"
}

Modified code:

func Size(a int) string {
    GoCover.Count[0] = 1
    switch {
    case a < 0:
        GoCover.Count[2] = 1
        return "negative"
    case a == 0:
        GoCover.Count[3] = 1
        return "zero"
    case a < 10:
        GoCover.Count[4] = 1
        return "small"
    case a < 100:
        GoCover.Count[5] = 1
        return "big"
    case a < 1000:
        GoCover.Count[6] = 1
        return "huge"
    }
    GoCover.Count[1] = 1
    return "enormous"
}

Although that annotating assignment might look expensive, it compiles to a single "move" instruction. Its run-time overhead is therefore modest, adding only about 3% when running a typical (more realistic) test.

That makes it reasonable to include test coverage as part of the standard development pipeline.


Heat maps:
A big advantage of this source-level approach to test coverage is that it's easy to instrument the code in different ways.
For instance, we can ask not only whether a statement has been executed, but how many times.


Basic blocks:
Coverage annotations is demarcated by branches in the program.
It's hard to do that by rewriting the source, though, since the branches don't appear explicitly in the source.

What the coverage annotation does is instrument blocks, which are typically bounded by brace brackets.
e.g
f() && g()

Coverage has no attempt to separately instrument the calls to f and g, regardless of the facts it will always look like they both ran the same number of times, the number of times f ran.

[Go] special directory recognized by go cmd

• internal
  https://golang.org/doc/go1.4#internalpackages
  Place source files in a directory named internal or in a sub-directory of a directory named internal.
  When the go command sees an import of a package with internal in its path, it verifies that the package doing the import is within the tree rooted at the parent of the internal directory.
  e.g:
  a package .../a/b/c/internal/d/e/f can be imported only by code in the directory tree rooted at .../a/b/c.
  It cannot be imported by code in .../a/b/g or in any other repository.
• testdata
  Go tool ignores directory named "testdata", making it available to hold ancillary data needed by the tests.
• vendor
  When the main module contains a top-level vendor directory and its go.mod file specifies go 1.14 or higher, the go command now defaults to -mod=vendor
  When -mod=vendor is set (explicitly or by default), the go command now verifies that the main module's vendor/modules.txt file is consistent with its go.mod file.
  By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification).
  
  Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.

[Go] useful tool chain commands

go build

Lists all the commands go build invokes.
$ go build -x


Used to pass flags to the Go compiler.
go tool compile -help lists all the flags that can
be passed to the compiler.
$ go build -gcflags

go test

Provides chatty output for the testing.
$ go test -v


Go test also supports this flag and reports races.
Use this flag during development to detect the races.
$ go test -race


Filter tests to run by regex and the -run flag.
The following command will only test examples.
$ go test -run=Example


Outputs a cover profile as testing a package,
then use go tool to visualize them on a browser.
$ go test -coverprofile=c.out && go tool cover -html=c.out


This flag allows you to delegate some work to an
external program from the Go tool.
$ go test -exec

go get

If an argument names a module but not a package (because there is no Go source code in the module's root directory), then the install step is skipped for that argument, instead of causing a build failure.
For example 'go get golang.org/x/perf' succeeds even though there is no code corresponding to that import path.

-u forces the tool to sync with the latest version of the repo.
go get grab version in this precedence:

1. latest tagged release version, such as v0.4.5 or v1.2.3
2. latest tagged pre-release version, such as v0.0.1-pre1
3. latest known commit
4. This default version selection can be overridden by adding an @version suffix to the package argument, as in 'go get golang.org/x/text@v0.3.0'
5. For modules stored in source control repositories, the version suffix can also be a commit hash, branch identifier, or other syntax known to the source control system, as in 'go get golang.org/x/text@master'
6. The version suffix @none indicates that the dependency should be removed entirely, downgrading or removing modules depending on it as needed.
7. The version suffix @latest explicitly requests the latest minor release of the module named by the given path.
   The suffix @upgrade is like @latest but will not downgrade a module if it is already required at a revision or pre-release version newer than the latest released version.
   The suffix @patch requests the latest patch release: the latest released version with the same major and minor version numbers as the currently required version.
   Like @upgrade, @patch will not downgrade a module already required at a newer version.
   If the path is not already required, @upgrade and @patch are equivalent to @latest.
   
   Note that branches with names that overlap with other module query syntax cannot be selected explicitly.
   For example, the suffix @v2 means the latest version starting with v2, not the branch named v2.

$ go help get
$ go get -u

The -t flag instructs get to consider modules needed to build tests of packages specified on the command line.
$ go get -ut


Just want to clone a repo to your GOPATH and skip
the building and installation phase, use -d.
$ go get -d golang.org/x/oauth2/...


If your package has additional dependencies for tests,
-t will allow you to download them during go-get.
If you don't pass -t, go get will only download the
dependencies for your non-test code.
$ go get -t

go list

https://dave.cheney.net/2014/09/14/go-list-your-swiss-army-knife
$ go list -f

go doc

Show doc of the package/function
$ go doc 'package name'

go tool

Show supported arch
$ go tool dist list -json
https://github.com/golang/go/blob/master/src/go/build/syslist.go
https://golang.org/doc/install/source#environment

others

Since Golang produces elf binary, we could strip it as well like C/C++ linker
$ ld -s

strip (man strip) works well in elf Golang binary.
$ strip -s go_binary

A Golang standard way:
$ go build -ldflags "-w"  // Omit the DWARF symbol table.
$ go build -ldflags "-s"   // Not work on Mac, Omit the symbol table and debug information.



Reference:
https://golang.org/cmd/link/

Use UPX to boost shrinking the binaray size as used for other elf:
$ upx go_binaray  // not good for startup performance
$ upx --ultra-brute go_binary

Print escape analysis
$ go build -gcflags '-m'