[Go] build Go into shared object with debug mode

$ go build -o awesome.so -buildmode=c-shared -gccgoflags=-g myprog.go 

[Go] CGO_LDFLAGS_ALLOW env

https://github.com/golang/go/issues/33521

Add linker flags to the CGO_LDFLAGS_ALLOW environment variable for building cgo.

e.g

-Wl,--whole-archive -lmy_lib

[Go] Build constraints

 https://golang.org/cmd/go/#hdr-Build_constraints

[Go] tooling update (compile time optimization, linker, compiler)

Reference:
https://rakyll.org/go-tool-flags/
https://golang.org/cmd/go/
https://github.com/golang/go/wiki/CompilerOptimizations
https://www.alexedwards.net/blog/an-overview-of-go-tooling#building-an-executable



Lists all the commands go build invokes
$ go build -x

Show flags that can be passed to compiler
$ go tool compile -help

e.g. to disable compiler optimizations and inlining, you can use the following the gcflags.
$ go build -gcflags="-N -l"


Disable bounds checking
$ go build -gcflags="-B"

Show compiler debug list:
$ go tool compile -d help 2>&1|bat

Show compiler -d ssa list:
compile: PhaseOptions usage:
$ go tool compile -d=ssa/<phase>/<flag>[=<value>|<function_name>]
$ go tool compile -d ssa/help 2>&1|bat

e.g If we want to check which line has the  "Bounds Check Elimination"
applied:
https://golang.org/src/cmd/compile/internal/ssa/checkbce.go
$ go build -gcflags="-d=ssa/check_bce/debug=1"



Linker help:
$ go tool link -help

"burn in" a (string) value to a specific variable in your application:
$ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo .

Strip debugging information from the binary.
$ go build -ldflags="-s -w" -o=/tmp/foo .  # Strip debug information from the binary



Escape analysis:
Gc compiler does global escape analysis across function and package boundaries. However, there are lots of cases where it gives up.
For example, anything assigned to any kind of indirection (*p = ...) is considered escaped.
Other things that can inhibit analysis are: function calls, package boundaries, slice literals, sub-slicing and indexing, etc.
Full rules are too complex to describe, so check the -m output.
$ go build -gcflags -m


Function Inlining:
Only short and simple functions are inlined.
To be inlined a function must contain less than ~40 expressions and does not contain complex things like loops, labels, closures, panic's, recover's, select's, switch'es, etc.


Non-scannable objects:
Garbage collector does not scan underlying buffers of slices, channels and maps when element type does not contain pointers (both key and value for maps)(this is called tracing).

This allows to hold large data sets in memory without paying high price during garbage collection.

For example, the following map won't visibly affect GC time:

type Key [64]byte // SHA-512 hash
type Value struct {
 Name      [32]byte
 Balance   uint64
 Timestamp int64
}
m := make(map[Key]Value, 1e8)

[Go] compile commands

List arguments for -gcflags:
$ go tool compile -help

i.e
$ go build -gcflags="-N -l"

List all commands go build do:
$ go build -x

[Go] profiling trick

Tip: Use the built-in go test flags if you need to profile your benchmarks:

-test.cpuprofile
-test.memprofile
-test.mutexprofile
-test.blockprofile

$ go test -bench=BenchmarkX -test.cpuprofile profile.out
$ go tool pprof profile.out

[Go] Optimizing Go

Reference:
Lock Laptop CPU performance (linux only) to avoid throttling.
https://github.com/aclements/perflock
High Performance Go Workshop
https://dave.cheney.net/high-performance-go-workshop/gopherchina-2019.html
cmd/compile
https://golang.org/pkg/cmd/compile/
go build -gcflags doc
https://github.com/golang/go/blob/master/src/cmd/compile/internal/ssa/compile.go
SSA:
https://godoc.org/golang.org/x/tools/go/ssa



Which function is inlined or not/heap alloc or not:
$ go build -gcflags='-m -m' *.go
$ go build -gcflags=-d=ssa/check_bce/debug=1 *.go



Prove path (SSA aka. Static Single-Assignment):
$ go build -gcflags=-d=ssa/prove/debug=1 *.go
$ go build -gcflags=-d=ssa/prove/debug=2 *.go


code snippet:
internally, string as key of a map internally converted to []byte:

m = make(map[string]string)
# clear it, it's fast after Go v1.11
for k := range m {
 delete(m, k)
}

Count string size::

n := len([]rune(str))

Tooling, dump SSA to ssa.html
$ GOSSAFUNC=FuncName go build && open ssa.html

[Go] escape analysis discussion

Notes about Golang's escape analysis from [golang-dev] forum discussion and Matthew Dempsky's work.

Reference:
Go Escape Analysis Flaws - Dmitry Vyukov, @dvyukov
https://groups.google.com/d/msg/golang-dev/jln8MwFpATc/xwBXzc9cEAAJ
https://github.com/golang/go/issues/23109#issuecomment-446342642

func call(f func()) {
    f()
}

func g() {
    var x int
    
    call(
        func() { x = 1 },  // No escape since 'x' is not used by caller
    )
}

Originally, closures always stored addresses of referenced variables.

At some point an optimization was added that captured variables that are not later modified by the outer function have their values, not their addresses, recorded in the closure.
Please refer to this question on stackoverflow, which is quite interesting due to this side-effect with golang's closure behavior:
https://stackoverflow.com/questions/42162879/mutex-within-loop-leads-to-unexpected-output

package main

import (
    "fmt"
    "sync"
)

func main() {
    mutex := new(sync.Mutex)

    for i := 1; i < 5; i++ {
        for j := 1; j < 5; j++ {
            mutex.Lock()
            go func() {
                fmt.Printf("%d + %d = %d\n", i, j, j+i)
                mutex.Unlock()
            }()
        }
    }
}

---
Result:
1 + 2 = 3
1 + 3 = 4
1 + 4 = 5
2 + 5 = 7
2 + 2 = 4
2 + 3 = 5
2 + 4 = 6
3 + 5 = 8
3 + 2 = 5
3 + 3 = 6
3 + 4 = 7
4 + 5 = 9
4 + 2 = 6
4 + 3 = 7
4 + 4 = 8

---

If not for this optimization, the same problems that force heap allocation above would force heap allocation even for:

func call(f func() int) { 
    f()
}

func g() {
    var x int
    call(func() int { return x })
}

If we tweak that example to modify x after the closure creation, that will disable the closure-value optimization:

func call(f func() int) {
    f()
}

func g() {
    var x int
    call(func() int { return x })  // 'x' escaped to heap since 'x++' from the caller
    x++
}

if the closure itself escapes, then the addresses of the variables are understood to escape too:

func call(f func() *int) *int { 
    f()
    return nil
}

func h() { 
    var y int
    call(
    func() *int { return &y }
    )
}

Or even this:

func h() {
    var y int
    _ = func() *int { return &y }()
}

Both of them decide that &y escapes, and there isn't even a call to analyze in the second.

Golang currently treats values returned by functions (including closures) as escaping to the heap, so there's really no point in worrying that f() might return a value from within the closure.

package p

//go:noinline
func call1(f func() error) error {
 // Leaks *f to result.
 return f()
}

func F1() error {
 y := new(int)
 return call1(func() error {
  y = nil
  return nil
 })
}


//go:noinline
func call2(f func() error) error {
 // No param leakage.
 f()
 return nil
}

func F2() error {
 y := new(int)
 return call2(func() error {
  y = nil
  return nil
 })
}

There's a discussion about interface as function parameter type which causing variable passing in heap allocated.
Reference:
https://www.reddit.com/r/golang/comments/9f9pu8/do_blank_interface_values_escape_to_the_heap/
https://www.reddit.com/r/golang/comments/badeql/golang_memory_escape_analysis_is_naive/
https://stackoverflow.com/a/44699604

The above stackoverflow answer is incorrect, it's not the interface{} being allocated on the heap but the variable it's taken should be allocated on the heap which interface{}'s second pointer will points to it.
(The first pointer will point to type information, or the word will contains the type information, depends on the implement)

[Go] useful tool chain commands

go build

Lists all the commands go build invokes.
$ go build -x


Used to pass flags to the Go compiler.
go tool compile -help lists all the flags that can
be passed to the compiler.
$ go build -gcflags

go test

Provides chatty output for the testing.
$ go test -v


Go test also supports this flag and reports races.
Use this flag during development to detect the races.
$ go test -race


Filter tests to run by regex and the -run flag.
The following command will only test examples.
$ go test -run=Example


Outputs a cover profile as testing a package,
then use go tool to visualize them on a browser.
$ go test -coverprofile=c.out && go tool cover -html=c.out


This flag allows you to delegate some work to an
external program from the Go tool.
$ go test -exec

go get

If an argument names a module but not a package (because there is no Go source code in the module's root directory), then the install step is skipped for that argument, instead of causing a build failure.
For example 'go get golang.org/x/perf' succeeds even though there is no code corresponding to that import path.

-u forces the tool to sync with the latest version of the repo.
go get grab version in this precedence:

1. latest tagged release version, such as v0.4.5 or v1.2.3
2. latest tagged pre-release version, such as v0.0.1-pre1
3. latest known commit
4. This default version selection can be overridden by adding an @version suffix to the package argument, as in 'go get golang.org/x/text@v0.3.0'
5. For modules stored in source control repositories, the version suffix can also be a commit hash, branch identifier, or other syntax known to the source control system, as in 'go get golang.org/x/text@master'
6. The version suffix @none indicates that the dependency should be removed entirely, downgrading or removing modules depending on it as needed.
7. The version suffix @latest explicitly requests the latest minor release of the module named by the given path.
   The suffix @upgrade is like @latest but will not downgrade a module if it is already required at a revision or pre-release version newer than the latest released version.
   The suffix @patch requests the latest patch release: the latest released version with the same major and minor version numbers as the currently required version.
   Like @upgrade, @patch will not downgrade a module already required at a newer version.
   If the path is not already required, @upgrade and @patch are equivalent to @latest.
   
   Note that branches with names that overlap with other module query syntax cannot be selected explicitly.
   For example, the suffix @v2 means the latest version starting with v2, not the branch named v2.

$ go help get
$ go get -u

The -t flag instructs get to consider modules needed to build tests of packages specified on the command line.
$ go get -ut


Just want to clone a repo to your GOPATH and skip
the building and installation phase, use -d.
$ go get -d golang.org/x/oauth2/...


If your package has additional dependencies for tests,
-t will allow you to download them during go-get.
If you don't pass -t, go get will only download the
dependencies for your non-test code.
$ go get -t

go list

https://dave.cheney.net/2014/09/14/go-list-your-swiss-army-knife
$ go list -f

go doc

Show doc of the package/function
$ go doc 'package name'

go tool

Show supported arch
$ go tool dist list -json
https://github.com/golang/go/blob/master/src/go/build/syslist.go
https://golang.org/doc/install/source#environment

others

Since Golang produces elf binary, we could strip it as well like C/C++ linker
$ ld -s

strip (man strip) works well in elf Golang binary.
$ strip -s go_binary

A Golang standard way:
$ go build -ldflags "-w"  // Omit the DWARF symbol table.
$ go build -ldflags "-s"   // Not work on Mac, Omit the symbol table and debug information.



Reference:
https://golang.org/cmd/link/

Use UPX to boost shrinking the binaray size as used for other elf:
$ upx go_binaray  // not good for startup performance
$ upx --ultra-brute go_binary

Print escape analysis
$ go build -gcflags '-m'

[Go] Execution modes

Reference:

Go Execution Modes - Ian Lance Taylor  https://goo.gl/mrzwCz
https://golang.org/cmd/link/
https://golang.org/cmd/go/#hdr-Build_modes
https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies
https://github.com/golang/go/issues/18246
https://blog.ksub.org/bytes/2017/02/12/exploring-shared-objects-in-go/

Legacy Go(v1.4.0) support 3 execution modes:

1. A statically linked Go binary.
   Default for program does not import the net or os/user packages and does not use cgo or SWIG.
2. A dynamically linked Go binary.
   Default for program imports the net or os/user packages and does not otherwise use cgo or SWIG
3. A Go binary linked with arbitrary non-Go code.
   Default for program uses cgo or SWIG.
   The interface between Go and non-Go code is a C style API
   (SWIG permits calling between C++ and Go, but this is implemented using a C style API).
   Can be selected with -ldflags -linkmode=external. (obsolete)

API styles

1. Go
2. C, i.e extern "C"

Design in mind

• Go code that is combined into a single executable image must be built with the same version of the Go toolchain.
• We require further that if any Go package appears more than once in the executable image,
  it must be built from the same source code. (Same as C++'s header file)
• This restriction comes from both ways: Golang <-> C

Go runtime:

• All Go code shares a single runtime. 
• All Go code uses the same memory allocator.
• The same goroutine scheduler.
• In general acts as though it were linked into a single Go program.

New execution modes:

• Go code linked into, and called from, a non-Go program. (Go v.1.5.0)
  Go code acts as a library that may be called by a non-Go program.
  A single Go library will be an archive, a .a file on Unix,
  or as a shared library, a .so file on Unix, providing a C style API.
  This mode supports people who must work with large existing programs, especially in C/C++.
  It permits them to extend those existing programs with new packages written in Go.
  i.e the binary code is sheer an ELF format.
• Go code linked into a shared library loaded as a plugin by a program (Go or non-Go) that supports a C style plugin API. (Go v1.6.0)
  i.e dlopen/dlmopen (dlmopen appears in glibc 2.3.4 for linker's namespace)
  https://sourceware.org/glibc/wiki/LinkerNamespaces
  Golang binary code in .so ELF format can be dlopened by C/C++.
• Go code linked into a shared library loaded as a plugin by a Go program that supports a general Go style plugin API.
  A shared library can be dlopened by Golang.
  https://golang.org/pkg/plugin/
• A Go program that uses a plugin interface, either C style or Go style, where plugins are implemented as shared libraries.
  Golang can dlopen any C/C++ .so libraries.
• Building a Go package, or collection of packages, as a shared library that may be linked into a Go program. (Go v1.6.0)
  Single/Multiple Golang packages build into single shared library, which can be linked to other Golang program.
  Updating the Go run-time to a new version requires rebuilding all Go programs that use it.
• A Go program built as a PIE--a Position Independent Executable. (Go v1.6.0)
  Go program is built as usual, but the resulting executable is position-independent, and may be relocated at run time.
  (i.e -fPIC in C/C++)

Go tool flags:

• -buildmode

archive:
Default build mode for a package that is not main.
Builds the package into a .a file.

c-archive: (Go v1.5.0)
Requires a main package, but the main function is ignored (init functions are run as usual).
Build the main package, plus all packages that it imports, into a single C archive file.
The only callable symbols will be those functions marked as exported.

shared: (Go v1.6.0)
Combine all the listed packages into a single shared library that will be used when building with the -linkshared option.

c-shared: (Go v1.6.0)
Requires a main package as for -buildmode=c-archive
Build the main package, plus all packages that it imports, into a single C shared library.
The only callable symbols will be those functions marked as exported.

plugin:
Requires a main package as for -buildmode=c-archive
Build the main package, plus all packages that it imports, into a single shared library that may be loaded as a run-time plugin.

exe:
Default build mode for a package named main.

pie: (Go v1.6.0)
This is like -buildmode=exe , but it builds a Position Independent Executable.

-linkshared:
Directs to Go tool to use link against shared libraries when available.
When no shared library is available for some imported package, the ordinary archive will be used instead.
The -linkshared flag may be used with
-buildmode=shared, exe, pie
As the name suggests, -linkshared is NOT used for -buildmode=archive or c-archive

Hands on:

Build Golang's std library into shared library.
$ go install -buildmode=shared std

Build the shared code:
$ go install -buildmode=shared -linkshared github.com/your/lib/code

Use the shared library:
$ go install -linkshared github.com/your/main/code  // which imports "github.com/your/lib/code"

With a SONAME, and beware to put the compiled binary into the POSIX SONAME location/version.
$ go install \
-ldflags '-extldflags -Wl,-soname,libpikachu.so.0' \
-buildmode=shared \
-linkshared \
github.com/your/lib/code

[Go] build flag

Check which local values will escape to heap at run time.
$ go build -gcflags -m  

Compile to different OS/ARCH.
$ export GOOS=linux; export amd64=amd64; go build

Show supported arch
$ go tool dist list -json https://github.com/golang/go/blob/master/src/go/build/syslist.go https://golang.org/doc/install/source#environment