Ataraxia through Epoché: [Go] [go module] wrap up

Reference:

https://blog.golang.org/using-go-modules
https://blog.golang.org/migrating-to-go-modules
https://blog.golang.org/publishing-go-modules
https://blog.golang.org/v2-go-modules
https://blog.golang.org/module-compatibility

Definition:

Modules:

A module is a collection of related Go packages that are versioned together as a single unit.
Modules must be semantically versioned in the form v(major).(minor).(patch), such as v0.1.0, v1.2.3, or v3.0.1.
pre-release version
The versioning leading v is required.
The go command automatically updates go.mod each time it uses the module graph, to make sure go.mod always accurately reflects reality and is properly formatted.
Because the module graph defines the meaning of import statements, any commands that load packages also use and therefore update go.mod, including go build, go get, go install, go list, go test, go mod graph, go mod tidy, and go mod why.
$ go mod why -m rsc.io/binaryregexp

Commands outside modules:

When module-aware mode is enabled explicitly (by setting GO111MODULE=on), most module commands have more limited functionality if no go.mod file is present.

For example, go build, go run, and other build commands can only build packages in the standard library and packages specified as .go files on the command line.

Previously, the go command would resolve each package path to the latest version of a module but would not record the module path or version. This resulted in slow, non-reproducible builds.

go get continues to work as before, as do go mod download and go list -m with explicit versions.
$ go list -m // list module instead of packages

Default Proxy:
https://proxy.golang.org

Default checksum db:
https://sum.golang.org

ENV: (reference: https://golang.org/doc/go1.13#modules)

$GOPRIVATE (show: $ go env GOPRIVATE)
controls which modules the go command considers to be private (not available publicly) and should therefore not use the proxy or checksum database. The variable is a comma-separated list of glob patterns (in the syntax of Go's path.Match) of module path prefixes.
$GOMOD (show: $ go env GOMOD)
The absolute path to the go.mod of the main module,
or the empty string if not using modules.
$GOPROXY (show: $ go env GOPROXY)
URL of Go module proxy. See 'go help goproxy'.
$GOTOOLDIR (show: $ go env GOTOOLDIR)
The directory where the go tools (compile, cover, doc, etc...) are installed.
$GOHOSTOS (show: $ go env GOHOSTOS)
The operating system (GOOS) of the Go toolchain binaries.
$GOHOSTARCH (show: $ go env GOHOSTARCH)
The architecture (GOARCH) of the Go toolchain binaries.
$GOEXE (show: $ go env GOEXE)
The executable file name suffix (".exe" on Windows, "" on other systems).
$GOTMPDIR (show: $ go env GOTMPDIR)
The directory where the go command will write temporary source files, packages, and binaries.
$GOINSECURE is a new environment variable that instructs the go command to not require an HTTPS connection, and to skip certificate validation, when fetching certain modules directly from their origins.
Like the existing GOPRIVATE variable, the value of GOINSECURE is a comma-separated list of glob patterns.

go.mod example:

--
module my/thing
go 1.12
require other/thing v1.0.2
require new/thing/v2 v2.3.4
exclude old/thing v1.2.3
replace bad/thing v1.4.5 => good/thing v1.4.5
--

verbs explain:
--
module, to define the module path;
go, to set the expected language version;
require, to require a particular module at a given version or later;
exclude, to exclude a particular module version from use; and
replace, to replace a module version with a different module version.
--

like go import's syntax, verbs can be coded as:

require (

new/thing v2.3.4

old/thing v1.2.3

)

GOPATH and Modules:

When using modules, GOPATH is no longer used for resolving imports.

However, it is still used to store downloaded source code (in GOPATH/pkg/mod)

and compiled commands (in GOPATH/bin).

'go get' checks out or updates a git repository also updates submodules.

Module proxy protocol:

If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems.
Setting GOPROXY to "off" disallows downloading modules from any source.
Otherwise, GOPROXY is expected to be the URL of a module proxy, in which case the go command will fetch all modules from that proxy.
No matter the source of the modules, downloaded modules must match existing entries in go.sum (see 'go help modules' for discussion of verification).

golang build cache path:

https://golang.org/cmd/go/#hdr-Build_and_test_caching

The go command also caches successful package test results.

See 'go help test' for details.

GOCACHE, defaulting to $HOME/.cache/go-build, we'll set to "/tmp/on"

Print current cache directory
$ go env GOCACHE

Setting the environment variable GOCACHE=off will cause go commands that write to the cache to fail.
$ GOCACHE=off

Clean all cache
$ go clean -cache

Removes all cached test results
$ go clean -testcache

The GODEBUG environment variable can enable printing of debugging information about the state of the cache:

GODEBUG=gocacheverify=1 causes the go command to bypass the use of any cache entries and instead rebuild everything and check that the results match existing cache entries.
GODEBUG=gocachehash=1 causes the go command to print the inputs for all of the content hashes it uses to construct cache lookup keys. The output is voluminous but can be useful for debugging the cache.
GODEBUG=gocachetest=1 causes the go command to print details of its decisions about whether to reuse a cached test result.

golang module cache path:

$ GOPATH/pkg/mod
We can remove the cached package manually here.

Clean downloaded cache:

$ go clean -modcache
reference: https://tip.golang.org/cmd/go/#hdr-Remove_object_files_and_cached_files

Backward compatible:

Set GO111MODULE=on if want to use go module within $GOPATH

Modules and vendoring

When the main module contains a top-level vendor directory and its go.mod file specifies go 1.14 or higher, the go command now defaults to -mod=vendor

When -mod=vendor is set (explicitly or by default), the go command now verifies that the main module's vendor/modules.txt file is consistent with its go.mod file.

By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification).

-mod=mod causes the go command to instead load modules from the module cache.

To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -mod=vendor'. (This is not recommended)
i.e put all dependencies under:
vendor/ folder

Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.

Readonly Modules

If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod described above.

When -mod=readonly is set, go commands will no longer fail due to a missing go directive or an erroneous // indirect comment.

-mod=readonly is now set by default when the go.mod file is read-only and no top-level vendor directory is present.

Instead, it fails when any changes to go.mod are needed. This setting is most useful to check that go.mod does not need updates, such as in a continuous integration and testing system.

The "go get" command remains permitted to update go.mod even with -mod=readonly, and the "go mod" commands do not take the -mod flag (or any other build flags).

Cache Control

-modcacherw is a new flag that instructs the go command to leave newly-created directories in the module cache at their default permissions rather than making them read-only.

The use of this flag makes it more likely that tests or other tools will accidentally add files not included in the module's verified checksum. However, it allows the use of rm -rf (instead of go clean -modcache) to remove the module cache.

Use other go.mod file

-modfile=file is a new flag that instructs the go command to read (and possibly write) an alternate go.mod file instead of the one in the module root directory.

A file named go.mod must still be present in order to determine the module root directory, but it is not accessed.

When -modfile is specified, an alternate go.sum file is also used: its path is derived from the -modfile flag by trimming the .mod extension and appending .sum.

Fresh up your git tag skills:

go module relies on git tag to do versioning.

Two kinds of git tags:

https://git-scm.com/book/en/v2/Git-Basics-Tagging
Annotated tag, as a commit object with tagger's information:
$ git tag -a v1.0.1 -m "version 1.0.1"
$ git show v1.0.1

Lightweight Tags, no information attached:
$ git tag v1.0.1-lw

Push local tags to remote:
$ git push origin <tagname>
push all tags:
$ git push origin --tags

Checking out Tags:
Will be in detached state. i.e not under current branch's HEAD commit.
$ git checkout 2.0.0

or checkout tag into branch: (better, thus you could do some changes and commit)
$ git checkout -b version2 v2.0.0

delete local tag:
https://stackoverflow.com/questions/5480258/how-to-delete-a-git-remote-tag
$ git tag -d v1.4

delete remote tag:
$ git push origin :refs/tags/v1.4
or
$ git push origin :tagname
or
$ git push --delete origin tagname

Go module follow semantic versioning:

https://semver.org/
Go uses repository tags to lookup for versions.

Only version >= 2 will have different import path.
i.e
github.com/buddhavs/module_test <= module base path, which contains 'util' package
import "github.com/buddhavs/module_test/util" // for version 0 or 1
import "github.com/buddhavs/module_test/v2/util" // for version 2
import "github.com/buddhavs/module_test/v3/util" // for version 3

Steps for init new module:

1.
$ go mod init github.com/buddhavs/module_test
Generates 'go.mod'
go.mod contains current projects path/version
i.e
module github.com/buddhavs/module_test <= module base path, which contains 'util' package

2.
Import current module needed by the application.
$ go mod tidy
Generates updated go.mod and new go.sum

go commands other than go mod tidy no longer remove a require directive that specifies a version of an indirect dependency that is already implied by other (transitive) dependencies of the main module.
go commands other than go mod tidy no longer edit the go.mod file if the changes are only cosmetic.

3.
Series of git init commands:
$ git init
$ git add * // includes go.mod
$ git commit -am "1 commit"
$ git push -u origin master

4.
Anyone who uses the package can now do follow commands as before:
$ go get github.com/buddhavs/module_test

5.
$ git tag v1.0.0 // tag the version

6.
$ git checkout -b v1
$ git push -u origin v1 --tags

Steps for create new minor/patch version module:

$ git commit -m "patch version."
$ git tag v1.0.1
$ git push --tags origin v1.0.1

Steps for create new major version module:

$ git commit -m "v2"
$ git checkout -b v2 # Recommended.
$ echo "module github.com/buddhavs/module_test/v2" > go.mod
$ git commit go.mod -m "Bump to v2"
$ git tag v2.0.0
$ git push --tags origin v2.0.0 # or master if don't have a branch

Consume modules:

1.
If using module v0 or v1 with minor or patch, the import path is as usual.
i.e

package main
import mt "github.com/buddhavs/module_test"
func main() {
  mt.Run()
}

2.
If using module > v1, import path with version number.
i.e

package main
import mt "github.com/buddhavs/module_test/v2"  // it's mentioned in the module's go.mod
func main() {
  mt.Run()
}

3.
go build will use the latest version of the main version of the module.

Consume modules from local file path:

https://github.com/golang/go/wiki/Modules#can-i-work-entirely-outside-of-vcs-on-my-local-filesystem
Modify go.mod with 'replace' key word:
go.mod:
--
module module_main
require module_test/v2 v2.0.1
replace module_test/v2 => ../module_test
--

Cmd:

list current project's needed modules:
$ go list -m all
Fails explicitly if -mod=vendor is set and information is requested for a module not mentioned in vendor/modules.txt

find updated modules:
$ go list -m -u all

get all modules source code only:
$ go get -d

upgrade all modules:
$ go get -u

get the latest version:
$ go get -u github.com/vsdmars/project@latest

show module versions:
$ go list -m rsc.io/sampler

use 'go get' to upgrade/downgrade modules:
$ go get -u // use the latest minor or patch releases (i.e. it would update from 1.0.0 to, say, 1.0.1 or, if available, 1.1.0)

$ go get -u=patch // use the latest patch releases (i.e., would update to 1.0.1 but not to 1.1.0)

$ go get package@version //update to a specific version (say, github.com/robteix/testmod@v1.0.1)

use local changes:
$ go mod edit -replace 'rsc.io/quote=../quote'

use remote version:
$ go mod edit -replace 'rsc.io/quote=github.com/myitcv/london-gophers-quote-fork@v0.0.0-myfork' // @TAG

Go Clean:

https://manpages.debian.org/testing/golang-go/go-clean.1.en.html
# -r clean to be applied recursively to all the dependencies of the packages named by the import paths.
$ go clean -r

Go Module Proxy

Download Protocol

Reference: https://docs.gomods.io/intro/protocol/

go command determines needing a module, it first looks at the local cache
(under $GOPATH/pkg/mods ). If it can’t find the files there, it then goes from the network.

To control what files go can download, setting the GOPROXY environment variable to point to our proxy’s URL.

For instance:
$ export GOPROXY=http://gproxy.association.local:8080

The proxy is a web server that responds to the module download protocol, which is an API to query and fetch modules. The web server may serve static files.

( figure from https://roberto.selbach.ca/go-proxies/ )

We can set GOPROXY to local directory which has the correct directory layout of the API protocol also works:
$ export GOPROXY=file://home/robteix/devel/go-proxy-blog
Project Athens, a go module proxy project: https://github.com/gomods/athens

Go 1.12

https://golang.org/doc/go1.12#modules
Go Install doesn't work anymore if GO111MODULE=ON
There will be a work around on golang v1.12
https://github.com/golang/go/issues/24250

$ go run x.go
$ go get rsc.io/2fa@v1.1.0
Both can now(go v.1.12) operate in GO111MODULE=on mode without an explicit go.mod file.

Commands such as

go get
go list
go mod download

behave as if in a module with initially-empty requirements.

In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).

The go directive in a go.mod file now indicates the version of the language used by the files within that module.

This changed use of the go directive means that if you use Go 1.12 to build a module, thus recording go 1.12 in the go.mod file, you will get an error when attempting to build the same module with Go 1.11 through Go 1.11.3. Go 1.11.4 or later will work fine, as will releases older than Go 1.11. If you must use Go 1.11 through 1.11.3, you can avoid the problem by setting the language version to 1.11, using the Go 1.12 go tool, via go mod edit -go=1.11

When an import cannot be resolved using the active modules, the go command will now try to use the modules mentioned in the main module's replace directives before consulting the module cache and the usual network sources.

If a matching replacement is found but the replace directive does not specify a version, the go command uses a pseudo-version derived from the zero time.Time (such as v0.0.0-00010101000000-000000000000).

Go 1.13