[Go][wrap up] unsafe package dive

Calls of the functions in unsafe standard package are evaluated at compile time.

Prefix knowledge:
unsafe.Sizeof()
Must read: https://golang.org/pkg/unsafe/#Pointer

2 packages that are implement through compiler implementation (not assembly hook):

1. builtin
2. unsafe

With in Go 1.13, Go has 26 kinds of types.



Reference:
Unsafe package: https://go101.org/article/unsafe.html

Using unsafe pointers comes from the fact that the unsafe mechanism is not protected by the Go 1 compatibility guidelines.

Code depending on unsafe pointers works today could break since a later Go version.

The unsafe.Pointer type is defined as

type Pointer *ArbitraryType

ArbitraryType just hints that a unsafe.Pointer value can be converted to any safe pointer values in Go (and vice versa).

unsafe.Pointer is like the void* in C language.

Unsafe pointers underlying types are unsafe.Pointer.
Zero values of unsafe pointers are represented with the pre-declared identifier nil.

unsafe standard package provides three functions:
(these 3 functions are considered constexpr functions, Go lang doesn't has the concept of const function; however, there's a proposal:  https://github.com/golang/proposal/blob/master/design/15292/2016-09-compile-time-functions.md)

1. func Alignof(variable ArbitraryType) uintptr
2. func Offsetof(selector ArbitraryType) uintptr
3. func Sizeof(variable ArbitraryType) uintptr

e.g (compiled with Go compiler version 1.13 on Linux AMD64 architecture)

package main

import "fmt"
import "unsafe"

func main() {
 var x struct {
  a int64
  b bool
  c string
 }
 const M, N = unsafe.Sizeof(x.c), unsafe.Sizeof(x)
 fmt.Println(M, N) // 16 32

 fmt.Println(unsafe.Alignof(x.a)) // 8
 fmt.Println(unsafe.Alignof(x.b)) // 1
 fmt.Println(unsafe.Alignof(x.c)) // 8

 fmt.Println(unsafe.Offsetof(x.a)) // 0
 fmt.Println(unsafe.Offsetof(x.b)) // 8
 fmt.Println(unsafe.Offsetof(x.c)) // 16
}

Unsafe Pointers Related Conversion Rules:

1. A safe pointer can be explicitly converted to an unsafe pointer, and vice versa.
2. An uintptr value can be explicitly converted to an unsafe pointer, and vice versa.
   But please note, a nil unsafe.Pointer shouldn't be converted to uintptr and back with arithmetic.
   i.e uip uintptr; uip = uip + 3; unsafe.Pointer(uip);
3. Only variable's address (not function, not const) can be converted to unsafe.Pointer type.

Beware:

An uintptr is an integer, not a reference(Thus no object reference count involved). 

Converting a Pointer to a uintptr creates an integer value with no pointer semantics.

Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.

1. Beware that even though unsafe.Pointer can convert to uintptr, the memory pointed by unsafe.Pointer after converted to uintptr is now safe to be garbage collected, thus doing a cast back is UNSAFE.

e.g

var z = new(int)
*z = 42;
var p2 = uintptr(unsafe.Pointer(z))
*(*int)(unsafe.Pointer(p2))) = 3 // dangerous!

2. Beware the addresses of some values might change at run time.
i.e stack can grow/shrink(thus copied to new location), variable might be garbage collected.
An -d=checkptr option might be added for the official Go compiler v1.14 to catch this mis-usage of unsafe.Pointer

3. Beware that we can use a runtime.KeepAlive function call to mark a value as still in using (reachable) before the call.
e.g

var z = new(int)
*z = 42;
var p2 = uintptr(unsafe.Pointer(z))
*(*int)(unsafe.Pointer(p2))) = 3 // dangerous!
runtime.KeepAlive(&z) // This line keep z referenced.

4. Beware that data member of a type(i.e struct) can be garbage collected independently.
In C++, this is controlled by programmer; however, it's unlikely we delete data member before deleting the whole object in C++.
e.g

type T struct {
 x int
 y *[1<<23]byte
}

func bar() {
 t := T{y: new([1<<23]byte)}
 p := uintptr(unsafe.Pointer(&t.y[0]))

 ... // use T.x and T.y

 // A smart compiler can detect that the value
 // t.y will never be used again and think the
 // memory block hosting t.y can be collected now.

 // Using *(*byte)(unsafe.Pointer(p))) is
 // dangerous here.

 // Continue using value t, but only use its x field.
 println(t.x)
}

5. Beware that *unsafe.Pointer is a general safe pointer type.
i.e *unsafe.Pointer can converted to unsafe.Pointer

Use Unsafe Pointers Correctly:
Reference:

https://golang.org/pkg/unsafe/#Pointer

1. convert a *T1 value to unsafe Pointer, then convert the unsafe pointer value to *T2 iff T1 is sizeof smaller then T2.

e.g

// note: it's different from uint64(float64) conversion
func Float64bits(f float64) uint64 {
 return *(*uint64)(unsafe.Pointer(&f))
}

func Float64frombits(b uint64) float64 {
 return *(*float64)(unsafe.Pointer(&b))
}

Real world case:
e.g

package main

import (
 "fmt"
 "unsafe"
)

func main() {
 type MyString string
 ms := []MyString{"C", "C++", "Go"}
 fmt.Printf("%s\n", ms)  // [C C++ Go]
 // ss := ([]string)(ms) // compiling error
 ss := *(*[]string)(unsafe.Pointer(&ms))
 ss[1] = "Rust"
 fmt.Printf("%s\n", ms) // [C Rust Go]
 // ms = []MyString(ss) // compiling error
 ms = *(*[]MyString)(unsafe.Pointer(&ss))
}

2. convert unsafe pointer to uintptr, then use the uintptr value.

3. convert unsafe pointer to uintptr, do arithmetic operations with the uintptr value, then convert it back.
Look for beware item(2, 4)
e.g

package main

import "fmt"
import "unsafe"

type T struct {
 x bool
 y [3]int16
}

const N = unsafe.Offsetof(T{}.y)
const M = unsafe.Sizeof([3]int16{}[0])

func main() {
 t := T{y: [3]int16{123, 456, 789}}
 p := unsafe.Pointer(&t)
 // "uintptr(p)+N+M+M" is the address of t.y[2].
 // this MUST be ONE LINE!!! Look for beware item(2, 4)
 ty2 := (*int16)(unsafe.Pointer(uintptr(p)+N+M+M))
 fmt.Println(*ty2) // 789
}

If above code must be split, use runtime.KeepAlive to mitigate the garbage collected issue, but this won't solve the stack grow/shrink issue. Also, it is NOT recommended to store the address of the end of a memory block (e.g last value of an array, not last value of a type, since type do padding) which PREVENTS the following type instance being garbage collected.

4. convert unsafe pointers to uintptr values as arguments of syscall.Syscall calls.
Usually, a function signature like this is dangerous:
due to beware item(2)

// Assume this function will not inlined.
func DoSomething(addr uintptr) {
 // read or write values at the passed address ...
}

This is fine due to Syscall is implemented differently by compiler:

syscall.Syscall(SYS_READ, uintptr(fd),
   uintptr(unsafe.Pointer(p)), uintptr(n))

However, DO NOT separate the conversion into multiple lines:

u := uintptr(unsafe.Pointer(p))
// At this time, the value referenced by p might
// have become unused and been collected already,
// or the address of the value has changed.
syscall.Syscall(SYS_READ, uintptr(fd), u, uintptr(n))

5. convert the uintptr result of reflect.Value.Pointer or reflect.Value.UnsafeAddr method call to unsafe pointer.
Both function returns uintptr by design(which force you to do unsafe.Pointer right away after the function called)

e.g

// Again, do not seperate the calls
p := (*int)(unsafe.Pointer(reflect.ValueOf(new(int)).Pointer()))

6. convert a reflect.SliceHeader.Data or reflect.StringHeader.Data field to unsafe pointer, and the inverse.
Just don't do it since the implement detail might change underneath the slice or string type.
e.g

package main

import "fmt"
import "unsafe"
import "reflect"

func main() {
 a := [...]byte{'G', 'o', 'l', 'a', 'n', 'g'}
 s := "Java"
 hdr := (*reflect.StringHeader)(unsafe.Pointer(&s))
 hdr.Data = uintptr(unsafe.Pointer(&a))
 hdr.Len = len(a)
 fmt.Println(s) // Golang
 // Now s and a share the same byte sequence, which
 // makes the bytes in the string s become mutable.
 a[2], a[3], a[4], a[5] = 'o', 'g', 'l', 'e'
 fmt.Println(s) // Google
}

e.g.

package main

import (
 "fmt"
 "unsafe"
 "reflect"
 "runtime"
)

func main() {
 a := [6]byte{'G', 'o', '1', '0', '1'}
 bs := []byte("Golang")
 hdr := (*reflect.SliceHeader)(unsafe.Pointer(&bs))
 hdr.Data = uintptr(unsafe.Pointer(&a))
 runtime.KeepAlive(&a) // needed, due to a's Slice's data is not needed.
 hdr.Len = 2
 hdr.Cap = len(a)
 fmt.Printf("%s\n", bs) // Go
 bs = bs[:cap(bs)]
 fmt.Printf("%s\n", bs) // Go101
}

Convert a byte slice to a string best practice:

This follows the item 1

• convert a *T1 value to unsafe Pointer, then convert the unsafe pointer value to *T2 iff T1 is sizeof smaller then T2.

func ByteSlice2String(bs []byte) string {
 return *(*string)(unsafe.Pointer(&bs))
}

Something about reflect.StringHeader and reflect.SliceHeader:

reflect.StringHeader / reflect.SliceHeader
Both of these struct has the design putting Data at the beginning of the struct/type due to make the original 'string'/'array' data structure as it is when assign to StringHeader/SliceHeader.

e.g StringHeader

package main

import "fmt"
import "unsafe"
import "reflect"

func main() {
 a := [...]byte{'G', 'o', 'l', 'a', 'n', 'g'}
 s := "Java"
 hdr := (*reflect.StringHeader)(unsafe.Pointer(&s))
 hdr.Data = uintptr(unsafe.Pointer(&a))
 hdr.Len = len(a)
 fmt.Println(s) // Golang
 // Now s and a share the same byte sequence, which
 // makes the bytes in the string s become mutable.
 a[2], a[3], a[4], a[5] = 'o', 'g', 'l', 'e'
 fmt.Println(s) // Google
}

e.g SliceHeader

package main

import (
 "fmt"
 "unsafe"
 "reflect"
 "runtime"
)

func main() {
 a := [6]byte{'G', 'o', '1', '0', '1'}
 bs := []byte("Golang")
 hdr := (*reflect.SliceHeader)(unsafe.Pointer(&bs))
 hdr.Data = uintptr(unsafe.Pointer(&a))
 runtime.KeepAlive(&a) // needed!
 hdr.Len = 2
 hdr.Cap = len(a)
 fmt.Printf("%s\n", bs) // Go
 bs = bs[:cap(bs)]
 fmt.Printf("%s\n", bs) // Go101
}

Consider using unsafe.Pointer instead of uintptr for a bit more safety due to later copies the value which release the RHS reference count, former keeps the reference count intact avoid GC to collect the memory.
e.g

type SliceHeader struct {
 Data unsafe.Pointer
 Len  int
 Cap  int
}

type StringHeader struct {
 Data unsafe.Pointer
 Len  int
}

func String2ByteSlice(str string) (bs []byte) {
 strHdr := (*StringHeader)(unsafe.Pointer(&str))
 sliceHdr := (*SliceHeader)(unsafe.Pointer(&bs))
 sliceHdr.Data = strHdr.Data
 sliceHdr.Len = strHdr.Len
 sliceHdr.Cap = strHdr.Len
 
 // The KeepAlive call is inessential now.
 //runtime.KeepAlive(&str)
 return
}

e.g Our defined string header vs. unsafe.StringHeader

package main

import (
 "fmt"
 "reflect"
 "unsafe"
)

type MyStringHeader struct {
 Data unsafe.Pointer
 Len  int
}

func main() {
 s := "Golang"
 hdr := (*reflect.StringHeader)(unsafe.Pointer(&s))
 fmt.Println(hdr.Len) // 6
 hdr2 := (*MyStringHeader)(unsafe.Pointer(&s))
 fmt.Println(hdr2.Len) // 6
}

Find the data type size:

package main

import (
 "fmt"
 "unsafe"
)

func main() {
 a := [3]int{1, 2, 3}

 basePtr := uintptr(unsafe.Pointer(&a[0]))
 secondPtr := uintptr(unsafe.Pointer(&a[2]))

 print("diff: " + fmt.Sprintf("%d", secondPtr-basePtr) + "\n")
}