[Go] tidwall/evio dig

Reference:

https://github.com/tidwall/evio

1. syscall.SO_REUSEADDR
   https://github.com/tidwall/evio/blob/master/vendor/github.com/kavu/go_reuseport/udp.go#L117
   reusePort
   https://github.com/tidwall/evio/blob/master/vendor/github.com/kavu/go_reuseport/tcp.go#L116
   example code: (var reusePort = 0x0F)

if err = syscall.SetsockoptInt(fd, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1); err != nil {
  syscall.Close(fd)
  return nil, err
 }

 if err = syscall.SetsockoptInt(fd, syscall.SOL_SOCKET, reusePort, 1); err != nil {
  syscall.Close(fd)
  return nil, err
 }

2. *os.File type is used as carrier for storing unix file descriptor which can be copied through fork()
   https://github.com/tidwall/evio/blob/master/vendor/github.com/kavu/go_reuseport/udp.go#L129
   e.g
   file = os.NewFile(uintptr(fd), getSocketFileName(proto, addr))
   if l, err = net.FilePacketConn(file); err != nil {
       return nil, err
   }
3. Use net.FilePacketConn to open os.File socket. https://golang.org/pkg/net/#FilePacketConn
4. set non-block IO
   https://golang.org/pkg/syscall/#SetNonblock
5. Use of epoll:
   epoll_ctl
   epoll_create1 https://github.com/tidwall/evio/blob/master/internal/internal_linux.go#L20
6. Produce/Consume syscall.SYS_EVENTFD2
   Used to manually trigger a epoll consume.
   https://github.com/tidwall/evio/blob/master/internal/internal_linux.go#L27
   https://github.com/tidwall/evio/blob/master/internal/internal_linux.go#L71
7. epoll wait, same as in C but in Go:
   https://github.com/tidwall/evio/blob/master/internal/internal_linux.go#L54
8. Use of syscall.ForkLock  (aka sync.RWMutex )
   It's used for CLOEXEC.
   Go runtime will call syscall.ForkLock.Lock()
   which prevents any thread from setting the CLOEXEC.
   in other words, any FD intended to be set with CLOEXEC(before kernel 2.6.23 this can't be set with atomic system call to open()) should call syscall.ForkLock.RLock() to guard the CLOEXEC setting.
   
   Quote from source:
   https://github.com/golang/go/blob/go1.14.1/src/syscall/exec_unix.go#L18
   Lock synchronizing creation of new file descriptors with fork.
   Time to use it:
   1) Pipe. Does not block. Use the ForkLock.
   2) Socket. Does not block. Use the ForkLock.
   3) Accept. If using non-blocking mode, use the ForkLock.
   Otherwise, live with the race.
   4) Open. Can block. Use O_CLOEXEC if available (Linux).
   Otherwise, live with the race.
   5) Dup. Does not block. Use the ForkLock. On Linux, could use fcntl F_DUPFD_CLOEXEC  instead of the ForkLock, but only for dup(fd, -1).
9. A Mutex must not be copied after first use.
   https://golang.org/src/sync/mutex.go
10. socketcall:
    http://man7.org/linux/man-pages/man2/socketcall.2.html
    https://golang.org/src/syscall/asm_linux_386.s?h=socketcall#L128
11. From Go 1.9 syscall.ForkExec starts using posix_spawn (equivalent to clone(2) syscall with CLONE_VFORK and CLONE_VM)
12. timerfd_create system call is not wrapped in Go, however can call with
    syscall.Syscall6
13. some unix system call wrapper under x/sys
     https://pkg.go.dev/golang.org/x/sys/unix?tab=doc
14. internal/poll.SendFile is a system call SendFile wrapper
    https://golang.org/pkg/internal/poll/#SendFile which is, fast, due to copy inside kernel space.