[Go] goroutine preemption

Go runtime/g0 schedules the goroutines when:

• system call
• blocking on channel
• sleeping
• waiting on a mutex
• when goroutine needs to grow it's stack size
• goroutines running for more than 10ms is marked as preemptible (the preemption is initiated/monitored by the thread sysmon), and  the preemption is done at the function prolog when the goroutine’s stack is increasing if without any above pause criteria
• For those goroutines does not increasing size, using runtime.Gosched() specifically to force the goroutine being preempted.
• In Go 1.14, goroutines will auto-preempt even no stack size changes or any criteria listed above meet.
  This can be turned off with $ GODEBUG=asyncpreemptoff=1

How does async preemp work?

1. A dedicated thread M called sysmon will watch every goroutines in every P running longer than 10ms.
2. Once over 10ms, sysmon will send event signal SIGURG to those goroutines.
3. Every P has a gsignal goroutine to handle signals. Once a goroutine  runs over 10ms receiving SIGURG will be parked into local queue and gsignal takes over handling the SIGURG.
   This completes the preemption.

 
Why choosing signel SIGURG?

In “Proposal: Non-cooperative goroutine preemption”:

• It should be a signal that’s passed-through by debuggers by default.
• It shouldn’t be used internally by libc in mixed Go/C binaries [...].
• It should be a signal that can happen spuriously without consequences.
• We need to deal with platforms without real-time signals [...].

Like thread cancellation point, goroutine can not be stop anywhere in the code. It has to be a safe point.

Go 1.14's async preemption also enables GC to STW by sending SIGURG to all goroutines.

[Go] g0's roles

g0 traits (for every M has g0)

•     Fix and larger stack size(ordinary goroutin occupies 2kb at start).
•     g0's stack usually doesn't grow.
•     responsible for goroutine creation.
•     responsible for defer function allocation.
•     GC operations. Including STW, mark and sweep operations.
•     Stack growth for running goroutins.
•     Schedule goroutine to run. (each goroutine end with calling runtime.goexit() which notify g0 it's finished) Or goroutine can call runtime.Goexit() manually)
•     Maintain 2 goroutine queues, one with recycled goroutines with an allocated stack(max 2kb size), one with recycled goroutines with empty stack)
•     There are 2 global goroutine queues as well protected by locks.

Each local goroutine queue for P has a maximum capacity of 256, and any new incoming goroutine is pushed to the global queue after local queue is full.

Each P also maintains a queue(64 size) for freed goroutines.(goroutines are recycleable )

Work-stealing

When a processor does not have any work, it applies the following rules until one can be satisfied:

• pull work from the local queue
• pull work from the global queue
• pull work from network poller
• steal work from the other P’s local queues

Threads can be blocked on system calls and the number of blocked threads is not limited in Go runtime.

Affinity limitation

P's Local goroutin queue will be used for all operations expect system calls

such as blocking operations on channels and selects, waiting on timers and locks.

Two features could restrict the affinity between a goroutine and a thread:

• Work-stealing.
  When a processor P does not have enough work in its local queue, it will steal goroutines from others P if the global queue and the network poller are empty. When stolen, the goroutines will then run on another thread.
• System calls.
  When a syscall occurs (e.g. files operations, http calls, database operations, etc.), Go moves the running OS thread in a blocking mode, letting a new thread processing the local queue on the current P.

However, with goroutins using the same channel will be grouped into same P during scheduling. And for those waiting for channel will be scheduled with high priority then other goroutin to run next.

[Go] consider the size of a channel

While setting the size of a channel,
should consider how go scheduling mechanism works.

A small size of channel(including an un-buffered channel) causing the producer to block the running goroutine thus forces a re-scheduling (on same physical thread most of the time).

While a larger size of channel can reduce the latency of producing data into the channel but having the consumer delay the processing.

Here's the time slice down into different stages(ref by Vincent Blanchon's article):

Measure the performance before hard-setting the channel size,
i.e make the size configurable.

Reference:
https://medium.com/a-journey-with-go/go-what-does-a-goroutine-switch-actually-involve-394c202dddb7

[Go] use sync.Pool with sense

https://github.com/golang/go/issues/23199

tl;dr
It's caller's responsibility to put back normalized size of instance back into
sync.Pool

Intermingling small and large size of instance put into the sync.Pool will
eventually getting all the instance inside the pool with large size.
(Read Reference 1)

thus either create a bunch of sync.Pool bucketize the items by size (aka. slab memory allocation) or only put back certain size of instance back to the pool.

Same idea as goroutin instance recycling.
Go runtime will not recycle any goroutin size > 2k.


Reference:
https://github.com/golang/go/issues/23199
http://vsdmars.blogspot.com/2019/01/split-stack-reading-notes-and-references.html

Reason:
the buffer inside the sync.Pool will remain LARGE and never shrink even serving small objects
code:

pool := sync.Pool{New: func() interface{} { return new(bytes.Buffer) }}

processRequest := func(size int) {
	b := pool.Get().(*bytes.Buffer)
	time.Sleep(500 * time.Millisecond) // Simulate processing time
	b.Grow(size)
	pool.Put(b)
	time.Sleep(1 * time.Millisecond) // Simulate idle time
}

// Simulate a set of initial large writes.
for i := 0; i < 10; i++ {
	go func() {
		processRequest(1 << 28) // 256MiB
	}()
}

time.Sleep(time.Second) // Let the initial set finish

// Simulate an un-ending series of small writes.
for i := 0; i < 10; i++ {
	go func() {
		for {
			processRequest(1 << 10) // 1KiB
		}
	}()
}

// Continually run a GC and track the allocated bytes.
var stats runtime.MemStats
for i := 0; ; i++ {
	runtime.ReadMemStats(&stats)
	fmt.Printf("Cycle %d: %dB\n", i, stats.Alloc)
	time.Sleep(time.Second)
	runtime.GC()
}

[split stack] reading notes and references

Reference:
gccgo split stack implementation

1. The stack can start splitting at any point.
2. The stack size is automatically recorded at program startup,
   and each thread startup.
3. The gold linker detects calls from split-stack code to non-split-stack
   code, and rewrites the function header to force a large stack segment to be allocated.
   i.e.
   When not using the gold linker, calls from split-stack code to non-split-stack code will just have whatever is left of the current stack segment, which may not be large enough.
   (look up to "Backward compatibility" section)

In the complex GCC ecosystem the linker is separate from the compiler.
GCC can't assume that gold is available at all.
When building gccgo, configure using
--with-ld=/path/to/gold

The -fuse-ld=gold option is newer than gccgo.
Ian supposes it would be nice if:
* the GCC configure process checks whether -fuse-ld=gold works; if so:
  * -fuse-ld=gold is passed to the libgo configure/build
  * -fuse-ld=gold is used by default by the gccgo driver program



Reference:
Split Stacks in GCC

• Split stacks permits a dis-contiguous stack which is grown automatically as needed. 
• This means that when running multiple threads, each starting with a small stack, and have the stack grow and shrink as required by the program.
• In Golang, each init. GoRoutine sized in 2kB, code reference:
  https://github.com/golang/go/blob/0098f8aeaceb5feec7462ae64f8ce91a473360c1/src/runtime/stack.go#L71

Obvious benefits

• The memory usage of a typical multi-threaded program can decrease significantly, as each thread does not require a worst-case stack size.
• It becomes possible to run millions of threads
  (either full NPTL threads or co-routines) in a 32-bit address space.

Basic explained

Stack will have a guaranteed zone which is always available.
Reference:
[LWN] Preventing stack guard-page hopping


The size of the guard area will be target specific.
It will include enough stack space to actually allocate more stack space.
Each function will have to verify that it has enough space in the current stack to execute.

The basic verification will be a comparison between the stack pointer and the current bottom of the stack plus the guaranteed zone size.
This will have to be the first operation in the function, and will also be target specific.

It must be fast, as it will be executed by each called function.

Two cases to consider.

1. For functions which require a stack frame less than the size of the guaranteed guard area, we can do a simple comparison between the stack pointer and the stack limit.
2. For functions which require a larger stack frame, we must do a comparison including the size of the stack frame.

Design options

1. Reserve a register to hold the bottom of the stack plus the guaranteed size. This will have to be a callee-saved register.
2. Use a TLS(Thread Local Storage) variable. In the general case, in a shared library, this will require calling the __tls_get_addr function.
   Reference:
   How fast is thread local variable access on Linux
   (GOLD elf linker)
   http://gittup.org/cgi-bin/man/man2html?gold+1 
   
   That means that that function will have to work without requiring any additional stack space.
   This is infeasible unless the whole system is compiled with split stacks.
   It would require dlopen's LD_BIND_NOW to be set, so that the __tls_get_addr function is resolved at program startup time.
   Even that is probably insufficient unless we can ensure that the space for the (TLS) variable is fully allocated.
   In general Ian doesn't think they can ensure this, because dlopen can cause a thread to require more space for TLS variables, and that space will be allocated on the first call to __tls_get_addr.
   Reference:
   http://man7.org/linux/man-pages/man8/ld.so.8.html
   LD_BIND_NOW (since glibc 2.1.1)
   If set to a nonempty string, causes the dynamic linker to
   resolve all symbols at program startup instead of deferring
   function call resolution to the point when they are first
   referenced.  This is useful when using a debugger.
3. Have the stack always end at a N-bit boundary.
   E.g., if we always allocate stack segments as a multiple of 4K,
   then align each one so that the stack always ends at a 12-bit boundary.
   Then the amount of space remaining on the stack is SP & 0xfff.
4. Introduce a new function call which handles the comparison of the stack pointer and the stack expansion.
5. Reuse the stack protector support field.
   When using glibc each thread descriptor has a field used by the stack protector.
   Of course it is then not possible to use split stacks in conjunction with stack protector.
6. At least on x86, arrange to allocate a new field in the TCB(thread control block) header accessible via %fs or %gs.
   This is probably the best solution, and it is the one implemented for i386 and x86_64.

Reference:
TCB Thread Control Block in linux kernel:
https://en.wikipedia.org/wiki/Thread_control_block

Expanding the stack

• Expanding the stack requires allocating additional memory.
• This additional memory will have to be allocated using only the stack space slot.
• All of the functions used to allocate additional stack space must be compiled to not use a split stack.
• A new function attribute, no_split_stack will be introduced to mean that the stack should not be split.
• It would also work to ensure that the stack is large enough that they do not need to split the stack during the allocation call.
• After expanding the stack, the function will copy any stack based parameters from the old stack to the new stack.
• Fortunately, all C++ objects which require a copy or move constructor are implicitly passed by reference,so copying the parameters on the stack is OK.
• For varargs functions, this is impossible in general, so we will compile varargs functions differently:
  they will use an argument pointer which is not necessarily based on the frame pointer.
  For functions which return objects on the stack, the objects will be returned on the old stack. (RVO)
  This should normally happen automatically, as the initial hidden parameter will naturally point to the old stack.
• When expanding the stack, the return address of the function will be managed to point to a function which will release the allocated stack block and reset the stack pointer to the caller.
  Reference:
  http://vsdmars.blogspot.com/2017/11/assembly-note.html
• The address of the old stack block, and the old stack pointer, will have been saved somewhere in the new stack block.

Backward compatibility

We want to be able to use split stack programs on systems with pre-built libraries compiled without split stacks.
This means that we need to ensure that there is sufficient stack space before calling any such function.

Each object file compiled in split stack mode will be annotated to indicate that the functions use split stacks.

This should probably be annotated with a note but there is no general support for creating arbitrary notes in GNU as.

Therefore, each object file compiled in split stack mode will have an empty section with a special name: .note.GNU-split-stack

If an object file compiled in split stack mode includes some functions with the no_split_stack attribute, then the object file will also have a .note.GNU-no-split-stack section.

This will tell the linker that some functions may not have the expected split stack prologue.

When the linker links an executable or shared library, it will look for calls from split-stack code to non-split-stack code.

This will include calls to non-split-stack shared libraries
(thus, a program linked against a split-stack shared library may fail if at runtime the dynamic linker finds a non-split-stack shared library;
it might be desirable to use a new segment type to detect this situation).

For calls from split-stack code to non-split-stack code, the linker will change the initial instructions in the split-stack (caller) function.
This means that the linker will have to have special knowledge of the instructions that the compiler emits.
The effect of the changes will be to increase the required frame-size by a number large enough to reasonably work for a non-split-stack.
This will be a target dependent number; the default will be something like 64K.
Note that this large stack will be released when the split-stack function returns.
Note that I'm disregarding the case of split-stack code in a shared library calling non-split-stack code in the main executable; that seems like an unlikely problem.


Function pointers are a tricky case.
In general we don't know whether a function pointer points to split-stack code.
Therefore, all calls through a function pointer will be modified to call (or jump to) a special function __fnptr_morestack.
This will use a target specific function calling sequence, and will be implemented as though it were itself a function call instruction.
That is, all the parameters will be set up, and then the code will jump to __fnptr_morestack.
The __fnptr_morestack function takes two parameters: the function pointer to call, and the number of bytes of arguments pushed on the stack.

[Go] Golang's memory management - Eben Freeman

Eben Freeman has given a great talk about Golang's memory management
This is a note for the talk and knowledge add-ons.

Reference:
GopherCon 2018 - Allocator Wrestling https://about.sourcegraph.com/go/gophercon-2018-allocator-wrestling

Knowledge from C++ world:
While coding in Go, which has escape analysis provided by the compiler,
consider writing the type instance as auto variable which avoid compiler putting
it into the heap.
Don't escape unnecessary auto variables by returning it's address.
Return by value is preferred if the type size is small.
(i.e If the type is large, it's instance will be put into heap anyhow.)
Since auto variables are not managed by GC :-)

Allocator internals

Design goals

• Efficiently satisfy allocations of a given size, but avoid fragmentation
  Solution: allocate like-sized objects in blocks
• Avoid locking in the common case
  Solution: maintain per-CPU caches
• Efficiently reclaim freeable memory
  Solution: use bitmaps for metadata, run GC concurrently

Heap is divided into two levels of structure: Arenas and Spans

(The idea is same as slab allocator:

https://en.wikibooks.org/wiki/The_Linux_Kernel/Memory#Slab_Allocator )

A global mheap struct keeps track of both of them:

https://github.com/golang/go/blob/master/src/runtime/mheap.go#L31

type mheap struct {
  arenas [1 << 22]*heapArena       // Covering map of arena frames
  free []mSpanList                 // Lists of fully free spans
  central []mcentral               // Lists of in-use spans
  // plus much more
}

Arenas are coarse sections of the available address space (64MB on 64-bit archs).
We allocate OS memory in units of arenas.
For each arena, there's metadata in a heapArena struct:
https://github.com/golang/go/blob/master/src/runtime/mheap.go#L201

type mheap struct {
  arenas [1 << 22]*heapArena    // Covering map of arena frames
  // ...
}

type heapArena struct {
   // page-granularity map to spans
  spans [pagesPerArena]*mspan
  // pointer/scalar bitmap (2bits/word)
  bitmap [heapArenaBitmapBytes]byte 
}

Span:

There are ~70 different mspan size classes.
https://github.com/golang/go/blob/master/src/runtime/mheap.go#L308

type mspan struct {
  startAddr  uintptr
  npages     uintptr
  spanclass  spanClass

  // allocated/free bitmap
  allocBits *gcBits
  // ...
}

Each P has an mcache holding a span of each size class.
https://github.com/golang/go/blob/master/src/runtime/mcache.go#L19
Ideally, allocations can be satisfied directly out of the mcache (thus they're fast).
In Go, a P is a scheduling context.
( Refer to Golang scheduler note: http://vsdmars.blogspot.com/2018/06/golang-go-scheduler.html )
Generally there's one P per core (M, aka. thread) and
at most one goroutine running per P at a time(P has running queue):

To allocate, we find the first free object in our cached mspan, then return its address:

Let's say we need 96 bytes.

First we'd look in the mcache for the mspan with 96-byte objects.

After allocation would look like this:

This design means that "most" memory allocations are fast and require no locking.

There are 3 quick steps:

1. Find a cached span with the right size ( mcache.mspan[sizeClass] )
2. Find the next free object in the span
3. If necessary, update the heap bitmap

Garbage collection

Go uses a tricolor concurrent mark-sweep garbage collector.

Reference:

Modern garbage collection A look at the Go GC strategy https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e

Tri-Color Incremental Updating GC https://stackoverflow.com/questions/2364274/tri-color-incremental-updating-gc-does-it-need-to-scan-each-stack-twice

Golang’s Real-time GC in Theory and Practice https://making.pusher.com/golangs-real-time-gc-in-theory-and-practice/

Tri-color mark steps:

• Initially, all objects are marked as white. (3 colors exist: white, grey and black)
• GC starts by marking goroutine stacks and globals.
• When GC reaches an object, mark it grey.
• When an object's referents are all marked, GC mark the object with black.
• At the end, objects are either white or black.
• White objects can then be swept and freed.

However; GC can be run concurrently, thus:

type S struct {
  p *int
}

func f(s *S) *int {
  r := s.p
  s.p = nil  // What if GC runs at this point? The returning pointer points to null.
  return r
}

Rough solution explained:
Preventing this from occurring, compiler translates pointer writes into potential calls into the write barrier.
i.e Mark it has referent during a pointer assignment.
i.e When GC is marking, the write barrier is turned on.

quote:

Mark Setup - STW When a collection starts, the first activity that must be performed is turning on the Write Barrier.
The purpose of the Write Barrier is to allow the collector to maintain data integrity on the heap during a collection since both the collector and application goroutines will be running concurrently.
In order to turn the Write Barrier on, every application goroutine running must be stopped.
This activity is usually very quick, within 10 to 30 microseconds on average.
That is, as long as the application goroutines are behaving properly.

Reference:

https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html#:~:text=The%20purpose%20of%20the%20Write,goroutine%20running%20must%20be%20stopped.

That is to say:
Allocating a big buffer of scalars is much cheaper than allocating a big buffer of pointers, because you have to follow the pointers.

Experiment with GC performance

1. Crude experimenting

• GOGC=off : Disables garbage collector
• GODEBUG=sbrk=1 : Replaces entire allocator with simple persistent allocator that gets big blocks from the OS and gives you successive slices of those as you allocate.

Problem with this approach:

Not useful for production code due to not reliable with large heap allocation.

2. Profiling

• Use pprof to check for runtime.mallocgc
• Use flamegraph viewer in the pprof web UI
• Or linux perf if pprof isn't enabled in the binary.

    

Problem with this approach:

• Program might not be CPU-bound
• Allocation might not be on critical path
• Time in background marking (gcBgMarkWorker) can mislead (time spent here doesn't necessarily mean you have net slowdown)

3. go tool trace

$ curl localhost:6060/debug/pprof/trace?seconds=5 > trace.out

$ go tool trace trace.out

Coding suggestion

• Limit pointers
  Compiler will show which variable is on heap.
  Software engineering 101, no matter what language you use, don't construct variables inside a loop.
  If you really really need to, the type to be constructed should not have pointer type inside, which helps the GC to reasoning.
  $ go build -gcflags="-m -m"
• Allocate in batches (Hand make your own slab allocator)
• Try to recycle objects (e.g., sync.Pool) ( Refer to note for Concurrency in Go - Katherine Cox-Buday http://vsdmars.blogspot.com/2018/10/golangbook-concurrency-in-go-katherine.html )

Further reference:

https://segment.com/blog/allocation-efficiency-in-high-performance-go-services/

https://blog.golang.org/ismmkeynote

Must read: https://www.slideshare.net/TylerTreat/so-you-wanna-go-fast-80300458

Note for Tyler Treat's slides:

Scheduler is fairly grouping goroutines holding the same sync.Mutex instance on same CPU core avoiding ping-pong effect.

Use defer only if there's more then one return lambda being registered.
If single return condition, don't use defer due to defer is relatively _SLOW_.

Use DOD for designing types http://vsdmars.blogspot.com/search/label/design_dod

Use interface is always slow than pure type due to another layer of indirection.

Convert []byte to string type judiciously.

Use sync.Pool while necessary.

“We generally don’t want sync/atomic to be used at all...Experience has shown us again and again that very very few people are capable of writing correct code that uses atomic operations...”
—Ian Lance Taylor
Why? This is why: http://vsdmars.blogspot.com/search/label/cpp_concurrent

The Go race detector doesn’t protect you from doing dumb stuff.

Unsafe is, in fact, unsafe.

Struct layout can make a big difference. (padding http://vsdmars.blogspot.com/2018/09/golangc-padding.html & ping-pong effect with mutex lock)

Go makes concurrency easy enough to be dangerous.

[Go][book] Concurrency in Go - Katherine Cox-Buday

Summary of reading:
Concurrency in Go - Katherine Cox-Buday

I agree with Eli Bendersky's point of view about this book i.e
if coming from the arena of C++, which is that, the book is relatively
can be considered as a 'reference book' for some golang tricks.

Before delve into golang concurrency, the understanding of
golang scheduler is a must.
Reference:
http://vsdmars.blogspot.com/2018/06/golang-go-scheduler.html
http://vsdmars.blogspot.com/2018/07/golangnote-analysis-of-go-runtime.html
HPX lib (C++)
Intel TBB (C++)

Understanding memory model is also essential to concurrency, although in golang, this is mostly hidden by it's run-time and channel.
Reference:
http://vsdmars.blogspot.com/2018/09/concurrency-c-wrap-up-2018.html


For advanced concurrency programming, take a look at
The Art of Multiprocessor Programming - Maurice Herlihy & Nir Shavit
Reference:
http://vsdmars.blogspot.com/2016/01/multiprocessor-programming-types-of.html

Jotting down reading notes for those are fun/useful :-)

RWLock:
[C++]
std::shared_mutex

[Golang]
sync.RWMutex

[Python]
None, and i coined one for fun:
https://github.com/verbalsaintmars/python_util/tree/master/rwlock

Cond Variable:
[C++]
std::condition_variable

[Golang]
sync.NewCond

sync.Once dead lock example:

var onceA, onceB sync.Once
var initB func()
initA := func() { onceB.Do(initB) }
initB = func() { onceA.Do(initA) }
onceA.Do(initA)

sync.Pool
sync.Pool can be the panacea for creating new type instance
inside a 'loop', i.e for loop.

Experienced engineer could spot the performance hit 'bad design'
by seeing new/make_shared/NewXXX etc. inside a for/while loop.

Reusing the type instance from being GCed boosts the performance;
however, just that, the pool stored type instance should be stateless
to avoid side-effect.

GC can still drain the Pool, thus a .Get() with drained Pool will create a new type's instance.

Further intriguing read:
Purpose of sync.Pool 
Learn High Performance Go 

Close(Channel):
close(channel)  acts as a signal for those blocking channels to continue.
e.g
https://play.golang.org/p/k3143TbiqO4

package main

import (
 "fmt"
 "time"
)

var ch = make(chan struct{})

func run(cnt int) {
 <-ch
 fmt.Println(cnt)
}

func main() {
 go run(1)
 go run(2)
 go run(3)
 go run(4)

 time.Sleep(10 * time.Second)
 close(ch)
 time.Sleep(10 * time.Second)
}

Select as normalized distribution seed:
Using select to act as a normalized distribution seed.
e.g:
https://play.golang.org/p/2m4VB9rhfcU

package main
import "fmt"

func main() {
 c1 := make(chan interface{})
 close(c1)
 c2 := make(chan interface{})
 close(c2)

 var c1Count, c2Count int

 for i := 1000; i >= 0; i-- {
  select {
  case <-c1:
   c1Count++
  case <-c2:
   c2Count++
  }
 }
 fmt.Println(c1Count)
 fmt.Println(c2Count)
}

Daisy-chain (it's simply elegant):
from slide:https://talks.golang.org/2012/concurrency.slide

func f(left, right chan int) {
    left <- 1 + <-right
}

func main() {
    const n = 10000
    leftmost := make(chan int)
    right := leftmost
    left := leftmost
    for i := 0; i < n; i++ {
        right = make(chan int)
        go f(left, right)
        left = right
    }
    go func(c chan int) { c <- 1 }(right)
    fmt.Println(<-leftmost)
}

Legacy golang code alert (prior Go1.5):

runtime.GOMAXPROCS(runtime.NumCPU())

i.e Go 1.5 is set to make the default value of GOMAXPROCS
the same as the number of CPUs on your machine,
so above code isn't necessary anymore.


Channel error/result back to caller:
https://play.golang.org/p/aCowW5mNEp1

package main

import "fmt"

func main() {
 type result struct {
  stat int
 }

 CheckErr := func(input chan int) <-chan result {
  r := make(chan result)

  go func() {
   defer close(r)
   for i := range input {
    r <- result{i}
   }
  }()

  return r
 }

 input := make(chan int)

 go func() {
  defer close(input)
  for i := range [10]struct{}{} {
   input <- i
  }
 }()

 for result := range CheckErr(input) {
  fmt.Println(result)
 }

}

The or-channel: 
https://play.golang.org/p/XsDYdU8ks5D

package main

import (
 "time"
)

func main() {

 var or func(channels ...chan interface{}) <-chan interface{}

 or = func(channels ...chan interface{}) <-chan interface{} {
  switch len(channels) {
  case 0:
   return nil
  case 1:
   return channels[0]
  }

  orDone := make(chan interface{})

  go func() {
   defer close(orDone) // Trick is here :-)

   switch len(channels) {
   case 2:
    select {
    case <-channels[0]:
    case <-channels[1]:
    }
   default:
    select {
    case <-channels[0]:
    case <-channels[1]:
    case <-channels[2]:
    case <-or(append(channels[3:], orDone)...):
    }
   }
  }()
  return orDone
 }

 chs := make([]chan interface{}, 7)
 for i, _ := range chs {
  chs[i] = make(chan interface{}, 1)
 }

 done := make(chan struct{})

 go func() {
  <-or(chs...)
  done <- struct{}{}
 }()
 time.Sleep(3 * time.Second)
 chs[0] <- struct{}{}
 <-done
}

Best Practices for Constructing Pipelines:
https://play.golang.org/p/4efPkGXUSoM

package main

import "fmt"

func main() {
 pipe := func(done chan struct{}, i ...int) chan int {
  retch := make(chan int)

  go func() {
   defer close(retch)
   for fi := range i {
    select {
    case <-done:
    case retch <- fi:
    }
   }
  }()

  return retch
 }

 done := make(chan struct{})
 for c := range pipe(done, 47, 38, 29) {
  fmt.Println(c)
 }
}

Fan-Out, Fan-In (auh, map-reduce):
https://play.golang.org/p/fmGAGFCuzDy
Fan-out:
Use for-range as pipeline channel to fan-out.

Take advantage of
var wg sync.WaitGroup
defer wg.done()
wg.Wait()
to fan-in.

package main

import "sync"

func main() {

 fanIn := func(done <-chan interface{}, channels ...<-chan interface{},
 ) <-chan interface{} {

  var wg sync.WaitGroup

  multiplexedStream := make(chan interface{})

  multiplex := func(c <-chan interface{}) {
   defer wg.Done()
   for i := range c {
    select {
    case <-done:
     return
    case multiplexedStream <- i:
    }
   }
  }

  wg.Add(len(channels))

  for _, c := range channels {
   go multiplex(c)
  }

  go func() {
   wg.Wait()
   close(multiplexedStream)
  }()

  return multiplexedStream
 }
 done := make(chan interface{})
 chan1 := make(chan interface{}, 1)
 chan2 := make(chan interface{}, 1)
 chan1 <- struct{}{}
 chan2 <- struct{}{}
 <-fanIn(done, chan1, chan2)
}

The or-done-channel:
https://play.golang.org/p/HsgZe8Ke9xU
Acted as a building block for other services.

package main

import (
 "fmt"
 "time"
)

func main() {

 orDone := func(done, c <-chan interface{}) <-chan interface{} {
  valStream := make(chan interface{})
  go func() {
   defer close(valStream)
   for {
    select {
    case <-done:
     return
    case v, ok := <-c:
     if ok == false {
      return
     }
     select {
     case valStream <- v:
     case <-done:
     }
    }
   }
  }()
  return valStream
 }
 ch := make(chan interface{}, 3)
 ch <- 1
 ch <- 2
 ch <- 3

 go func() {
  time.Sleep(3 * time.Second)
  close(ch)
 }()

 for v := range orDone(make(chan interface{}), ch) {
  fmt.Println(v)
 }
}

The tee-channel:
https://play.golang.org/p/bcbVnfmUdBT
interleave output.

package main

import (
 "fmt"
 "time"
)

func main() {

 orDone := func(done, c <-chan interface{}) <-chan interface{} {
  valStream := make(chan interface{})
  go func() {
   defer close(valStream)
   for {
    select {
    case <-done:
     return
    case v, ok := <-c:
     if ok == false {
      return
     }
     select {
     case valStream <- v:
     case <-done:
     }
    }
   }
  }()
  return valStream
 }

 done := make(chan interface{})
 ch := make(chan interface{}, 3)
 ch <- 1
 ch <- 2
 ch <- 3

 tee := func(done <-chan interface{}, in <-chan interface{}) (_, _ <-chan interface{}) {
  out1 := make(chan interface{})
  out2 := make(chan interface{})

  go func() {
   defer close(out1)
   defer close(out2)

   for val := range orDone(done, in) {
    var out1, out2 = out1, out2
    for i := 0; i < 2; i++ {
     select {
     case <-done:
      return
     case out1 <- val:
      out1 = nil // next loop out1 will be ignored.
     case out2 <- val:
      out2 = nil // ditto
     }
    }
   }
  }()
  return out1, out2
 }

 go func() {
  time.Sleep(3 * time.Second)
  done <- struct{}{}
 }()

 a, b := tee(done, ch)
 for v := range a {
  fmt.Printf("a val: %v\n", v)
  fmt.Printf("b val: %v\n", <-b)
 }
}

The bridge-channel:
https://play.golang.org/p/jvTDGZbFeZO
Channel contains channels.
Fan-out by assigning each sub-channels into
fan-in channel for caller to consume.

Till now you can see most of the implements from the book
use 0 buffer channel.

It's the lazy-binding technique which only processes data until
the caller consumes the processed data.

Like 'yield' (i.e coroutine, light weight thread/green thread)
in major languages, e.g C#, Python
Reference:
[C++]
http://c9x.me/articles/gthreads/intro.html

package main

import (
 "fmt"
 "time"
)

func main() {
 orDone := func(done, c <-chan interface{}) <-chan interface{} {
  valStream := make(chan interface{})
  go func() {
   defer close(valStream)
   for {
    select {
    case <-done:
     return
    case v, ok := <-c:
     if ok == false {
      return
     }
     select {
     case valStream <- v:
     case <-done:
     }
    }
   }
  }()
  return valStream
 }

 bridge := func(
  done <-chan interface{},
  chanStream <-chan <-chan interface{},
 ) <-chan interface{} {

  valStream := make(chan interface{})

  go func() {
   defer close(valStream)

   for {
    var stream <-chan interface{}

    select {
    case maybeStream, ok := <-chanStream:
     if ok == false {
      return
     }
     stream = maybeStream
    case <-done:
     return
    }

    for val := range orDone(done, stream) {
     select {
     case valStream <- val:
     case <-done:
     }
    }
   }
  }()
  return valStream
 }

 done := make(chan interface{})
 base := make(chan interface{}, 1)
 next := make(chan (<-chan interface{}), 1)
 base <- 42
 next <- base

 go func() {
  time.Sleep(3 * time.Second)
  close(base)
  close(next)
 }()

 result := bridge(done, next)
 for i := range result {
  fmt.Println(i)
 }
}

Queuing:
Buffered channels piped together then run in sequence.
The receiving type and returning type could be the same thus forms a Monad..
Reference:
http://vsdmars.blogspot.com/2017/09/fp-functor-applicative-monoid-monad.html
http://vsdmars.blogspot.com/2016/08/fp-related-readstudy-links.html



The context Package:
Familiar with the usage, skim over this section.

Timeouts and Cancellation:
Use context timeout mechanism, passing down context instance
through the calling stack.


Heartbeat:
https://play.golang.org/p/4RHoiIjnLLL
Use time.Tick(interval) https://golang.org/pkg/time/#Tick  as returning channel.
Can be used as interval probing check.

time.After can be used to check if a select statement is in healthy state within the timeout.

package main

import (
 "fmt"
 "time"
)

func main() {
 doWork := func(done <-chan interface{},
  pulseInterval time.Duration) (<-chan interface{}, <-chan 
time.Time) {

  heartbeat := make(chan interface{})
  results := make(chan time.Time)

  go func() {
   defer close(heartbeat)
   defer close(results)

   pulse := time.Tick(pulseInterval)
   workGen := time.Tick(2 * pulseInterval)

   sendPulse := func() {
    select {
    case heartbeat <- struct{}{}:
    default:
    }
   }

   sendResult := func(r time.Time) {
    for {
     select {
     case <-done:
      return
     case <-pulse:
      sendPulse()
     case results <- r:
      return
     }
    }
   }

   for {
    select {
    case <-done:
     return
    case <-pulse:
     sendPulse()
    case r := <-workGen:
     sendResult(r)
    }
   }
  }()

  return heartbeat, results
 }

 done := make(chan interface{})

 go func() {
  defer close(done)
  time.Sleep(3 * time.Second)
 }()

 h, r := doWork(done, 120*time.Second)

 for {
  select {
  case _, ok := <-h:
   if ok == false {
    return
   }
  case _, ok := <-r:
   if ok == false {
    return
   }
  case <-time.After(1 * time.Second):
   fmt.Println("worker goroutine is not healthy!")
   return
  }
 }
}

Purpose of nil channel:
Avoid busy loop.

func merge(a, b <-chan int) <-chan int {
 c := make(chan int)
 go func() {
  defer close(c)
  for a != nil || b != nil {
   select {
   case v, ok := <-a:
    if !ok {
     fmt.Println("a is done")
     a = nil
     continue
    }
    c <- v
   case v, ok := <-b:
    if !ok {
     fmt.Println("b is done")
     b = nil
     continue
    }
    c <- v
   }
  }
 }()
 return c
}

[LLD 7.0.0] Split stacks.

Ian Lance Taylor [Split Stacks on GCC] https://gcc.gnu.org/wiki/SplitStacks


And thus smells like a usage for goroutine, and tada! :
Ian Lance Taylor [The Go frontend for GCC] https://talks.golang.org/2010/gofrontend-gcc-summit-2010.pdf

Check out the emitted assembly code for further study.


Reference:
https://stackoverflow.com/questions/46313106/how-to-use-fsplit-stack-on-a-large-project

[Go] goroutine scheduling points

Places where Goroutines may yield to others are:

• Channel send and receive operations, if those operations would block.
• The Go statement, although there is no guarantee that new goroutine will be scheduled immediately.
• Blocking syscalls like file and network operations.
• After being stopped for a garbage collection cycle.

reference:

https://dave.cheney.net/2014/06/07/five-things-that-make-go-fast

[Go][goroutine] linux namespace with goroutine

Linux Namespaces and Go Don't Mix
HN: Linux Namespaces and Go Don't Mix (weave.works)
reddit: https://www.reddit.com/r/golang/comments/6ew883/linux_namespaces_and_go_dont_mix/


Cause:
When ever things are blocking, go runtime will fork a M(physical thread) into spinning mode for new goroutine to run before every things are blocked.

However, go runtime will fork the thread with whatever Linux Namespace it's in with the same Linux Namespace scope, which causing problem.

[Go] golang memory model - with unbuffered channel

reference:

https://golang.org/ref/mem

There is a difference between:

make(chan int) // unbuffered equals to make(chan int, 0)

with

make(chan int, 1) // buffered