[Go] goroutine preemption

Go runtime/g0 schedules the goroutines when:

• system call
• blocking on channel
• sleeping
• waiting on a mutex
• when goroutine needs to grow it's stack size
• goroutines running for more than 10ms is marked as preemptible (the preemption is initiated/monitored by the thread sysmon), and  the preemption is done at the function prolog when the goroutine’s stack is increasing if without any above pause criteria
• For those goroutines does not increasing size, using runtime.Gosched() specifically to force the goroutine being preempted.
• In Go 1.14, goroutines will auto-preempt even no stack size changes or any criteria listed above meet.
  This can be turned off with $ GODEBUG=asyncpreemptoff=1

How does async preemp work?

1. A dedicated thread M called sysmon will watch every goroutines in every P running longer than 10ms.
2. Once over 10ms, sysmon will send event signal SIGURG to those goroutines.
3. Every P has a gsignal goroutine to handle signals. Once a goroutine  runs over 10ms receiving SIGURG will be parked into local queue and gsignal takes over handling the SIGURG.
   This completes the preemption.

 
Why choosing signel SIGURG?

In “Proposal: Non-cooperative goroutine preemption”:

• It should be a signal that’s passed-through by debuggers by default.
• It shouldn’t be used internally by libc in mixed Go/C binaries [...].
• It should be a signal that can happen spuriously without consequences.
• We need to deal with platforms without real-time signals [...].

Like thread cancellation point, goroutine can not be stop anywhere in the code. It has to be a safe point.

Go 1.14's async preemption also enables GC to STW by sending SIGURG to all goroutines.

[Go] g0's roles

g0 traits (for every M has g0)

•     Fix and larger stack size(ordinary goroutin occupies 2kb at start).
•     g0's stack usually doesn't grow.
•     responsible for goroutine creation.
•     responsible for defer function allocation.
•     GC operations. Including STW, mark and sweep operations.
•     Stack growth for running goroutins.
•     Schedule goroutine to run. (each goroutine end with calling runtime.goexit() which notify g0 it's finished) Or goroutine can call runtime.Goexit() manually)
•     Maintain 2 goroutine queues, one with recycled goroutines with an allocated stack(max 2kb size), one with recycled goroutines with empty stack)
•     There are 2 global goroutine queues as well protected by locks.

Each local goroutine queue for P has a maximum capacity of 256, and any new incoming goroutine is pushed to the global queue after local queue is full.

Each P also maintains a queue(64 size) for freed goroutines.(goroutines are recycleable )

Work-stealing

When a processor does not have any work, it applies the following rules until one can be satisfied:

• pull work from the local queue
• pull work from the global queue
• pull work from network poller
• steal work from the other P’s local queues

Threads can be blocked on system calls and the number of blocked threads is not limited in Go runtime.

Affinity limitation

P's Local goroutin queue will be used for all operations expect system calls

such as blocking operations on channels and selects, waiting on timers and locks.

Two features could restrict the affinity between a goroutine and a thread:

• Work-stealing.
  When a processor P does not have enough work in its local queue, it will steal goroutines from others P if the global queue and the network poller are empty. When stolen, the goroutines will then run on another thread.
• System calls.
  When a syscall occurs (e.g. files operations, http calls, database operations, etc.), Go moves the running OS thread in a blocking mode, letting a new thread processing the local queue on the current P.

However, with goroutins using the same channel will be grouped into same P during scheduling. And for those waiting for channel will be scheduled with high priority then other goroutin to run next.

[Go] posix thread TLS(slow) call by Go

Thread Local Storage(TLS)  is slow. (TLPI Ch.31.3.1, Ch.31.4)

Discussion thread:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/golang-nuts/tGamryo50BY/gKxPGwBdAwAJ


Steps:

1. Using runtime.LockOSThread to lock this go-routine to the underlying P
2. Call cgo/c SO.
3. Use channel to pass down messages, other go-routine might be scheduled out of this P(most likely won't if listen to same channel)

[Go] consider the size of a channel

While setting the size of a channel,
should consider how go scheduling mechanism works.

A small size of channel(including an un-buffered channel) causing the producer to block the running goroutine thus forces a re-scheduling (on same physical thread most of the time).

While a larger size of channel can reduce the latency of producing data into the channel but having the consumer delay the processing.

Here's the time slice down into different stages(ref by Vincent Blanchon's article):

Measure the performance before hard-setting the channel size,
i.e make the size configurable.

Reference:
https://medium.com/a-journey-with-go/go-what-does-a-goroutine-switch-actually-involve-394c202dddb7

[Go] runtime.Goexit()

https://golang.org/pkg/runtime/#Goexit

It's been registered into call stack while gorouting being created and being called during stack rewinds.

Which notifies scheduler G0 to schedule next goroutin inside the P's queue to run.


Reference:
Assembly notes: http://vsdmars.blogspot.com/2017/11/assembly-note.html

[Go] use sync.Pool with sense

https://github.com/golang/go/issues/23199

tl;dr
It's caller's responsibility to put back normalized size of instance back into
sync.Pool

Intermingling small and large size of instance put into the sync.Pool will
eventually getting all the instance inside the pool with large size.
(Read Reference 1)

thus either create a bunch of sync.Pool bucketize the items by size (aka. slab memory allocation) or only put back certain size of instance back to the pool.

Same idea as goroutin instance recycling.
Go runtime will not recycle any goroutin size > 2k.


Reference:
https://github.com/golang/go/issues/23199
http://vsdmars.blogspot.com/2019/01/split-stack-reading-notes-and-references.html

Reason:
the buffer inside the sync.Pool will remain LARGE and never shrink even serving small objects
code:

pool := sync.Pool{New: func() interface{} { return new(bytes.Buffer) }}

processRequest := func(size int) {
	b := pool.Get().(*bytes.Buffer)
	time.Sleep(500 * time.Millisecond) // Simulate processing time
	b.Grow(size)
	pool.Put(b)
	time.Sleep(1 * time.Millisecond) // Simulate idle time
}

// Simulate a set of initial large writes.
for i := 0; i < 10; i++ {
	go func() {
		processRequest(1 << 28) // 256MiB
	}()
}

time.Sleep(time.Second) // Let the initial set finish

// Simulate an un-ending series of small writes.
for i := 0; i < 10; i++ {
	go func() {
		for {
			processRequest(1 << 10) // 1KiB
		}
	}()
}

// Continually run a GC and track the allocated bytes.
var stats runtime.MemStats
for i := 0; ; i++ {
	runtime.ReadMemStats(&stats)
	fmt.Printf("Cycle %d: %dB\n", i, stats.Alloc)
	time.Sleep(time.Second)
	runtime.GC()
}

[Go] cgo function call act as a system blocking call for Go's scheduler point of view

email thread:
https://groups.google.com/forum/#!msg/golang-nuts/QydReNgFe00/CsMsWKdzAwAJ

Current fact(GO 1.14):
the Go scheduler will actively try to interrupt CGO calls that take too long.


Quote from Ian Lance Taylor:
When a cgo call starts it has a P attached to it (in the scheduler, a P is a virtual processor; there are exactly GOMAXPROCS P's at all times).

If the cgo call completes quickly, it will simply carry on with the same P.

When the system monitoring thread wakes up, it will check each P to see if it has been waiting for a cgo call to complete for more than a scheduler tick (20 microseconds, more or less).

If so, the P will be stolen and some other M (operating system thread) will be woken up to start using the P and running Go code. (added: since C/C++ code is considered in blocking state)

When the cgo call completes, the goroutine will see that it no longer has a P, and will go to sleep waiting for a P to become available (more or less as though the goroutine called runtime.Gosched).

So in that sense, cgo calls will be interrupted: the P will be removed and reassigned to do other work. However, the actual C/C++ code running on the M (operating system thread) will not be affected.  And the G (goroutine) will of course remain asleep waiting for the cgo call to complete. (This is all a description of the current 1.14 scheduler, and it may be different in other releases.)

[Go] mcache ∈P

Reference:

https://groups.google.com/forum/#!msg/golang-nuts/umeli6cYXT4/okFqZm-8BQAJ

mcache belongs to the P:

https://github.com/golang/go/blob/7955ecebfc85851d43913f9358fa5f6a7bbb7c59/src/runtime/runtime2.go#L557

Quote from Ian Lance Taylor:

We used to have the possibility of an M using the mcache of a P even though it was not running on the P.
That happened in the helpgc function which was run by gchelpers.
But gchelpers was removed in https://golang.org/cl/134785, so it may be that we no longer need the mcache field on an M.

[Go] channel in detail

Channel traits:

• goroutine is safe
• store and pass values between goroutines, Golang is a value based language same as C and C++
• provides FIFO semantic
• can cause goroutines to block and unblock

How would you design a channel?

Ring buffer!

With ring buffer, we need:

• the type that this ring buffer holds, thus compiler knows the size of the ring buffer which could allocate it's memory.
• start idx(hchan.recvx), end idx(hchan.sendx) (thus we know the current data size)
  while hchan.recvx == hchan.sendx means either 0 data or ring is full.
• ring buffer size
• how many senders/receivers are consuming the ring
  thus we need mutex lock

Let's take a look at hchan struct in golang chan.go implement:
https://github.com/golang/go/blob/master/src/runtime/chan.go#L32


hchan instance usually allocated on the heap, unless this channel is only been used in one single thread(stack, to be precise).


ch <- 42 will copy data into the ring buffer with mutex locks hchan, increase the hchan.sendx.
<- ch will copy the value from ring buffer with mutex locks hchan, increase the hchan.recvx.


If the ring buffer is full, send blocks, and golang scheduler suspends the send goroutine (aka. user space thread, not the OS thread) until there's room in the ring buffer.

Please refer to previous post about scheduler:
http://vsdmars.blogspot.com/2018/06/golang-go-scheduler.html
Golang scheduler design doc:
https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw


Each (G)oroutine is managed by (P)rocessor, and each (P)rocessor can only be run by one (M)thread.
(P)rocessor has the runQ (list) holds (G)oroutines.

If the ring buffer is full, the sender pauses, scheduler calls 'gopark' to remove the caller (G)oroutine from (M)thread into (P)rocessor's runQ(of (G)oroutines).
https://github.com/golang/go/blob/master/src/runtime/proc.go#L284:6


hchan has the information of senders and receivers.
https://github.com/golang/go/blob/master/src/runtime/chan.go#L53:6

hchan.sendq // waitq
hchan.recvq  // waitq

type waitq struct {
    first *sudog
    last  *sudog
}

sudog:
https://github.com/golang/go/blob/master/src/runtime/runtime2.go#L276

sudog.elem represends the data which is about to send(copy)/receive(copy) to/from the ring buffer.



Here's the interesting part, the implementation optimized sending/receiving data by skip copy data into/out from ring buffer iff there's a waiting (G)oroutine, which directly store the sending/receiving pointer to data into the waiting (G)oroutine's sudog.elem
Receive optimized:
https://github.com/golang/go/blob/master/src/runtime/chan.go#L190
Send optimized:
https://github.com/golang/go/blob/master/src/runtime/chan.go#L473



Fast path implement:
Sender:
https://github.com/golang/go/blob/master/src/runtime/chan.go#L159
Receiver: (atomic.Load involved)
https://github.com/golang/go/blob/master/src/runtime/chan.go#L437


Quote from Ian Lance Taylor:
https://groups.google.com/forum/#!topic/golang-nuts/L-bGvB52UrU
The runtime package is always compiled by a known compiler, and is permitted to know how that specific compiler behaves. If the compiler changes, while still staying within spec, the runtime package will change too. You can't argue about the language's memory model based on the code in the runtime package.


Reference:
Ring buffer in C:
https://embeddedartistry.com/blog/2017/4/6/circular-buffers-in-cc
chan.go:
https://github.com/golang/go/blob/master/src/runtime/chan.go
gopark:
https://github.com/golang/go/blob/master/src/runtime/proc.go#L284:6
goready:
https://github.com/golang/go/blob/master/src/runtime/proc.go#L310:6
Benign Data Races: What Could Possibly Go Wrong?
https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
The Go Memory Model:
https://golang.org/ref/mem

[Go] Golang's memory management - Eben Freeman

Eben Freeman has given a great talk about Golang's memory management
This is a note for the talk and knowledge add-ons.

Reference:
GopherCon 2018 - Allocator Wrestling https://about.sourcegraph.com/go/gophercon-2018-allocator-wrestling

Knowledge from C++ world:
While coding in Go, which has escape analysis provided by the compiler,
consider writing the type instance as auto variable which avoid compiler putting
it into the heap.
Don't escape unnecessary auto variables by returning it's address.
Return by value is preferred if the type size is small.
(i.e If the type is large, it's instance will be put into heap anyhow.)
Since auto variables are not managed by GC :-)

Allocator internals

Design goals

• Efficiently satisfy allocations of a given size, but avoid fragmentation
  Solution: allocate like-sized objects in blocks
• Avoid locking in the common case
  Solution: maintain per-CPU caches
• Efficiently reclaim freeable memory
  Solution: use bitmaps for metadata, run GC concurrently

Heap is divided into two levels of structure: Arenas and Spans

(The idea is same as slab allocator:

https://en.wikibooks.org/wiki/The_Linux_Kernel/Memory#Slab_Allocator )

A global mheap struct keeps track of both of them:

https://github.com/golang/go/blob/master/src/runtime/mheap.go#L31

type mheap struct {
  arenas [1 << 22]*heapArena       // Covering map of arena frames
  free []mSpanList                 // Lists of fully free spans
  central []mcentral               // Lists of in-use spans
  // plus much more
}

Arenas are coarse sections of the available address space (64MB on 64-bit archs).
We allocate OS memory in units of arenas.
For each arena, there's metadata in a heapArena struct:
https://github.com/golang/go/blob/master/src/runtime/mheap.go#L201

type mheap struct {
  arenas [1 << 22]*heapArena    // Covering map of arena frames
  // ...
}

type heapArena struct {
   // page-granularity map to spans
  spans [pagesPerArena]*mspan
  // pointer/scalar bitmap (2bits/word)
  bitmap [heapArenaBitmapBytes]byte 
}

Span:

There are ~70 different mspan size classes.
https://github.com/golang/go/blob/master/src/runtime/mheap.go#L308

type mspan struct {
  startAddr  uintptr
  npages     uintptr
  spanclass  spanClass

  // allocated/free bitmap
  allocBits *gcBits
  // ...
}

Each P has an mcache holding a span of each size class.
https://github.com/golang/go/blob/master/src/runtime/mcache.go#L19
Ideally, allocations can be satisfied directly out of the mcache (thus they're fast).
In Go, a P is a scheduling context.
( Refer to Golang scheduler note: http://vsdmars.blogspot.com/2018/06/golang-go-scheduler.html )
Generally there's one P per core (M, aka. thread) and
at most one goroutine running per P at a time(P has running queue):

To allocate, we find the first free object in our cached mspan, then return its address:

Let's say we need 96 bytes.

First we'd look in the mcache for the mspan with 96-byte objects.

After allocation would look like this:

This design means that "most" memory allocations are fast and require no locking.

There are 3 quick steps:

1. Find a cached span with the right size ( mcache.mspan[sizeClass] )
2. Find the next free object in the span
3. If necessary, update the heap bitmap

Garbage collection

Go uses a tricolor concurrent mark-sweep garbage collector.

Reference:

Modern garbage collection A look at the Go GC strategy https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e

Tri-Color Incremental Updating GC https://stackoverflow.com/questions/2364274/tri-color-incremental-updating-gc-does-it-need-to-scan-each-stack-twice

Golang’s Real-time GC in Theory and Practice https://making.pusher.com/golangs-real-time-gc-in-theory-and-practice/

Tri-color mark steps:

• Initially, all objects are marked as white. (3 colors exist: white, grey and black)
• GC starts by marking goroutine stacks and globals.
• When GC reaches an object, mark it grey.
• When an object's referents are all marked, GC mark the object with black.
• At the end, objects are either white or black.
• White objects can then be swept and freed.

However; GC can be run concurrently, thus:

type S struct {
  p *int
}

func f(s *S) *int {
  r := s.p
  s.p = nil  // What if GC runs at this point? The returning pointer points to null.
  return r
}

Rough solution explained:
Preventing this from occurring, compiler translates pointer writes into potential calls into the write barrier.
i.e Mark it has referent during a pointer assignment.
i.e When GC is marking, the write barrier is turned on.

quote:

Mark Setup - STW When a collection starts, the first activity that must be performed is turning on the Write Barrier.
The purpose of the Write Barrier is to allow the collector to maintain data integrity on the heap during a collection since both the collector and application goroutines will be running concurrently.
In order to turn the Write Barrier on, every application goroutine running must be stopped.
This activity is usually very quick, within 10 to 30 microseconds on average.
That is, as long as the application goroutines are behaving properly.

Reference:

https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html#:~:text=The%20purpose%20of%20the%20Write,goroutine%20running%20must%20be%20stopped.

That is to say:
Allocating a big buffer of scalars is much cheaper than allocating a big buffer of pointers, because you have to follow the pointers.

Experiment with GC performance

1. Crude experimenting

• GOGC=off : Disables garbage collector
• GODEBUG=sbrk=1 : Replaces entire allocator with simple persistent allocator that gets big blocks from the OS and gives you successive slices of those as you allocate.

Problem with this approach:

Not useful for production code due to not reliable with large heap allocation.

2. Profiling

• Use pprof to check for runtime.mallocgc
• Use flamegraph viewer in the pprof web UI
• Or linux perf if pprof isn't enabled in the binary.

    

Problem with this approach:

• Program might not be CPU-bound
• Allocation might not be on critical path
• Time in background marking (gcBgMarkWorker) can mislead (time spent here doesn't necessarily mean you have net slowdown)

3. go tool trace

$ curl localhost:6060/debug/pprof/trace?seconds=5 > trace.out

$ go tool trace trace.out

Coding suggestion

• Limit pointers
  Compiler will show which variable is on heap.
  Software engineering 101, no matter what language you use, don't construct variables inside a loop.
  If you really really need to, the type to be constructed should not have pointer type inside, which helps the GC to reasoning.
  $ go build -gcflags="-m -m"
• Allocate in batches (Hand make your own slab allocator)
• Try to recycle objects (e.g., sync.Pool) ( Refer to note for Concurrency in Go - Katherine Cox-Buday http://vsdmars.blogspot.com/2018/10/golangbook-concurrency-in-go-katherine.html )

Further reference:

https://segment.com/blog/allocation-efficiency-in-high-performance-go-services/

https://blog.golang.org/ismmkeynote

Must read: https://www.slideshare.net/TylerTreat/so-you-wanna-go-fast-80300458

Note for Tyler Treat's slides:

Scheduler is fairly grouping goroutines holding the same sync.Mutex instance on same CPU core avoiding ping-pong effect.

Use defer only if there's more then one return lambda being registered.
If single return condition, don't use defer due to defer is relatively _SLOW_.

Use DOD for designing types http://vsdmars.blogspot.com/search/label/design_dod

Use interface is always slow than pure type due to another layer of indirection.

Convert []byte to string type judiciously.

Use sync.Pool while necessary.

“We generally don’t want sync/atomic to be used at all...Experience has shown us again and again that very very few people are capable of writing correct code that uses atomic operations...”
—Ian Lance Taylor
Why? This is why: http://vsdmars.blogspot.com/search/label/cpp_concurrent

The Go race detector doesn’t protect you from doing dumb stuff.

Unsafe is, in fact, unsafe.

Struct layout can make a big difference. (padding http://vsdmars.blogspot.com/2018/09/golangc-padding.html & ping-pong effect with mutex lock)

Go makes concurrency easy enough to be dangerous.

[Go][note] Analysis of the Go runtime scheduler paper note

Reference:
Analysis of the Go runtime scheduler paper


Goroutines communicate through a construct known as channels,
which are essentially synchronized message queues.

Go Runtime manages

• scheduling, 
• garbage collection,
• and the runtime environment for goroutines among other things.

Multiple threads are often necessary to ensure that goroutines
 are not unnecessarily blocked.

When one goroutine makes a blocking call, the thread running it must block.
(不一定 有些block為 userspace, 則 thread可以不用block.
e.g:

• network input
• sleeping
• channel operations
• blocking on primitives in the sync package.

)

M's top is set to GOMAXPROCS.

--
3 types used to handle scheduler/goroutine

• THE G STRUCT (aka. goroutine)
  A G struct represents a single goroutine. It contains the fields necessary to keep track of its stack and current status. It also contains references to the code that it is responsible for running.
• THE M STRUCT (aka. thread)
  The M struct is the Go runtime’s representation of an OS thread.
  It has pointers to fields such as the global queue of G’s, the G that it is currently running, its own cache, and a handle to the scheduler.
• THE SCHED STRUCT (not logical processor, just the scheduler, this is old design with Sched Sched lock)
  The Sched struct is a single, global struct[9] that keeps track of the different queues of G’s and M’s and some other information the scheduler needs in order to run, such as the global Sched lock.
  There are two queues containing G structs, one is the runnable queue where M’s can find work, and the other is a free list of G’s. There is only one queue pertaining to M’s that the scheduler maintains; the M’s in this queue are idle and waiting for work. In order to modify these queues, the global Sched lock must be held.

The runtime starts out with several G's.

• garbage collection
• scheduling
• represents the user's Go code

The M will not be blocked by G's hitting channel block.
If there's a channel block inside the goroutine, we simply mark
the G into wait state and put it back into the run queue, while the
channel is unblocking, we put the channel back to scheduling.

One problem is the scheduler's excessive reliance on the global Sched lock.
In order to modify the queues of M’s and G’s, or any other global Sched field for that matter, this single lock must be held.
This creates some problems when dealing with larger systems, particularly “high throughput servers and parallel computational programs",
which causes the scheduler to not scale well.

[issue] Sched lock:
Introduce logic processor, P.
There are exactly GOMAXPROCS P’s, and a P would be another required resource for an M in order for that M to execute Go code.

Whenever a new G is created, it is placed at the back of the queue of the P on which it was created, thus ensuring that the new G will eventually run.

When a P does not have any G’s in its queue, it will randomly pick a victim P and steal half of the G’s from the back of the victim’s queue.

[issue] M’s continuously blocking and unblocking:

Three main events that can cause an M to be temporarily incapable of running Go code:

• when a new G is spawned,
• an M enters a syscall,
• an M transitions from idle to busy.

Before becoming blocked for any of these reasons,
the M must first ensure that there is at least one spinning M,
unless all P’s are busy.

Why?
Thus the blocking M's P's running queue can hand off tasks to the spinning M.


[issue] Creating a goroutine needs lot's of memory:
Not allocating the G and stack for a new goroutine unless they are really required.

We require just six words for the creation of a goroutine that runs to completion without making function calls or allocating memory.

[issue] Locality to a CORE CPU:
P’s are an abstraction created by the run-time that the OS knows nothing about, whereas M’s represent kernel threads.

Most modern kernels will provide for affinity between threads and physical processors. Hence, better G to M locality will give us better cache performance.

Scheduling:
Put a job's tasks spread out into P's. (they are not dependent)
If can't, put into next row(level) of P's.
Like k8s or any batch scheduling technique.

If Go's P structs were used, instead of processors, this idea could work to schedule multiple goroutines that use the same channels simultaneously.

This has the potential to reduce the time spent blocking M's and G's;
however, this may require significant changes to the channel infrastructure.

Contention Aware Scheduling:
goroutines using same channel should schedule into same P.
As for task stealing from other P, a P should steal a group of
G's which are grouping together due to using the same channel.

Type system:
“A type system that was used for classification by behavior rather than implementation, or naming.”

[Go] The Go scheduler

Reference:
Morsing's The Go Scheduler article
Dmitry Vyukov's Scalable Go Scheduler Design doc
MIT Paper: Scheduling Multithreaded Computations by Work Stealing
Intel TBB Task Scheduler
Go's work-stealing scheduler
Command trace

Why a userspace scheduler for go?
Because of garbage collector.

The Go garbage collector requires that all threads are stopped when running a collection and that memory must be in a consistent state.
This involves waiting for running threads to reach a point where we know that the memory is consistent.

Golang is using M:N thread model, i.e M as gorotines, N as OS threads.

layout:

G: goroutine (aka. Task, can be stolen)
M: OS thread (machine, aka. OS thread)
P: processor, NOT CPU processor, but a logical one (aka. context, can steal, own 1 M)
The number of P is set to GOMAXPROCS/runtime function GOMAXPROCS()
Each P has it's own local runqueue, which contains Gs. 
This design relieve mutex contention on single thread. 
Why? Because there's no mutex variable crossing CPU cores , which there's no WB/RB barriers involves.

Why P(context)? Because P can be moving around the threads.
If 1 P is blocked due to it's goroutine called a syscall, we could have that goroutin + M stay and moving the P with it's runqueue to other thread to run, consider P as a package can be move around to other threads.

The scheduler makes sure there are enough threads to run all contexts.

When the syscall returns, the (blocking) thread must try and get a context(P) in order to run the returning goroutine.
The normal mode of operation is to steal a context(P) from one of the other threads. If it can't steal one, it will put the goroutine on a global runqueue, put itself(i.e the 'was' blocking thread) on the thread cache and go to sleep.

Global runqueue:

• Global queue: G's can be put into here if original P is blocked and resumed(awake) but can't grab a M.
• The global runqueue is a runqueue that contexts(P) pull from when they run out of their local runqueue.
• Contexts(P) also periodically check the global runqueue for goroutines. Otherwise the goroutines on global runqueue could end up never running because of starvation.

Logic sequence:
runtime.schedule() {
    // only 1/61 of the time, check the global runnable queue for a G.
    // if not found, check the local queue.
    // if not found,
    //     try to steal from other Ps.
    //     if not, check the global runnable queue.
    //     if not found, poll network.
}

Task Stealing:

• If local runqueue is drained, look into global runqueue.
• If the global runqueue is empty, the P will steal from other P's local runquene, half of them.

Spinning threads(M):

A thread is spinning if:

• An M with a P assignment is looking for a runnable goroutine.
• An M without a P assignment is looking for available Ps.
• Scheduler also unparks an additional thread and spins it when it is readying a goroutine if there is an idle P and there are no other spinning threads.

Logical Processor: (This is the essential of Go)

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/golang-nuts/5Zo403s3LC0/P4JwsHsKAQAJ

The purpose of a P is to limit the amount of total concurrency running Go code. 

By default set the number of P's to the number of CPU cores on the system (including hyperthreading).  

The user can control it by setting the GOMAXPROCS environment variable, and the program can control it by calling runtime.GOMAXPROCS

M's, on the other hand, which are operating system threads, are started as needed, so the user and program have no control over them.  

G's, which are goroutines, are started by the program but there is no way to limit the total number of goroutines. 

So using P's is how the Go runtime tries to keep the program running efficiently without thrashing.

Design Doc:

https://docs.google.com/document/d/1TTj4T2JO42uD5ID9e89oa0sLKhJYD0Y_kqxDv3I3XMw/edit