[structure concurrency][Go][design] Graceful Shutdown

Recently I'm working on Actor design utilizing with Golang's channel.

Link: https://github.com/vsdmars/actor

While bumped into Martin Sústrik's blog post:
Graceful Shutdown: http://250bpm.com/blog:146
Along with the discussion: https://trio.discourse.group/t/graceful-shutdown/93
Structured concurrency resources: https://trio.discourse.group/t/structured-concurrency-resources/21

It's a nice summary, touched the well known concept of pthread cancellation point as well as Golang's context.Done()/Cancel() pattern.

Notes:
In-band cancel:
through application level channel

out-of-band cancel:
through language run-time

Sending a graceful shutdown request cannot possibly be a feature of the language. It must be done manually by the application.


POSIX C thread:
Reference:
https://stackoverflow.com/a/27374983
http://man7.org/linux/man-pages/man7/pthreads.7.html (Cancellation points)

The rules of hard cancellation are strict:
Once a coroutine has been hard-canceled the very next blocking call will immediately return ECANCELED.
In other words, every single blocking call is a cancellation point.

As an additional note, if looking at the POSIX requirement for cancel points, virtually all blocking interfaces are required to be cancel points.
Otherwise, on any completely blocked thread (in such call), there would be no safe way to terminate that thread.

Graceful shutdown can terminate the coroutine only when it is idling and waiting for a new request.

Reasoning:

If we allowed graceful shutdown to terminate the coroutine while it is trying to send the reply, it would mean that a request could go unanswered.

And that doesn't deserve to be called "graceful shutdown".
(e.g. Golang, context.Done() pattern)

Definition summarize:

Hard cancellation:

- Is triggered via an invisible communication channel created by the language runtime.

- It manifests itself inside the target coroutine as an error code (ECANCELED in libdill) or an exception (Cancelled in Trio).

- The error (or the exception) can be returned from any blocking call.

- In response to it, the coroutine is not expected to do any application-specific work. It should just exit.

Graceful shutdown:

- Is triggered via an application-specific channel.

- Manifests itself inside the target coroutine as a plain old message.

- The message may only be received at specific, application-defined points in the coroutine.

- In response to it, the coroutine can do arbitrary amount of application-specific work.

Hard cancellation is fully managed by the language.

Graceful shutdown is fully managed by the application.

Golang library reference:

go-resiliency/deadline: https://github.com/eapache/go-resiliency/tree/master/deadline

Godoc: https://godoc.org/github.com/eapache/go-resiliency/deadline#example-Deadline

[Actor model] Some real-world implementation details from AKKA

Found this talk on YT recently, my comments + notes below:

Introduction to the Actor Model for Concurrent Computation - John Murray

Actor is:

• persistent
• has internal state
• asyc
• Independent event-loop + memory
• Mailbox (receive message)
• React in FIFO order

Well,
actor model is concurrent with regard to a system of actions.
actor model is Not concurrent with regard to data.
(surely we can do implement concurrent thread inside one actor.)

Using channel as mutex to single variable

So, how to make a mutable variable access by multiple threads?

In Golang, we have channel.

• Create a channel with the length of 1.
• Put the mutable variable in, and mutiple goroutines have access to that channel.
• However, only one goroutine can successful retrieve the mutable varible inside the length of 1 channel, other goroutines since the channel has no value, it will block.
• This acts as a MUTEX for goroutines.
• Once the goroutine which having the mutable variable changed,
  it puts it back to the channel thus other goroutines can access.

Actor can:

• create more Actors
• receive messages and response
• make local decisions
• perform arbitrary, side-effecting action
• send messages
• respond to the sender 0 or more times
  (to clearfy here, by "respond" means
  send message to the sender Actor
  Not duplex channel respond)
• Process exactly one message at a time

Actor communication is:

• No channels or internediaries (such as in CSP, e.g. golang)
• "best effor" delivery
• at-most-once delivery
• Messages can take arbitrary long to be delivered (has no concept of time)
• No message ordering guarantees

Actor address:

• identify the actor
• may also represent a proxy / forwarder to an Actor
• contains location and transport information
• don't care where the Actor lives
  (can be inside the same process, different nodes, different containers, etc)
• one address may represent many Actors(pool)

Error handling:

• Supervision
• the running state of an Actor is monitored and managed by another Actor(the Supervisor)
• Supervision has:
• constantly monitors running state of actor
• can perform actions based on the state of the Actor( e.g unhandled error, restart Actor)
  (In Golang, we take advantage of context.Context)
• transparent life-cycle management
• addresses do not change during restarts
  (we implemented with Actor's hash(name + uuid + hostname) as 'address')
  BE WARE.
  This only has meaning iff Actor is not pure, which is STATEFUL.
• Persist state into local sqlite, loaded with (name + uuid)
• mailboxes are persisted outside the Actor instances (Auh, K.I.S.S)
  I doubt the use of Supervisor idea.
  Think about this, does the Sender care how the Receiver act when it receives the message? ;-P

 

Implement Address contains these as a group:

• mailbox
• Actor

Anti use-case:

• Working on a non-concurrent system
• performance critical applications
• non-concurrent communication is involved
• no mutable state

Draw backs:

• too many Actors
• testing
• debugging

Extra material:
Don't use Actors for concurrency


CRDT:
https://medium.com/@istanbul_techie/a-look-at-conflict-free-replicated-data-types-crdt-221a5f629e7e

[LMAX][ACTOR MODEL][Note] A bit about LMAX

Reference:
[AKKA] How the Actor Model Meets the Needs of Modern, Distributed Systems
[AKKA] What problems does the actor model solve?



[Video] The Actor Model (everything you wanted to know...)

• Processing
• Storage
• Communication

One actor is no actor

Actor comes in system

Actor has address

Everything is an actor

When actor receives messages, it can

• Create more actors
• Send messages to other actors that it has the addresses to send to
• Designate what to do with the next message it receives

Future

They are actors can be send around(like a lambda expression/closure)

Address

Address : Actor is many-to-many arch.

Message

Messages are received NO guarantee in order.

It's like the IP(not TCP) model.
IP sequence without considering order(upper layer will do the check).

Sender persists message.

There's NO channels.

Same message can be send at MOST 1 time. No duplicate.

Message sent is not time-bounded. Which can take a long time to arrive to another actor.

Message being handled one at a time, which gives us synchronization.

non-deterministic vs. in-deterministic

Nondeterministic (turing) vs Indeterministic (actors)

E.g: A message that increments a number and then sends the same message again.
Another message, to stop and report the number is on its way. How many increments can happen before the stop message arrives.

arbiter

• Not something you can make out of just (and gates/or gates) and other boolean components.
• Equal number of outputs to inputs. 
• The output order is not determined by the order of input. 
• Unbounded by processing time but the probability of output goes down exponentially as time goes on. 
• Mutated state isn't for the current message.
  Mutated state is for the next message.

LMAX

Paper:

Disruptor: High performance alternative to bounded queues for exchanging data between concurrent threads
(neat paper touch the basic idea of MESI / false sharing / prefetch etc.)


The LMAX Architecture

• Event sourcing (log, RAFT)
• snapshot (RAFT, taking snap shot with commited log)
• unlike RAFT, Business Logic Processor keep multiple active node, each input event is processed by multiple active node.

Concurrent execution of code is about two things:

• mutual exclusion and
• visibility of change.

CAS approach is significantly more efficient than locks because it does not require a context switch to the kernel for arbitration.

However CAS operations are not free of cost.
(Details can be read from book: The art of multiprocessor programming)

The processor must lock its instruction pipeline to ensure atomicity and employ a memory barrier to make the changes visible to other threads.

The ideal algorithm would be one with only a single thread owning all writes to a single resource with other threads reading the results.

To read the results in a multi-processor environment requires memory barriers to make the changes
visible to threads running on other processors.


Solution:
Ensuring that any data should be owned by only one thread for write access, therefore eliminating write contention.


When designing a financial exchange in a language that uses garbage collection, too much memory allocation can be problematic. (golang, and the benefit of using C++)

The LMAX Architecture
https://martinfowler.com/articles/lmax.html

The best programmers prefer profilers and test cases to speculation.

reference:
https://en.wikipedia.org/wiki/Little%27s_law
Ring buffer basics
Circular Buffers in C/C++